Month: July 2015

I hate pressure.

I hate pressure so much that I do whatever it takes to avoid it, consequences be damned.

The pressure to have a really awesome personal site as a web developer kept me from ever making one.

The pressure to be the kinda guy that girls would wanna date kept me from appreciating myself, quirks and all.

The pressure to live up to people’s expectations kept me from thinking critically about what I was doing and if it really made sense to me.

Pressure inflates certain points and downplays others. It becomes so much harder to make clear-headed rational decisions when it matters most.

“Dude, you can’t get diamonds without pressure”

Fuck diamonds, fuck pressure and fuck you too.

You do have a point though.

Pressure pushes us accomplish things we would never have, under normal circumstances.

I’m not saying pressure is stupid or pointless. It’s definitely made me more aware of how much further out my real boundaries are.

Even when I fail to overcome it, I still learn valuable lessons.

After years of being overwhelmed by the pressure for an awesome site, I learned that waiting for things to be perfect isn’t worth it.

Conditions not being perfect don’t put me under pressure anymore.

I never became the guy I thought girls would want, but I learned that some girls like me for who I am, and it’s much better to be in a pressure-free relationship than to continually struggle to live up to arbitrary expectations.

When you overcome pressure, you expand your limits. When you cave to it, you learn about yourself. As frustrating as it is, it definitely has its benefits.

I guess the best way to summarize this is “fuck pressure but embrace it”.

I’m in a Whatsapp group conversation with a few friends of mine. Our conversations typically revolve around hip-hop, decoding lyrics and proving how much more talented our favorite rappers are than everyone else’s favorites.

Eminem is Alex’s favorite. He’s been rapping for many years, has a lot of awards, is critically acclaimed, earned the respect of his peers and is financially successful. He’s also succeeded in an industry that doesn’t have very many people who look like him. Totally understandable pick.

Nivlek likes Kendrick Lamar and Lupe Fiasco. Both newer faces than Eminem. More importantly, supremely talented. Layers of cleverly laid references in their lyrics. Definitely not as financially successful as Eminem, but their wordplay is outstanding. They produce high-quality music.

 

Me? I like MF DOOM. Crazy intriguing lyrics and overall very talented. Even produces and makes beats. His face is always hidden behind a giant metal mask. Financial success? Who knows. Most hip-hop fans don’t even know him. He’s that obscure.

I brought this up because after my conversations with Ire, I’ve been thinking a fair bit.

What does success look like to me?

Is it looking more like Kendrick? Overflowing with talent and a promising future in the limelight? Or do I want to be like MF DOOM? Completely out of the public’s eye, but those in need my skills know where to find me?

Do I wanna be like Eminem? Most of the world would never consider me successful if I don’t have an obvious pile of money. A nice big house. Great car. Very popular clients. Plenty of accolades. Funded by popular investors. A product that’s considered a unicorn (don’t ask .. it’s a nerd thing). Do I need the trappings of success to be a success?

Can I be more like Kendrick? I’d be the guy the Eminems of software engineering wanna work with because I’m oozing raw talent so they need me to build something amazing, special and really technically impressive that’d give them an edge? Is this what success looks like?

Or is it MF DOOM that speaks to me more? Unknown to the general public, but the person that Kendricks and Lupes of software engineering look up to me because I build things that help them be awesome. It’s probably not as financially rewarding, but do I need mountains of cash to be a success?

Had an interesting lunch with Ire today.

She writes about web development on bitsofcode, which is ridiculously popular. People have shared stuff she’s written on sites like Reddit, and some of her posts have gotten to the front page. Amazing!

In contrast, nobody reads what I write or uses what I’ve made. I’d come to believe that will change over time. Then I met Ire.

She started writing in March and already gets so many people reading her blog and using the things she’s made. She clearly knows a thing or two about making things people want. I asked her if I could buy her lunch, hoping I can pick her brain over the meal, and she agreed.

 

During our conversation, she asked me a few pointed questions. Answering them opened my eyes.

“Who is your audience? Who do you write for?”

“I write for myself.”

“Oh you mean people like you?”

“No .. I actually mean for myself. Once, I wrote a post with literally no objective. I let my mind wander and just wrote.”

Interesting. Why was I surprised nobody else reads my blog?

She writes to help others learn about building websites. She researches and puts together brilliant, concise pieces that help you understand a little bit more. An overwhelming majority of her posts contain information that is valuable to me, and I’ve been building sites for more than 5 years.

People find her posts useful and share it with others. Her traffic isn’t coincidental, really. She writes for others and the results are commensurate.

 

Even the things she’s made gets use.

“When I’m building, I make it really easy for people to initially use and customize. They can go deeper and tweak it more later on, but it needs to be super easy for them to use in the beginning.”

When I build outside work, it’s typically to an audience of one. Myself, or the friend who is going to end up using it. It usually requires modification for others to use it. Why am I surprised nobody uses stuff I’ve made? It requires an upfront commitment in the first place. Nobody has that kind of time.

Overall, it was a very enlightening conversation. I’m writing to an audience of one. I’m building for a user base of one. The results are commensurate.

Adim and I started blogging to start networking with developers in Nigeria. We felt isolated and wanted to get to meet our peers, connect with them and work on interesting things together.

To be honest, I’d also like for people to know what I do, what I’m capable of and want to work with me. That’s why I started writing more frequently, documenting my thought process as I work.

The trouble is I’m writing with no consideration for my target audience.

This is more like a public diary than anything else, at the moment. Nothing wrong with that, but it doesn’t address why I write.  If I want specific results, I need to do things that move me at least one step in that direction.

I need to take some time to write for others if I want them to read my writing. It needs to be packaged in a format that they can quickly glean the information, because apparently other people don’t like reading. I knew this, but since I love reading, never really optimized my posts for most people who don’t like reading.

Making sure things I make are easy for people to pick up and use with as little configuration as possible is super critical. Adim’s always talking about things “it just works” when we get into our Apple vs Android / Windows discussions. Build it so it works with the least amount of fuss and configuration. Make it so that it just works. I guess I’m starting to understand what he means.

I’m not going to stop writing for myself. I like doing these sorts of things. What I need to do is keep in consideration why I write.

On a regular schedule, package posts for others. When I’m extracting some code to form a library, make it as “drop in” as possible, with excellent default settings.

I write and build software for others to enjoy. It’s time I started ensuring the things I make reflect that.

Yesterday I resolved to thoroughly test my current project before building any new features.

Figured the best place to start would be in a control flow library I wrote, which is used all over the API server. I use it to make complex processes manageable and readable.

Comments are practically redundant because the name of each step explains what the snippet of code does.

I use the library all over the codebase, so it makes sense to thoroughly test it and proceed from there.

Imagine how humbled I was when it failed 20% of my tests.

I may have gotten away with this for as long as I was the only person using it, but as soon as anyone else decided to use cjs-task, they had a 1 in 5 chance of having it fail them unexpectedly.

I’ve been building on a shaky codebase.

Time to pass all the tests. Then write some more.

Note to self: WRITE AUTOMATED TESTS, STUPID.

I have an embarrassing programming secret. Please don’t judge me harshly.

I don’t use automated tests.

 

I’m not a complete moron. Whatever I’m currently working on is thoroughly tested manually before being committed in git. And I commit pretty frequently with descriptive commit messages. I also fork and merge religiously.

I haven’t really felt the need to set up automated tests. Not til yesterday.

THAT ONE TIME I WRECKED IT

I’m currently building an API server for a project. An offline-capable smart spreadsheet browser app.

Made some updates to the client and wired it up to the API endpoint to test the feature and the server came crashing down. My changes were to the client, which was in a separate repo, so I didn’t change anything on the server. Restarted the API server and tested it again. Crash and burn.

What went wrong?

A quick investigation pointed to when I renamed a few variables to make things more readable. I didn’t change every instance of a particular variable. Two slipped by me. I didn’t even realize there were two instances til I fixed one, pushed the change up and the server crashed again.

LESSON LEARNED

It’s supremely easy to make mistakes or not check as thoroughly as you think, which introduces bugs into you code.

Manual testing is okay, but you can’t count on it to catch every single problem. And every single problem matters in production. It’s the difference between a well-running app and steaming pile.

No more relying on myself to make sure everything is working properly. I’m writing a complete test suite for the server before I build any further.

From now on, I’ll test it so I don’t wreck it.

 

“Twelve scanners! For that Sheraton job there were three scanners and it was too much work to review properly. The workload is going to be insane.”

Homeboy, the pessimist he is, was highlighting the downside of his company’s most recent contract. To be fair, he was one of the supervisors on the Sheraton gig and was literally drowning in the work, so the pessimism wasn’t unmerited.

I’d dropped by to return a flash drive and somehow ended up in a lengthy discussion about his challenges at work.

“What makes a supervisor’s role so hard?”

“We need to make sure the scanners are doing their job correctly. We were excited about almost completing the last job when we discovered a problem two scanners made weeks ago. We had to spend an extra month fixing it up.”

“What exactly do supervisors do?”

“We check to see if the scanned documents tally up with the report sheet the scanner produces. A document is a collection of PDF files. We make sure the PDFs are in the right document and have the right number of pages.”

“That sounds tedious.”

“In a week, a scanner can generate up to a thousand PDFs. It’s very easy to make a mistake and put a file in the wrong document or write the wrong page tally. It’s our job to catch and correct those mistakes.”

I stood in silence as I absorbed the gravity of the situation. 1000 PDFs per scanner is a frightening amount of data to manually verify. Pessimism must be contagious because I felt myself coming down with it. Making software is tricky, but it certainly wasn’t a tireless grind like this.

Maybe I could build something that could help him. He could select the folder containing the documents and it will determine what PDFs are in what documents, how many pages each PDF has and generate a report.

“I’ll build something to help you this weekend. Don’t sweat it”.

“Dude I’ll never ask you for your sister’s phone number again in my life if you do”.

PLANNING THE APP

I’ve only built web and browser apps, but everything about this sounds like a desktop app.

It can’t be a browser app. There aren’t APIs in the browser that allow me to crawl a client’s system. It can’t be a web app either. Uploading that many PDFs is seriously time-consuming and data intensive.

If I could get Node.js on his system, I can treat his computer like a server and build a web interface for him to interact with. Made for his computer, using web technology.

My search led me to Electron. Desktop applications built with Node.js? Everything I need to build something for a desktop with tools I use daily.

Getting “Hello World” up and running in Electron was a piece of cake. The documentation gave me enough information to do so on the first page. The code was easy to reason about and immediately dealt with the finer points of what differentiated electron apps from regular Node.js apps.

There’s a main process and a render process. Think of main as the server and render as a browser.  In a traditional web application, all the brains are on the server and the client browser talks to the server by sending text via HTTP. In Electron, the render process talks to the main process by sending text through a package called ipc.  You coordinate both processes via ipc to accomplish your object just like the browser and server are coordinated via http requests.

Once I understood this much, the plan was crystallized. Use node’s fs package to do the crawling, use comma-separated-values to generate the report excel doc and use the interface to collect input from Homeboy.

Time to start wiring this thing up!

BUILDING IT OUT

Being able to drag and drop the folder containing the documents would be most intuitive way to get the document location, so I started building from there.

Normal browsers have a security model that prevent you from getting a file’s full path with the File API, but it was there for the taking in Electron.

Used ipc to send the paths to the main process and started writing the script to crawl the paths.

The first version I wrote was procedural. Get a list of folders and retrieve their content.  If any are PDFs, report them. If they are folders, go into those and do the same thing.

No matter how deeply nested the folder structure was or how files there were, it returned the information in less than a second. Amazing fast!

Every virtue the first version had was made irrelevant by how unusable the user interface was. It had one line of unstyled text that said little more than “drop folders here”. When you do, it creates some barely-styled boxes with the folder path, number of files and number of PDFs. It worked though, so I was super excited to demo it to him.

FIRST DEMO

Homeboy watched it run through the test data and display the results.

“Can it tell me how many pages are in the PDFs?”

“That’s the very next feature I’m adding.”

“Okay. It needs to be able to tell me how many pages are in the PDFs and create an Excel sheet with that information.”

Was it too much to expect a little gratitude at this stage? No matter. I’ll just use a package from npm to get the page count. Shouldn’t take me more than 15 minutes max.

Two npm packages and as many hours later, still no page counts.

PDF PAGE COUNTS

My first attempt was using an aptly named package: pdf_page_count. The package kept returning with “undefined” no matter what I did. Homeboy’s periodical “is it done yet?” didn’t help matters. After taking a look at how many people downloaded it recently, I came to the conclusion that the package was broken. Needed to find an alternative.

pdf2json seemed to fit the bill. Way more uses and downloads than pdf_page_count, so I installed it and gave it a shot.

No such luck.

I was so close! All I need is the page counts and I can output out on the interface and generate the excel report.

SO. CLOSE.

It didn’t help that the instructions for pdf2json  were woeful, so it felt like I was listening to the wrong event or setting the listener on the wrong object or something.

In my desperation I started stepping through the source code and adding log statements so I can follow the process through and see what’s wrong.

I was listening to the wrong object. The right one nested inside. Quick change, restart app and … nothing.

Turns out the information isn’t passed to the nested object. You have to check the outer one again. Quick change, restart app and … still nothing?!?

SO. FUCKING. CLOSE. WHY ME? WHY IS LIFE SO HARD? WHY DO BIRDS SUDDENLY DISAPPEAR? WHY IS JADAKISS AS HARD AS …wait a minute …

Oh wow. Turns out I was only passing the name of the PDF, not where it was located. Quick change, restart app and …. PAGE NUMBERS!

Quickly exported the app to the other laptop for him to test and proceeded to watch it hang up his computer for five minutes before producing the report.

“Is it exporting to Excel yet?”

BACK TO THE LAB

Though I was able to extract page numbers, I was completely unsatisfied with its performance. It took an unreasonably long time to get.

To make matters worse, it completely hung up the system while getting the page counts. Forget multi-tasking. The mouse barely responded to movement and you certainly couldn’t click on anything.

pdf2json grabs a whole lot more than page counts. It tells you the height of each page, what is contained on every line (vertical AND horizontal), text and so much more. Useful information for someone else but totally irrelevant to Homeboy’s needs.

I took another look at pdf_page_count, armed with the knowledge that its failure was most likely due to me not passing the right page path.

A quick swap later it was running correctly. Way faster too! Understandable, since it’s a one-trick pony. Or a one-trick Veyron, speed improvement considered.

Processing the PDFs still hung up the system though. The app would navigate to the folders and go as deeply as it could, sorting between directories and files and trying to get the page numbers from the PDFs. My sample data has 2999 PDFs in total and the current architecture processes them all at the same. No wonder the system becomes completely unusable.

I created a queue where the PDFs to process are placed. I instructed the app to process a maximum of 8 at a time (get it? 8 looks like an upright mobius strip). The end result is not as many PDFs are processed at once. Definitely wouldn’t be as fast as opening all 2999 at once, but the system doesn’t experience any performance degradation. He’d be able to drop a bunch of folders to be processed and can muck around on Facebook, check back in 5 minutes and it’s all done.

UGLY DUCKLING TO SWAN

I could have aimed for feature complete at this point and done the Excel exporting but the unusable user interface bothered me far too much. I knew when the app was processing and when it stopped simply because I had a console attached to the app. This shouldn’t be background information. This is data the app user needs to know, even if they don’t know they need it.

I wired up ipc to log how many PDFs were queued up twice in a second so the user knows what’s left to process and how fast the system is running through it. I put an indicator at the top left that’s green when there is no processing going on, red with a pulsing orange light when its busy. A quick glance should be good enough to let you know what exactly is going on.

Not quite the most beautiful swan, but this was a Sunday afternoon. Being feature complete at the end of the day was far more important than making it the most pretty thing. Enhancements could wait.

FEATURE COMPLETE

I left the Excel report til very late because I’d just worked on a Meteor app with Adim and a feature I worked on was exporting datasets to Excel. My understanding of the problem space and what it takes is still very fresh.

Generating the data wasn’t a problem, but the method of export is slightly different than it would be in a browser. Data URIs were not working in Electron, so I pushed the data back to the main process and wrote the file out with Node fs package.

Admittedly the workaround didn’t come to me immediately but I realized I’m not alone, searched and found a post online detailing the problem as well as a possible solution and figured out exactly how to do it in my app. Problem solved, feature complete.

FINAL DEMO

“Duuuuuuuude. This thing is awesome! You don’t know how much easier you just made my life. Thanks a lot man!”

Might be because I was expecting some pessimism but I was extremely pleased to hear how happy he was with it. He’s taken it to his office and is currently giving it out to his coworkers to use. I’m totally floored.

There’s only a handful of people using things I’ve made or libraries I’ve written, so every additional person enjoying what I’ve worked on brings a feeling of joy and motivates me to work harder to build more things.

Definitely going to be build more things and writing about it 🙂