Serially Iterating An Array, Asynchronously

Meet Derick Bailey. He runs Watch Me Code, has written a few programming books and blogs a fair bit, so he’s always on my radar when I need to learn a little bit more about programming.

A few months ago, he wrote a post, Serially Iterating An Array, Asynchronously, which was about a very specific programming issue he had to solve.

I recently found myself looking at an array of functions in JavaScript. I needed to iterate through this list, process each function in an asynchronous manner (providing a “done” callback method), and ensure that I did not move on to the next item in the list until the current one was completed.

Wait a minute … I faced that same problem a while ago!

My Node.js Express server routes have a list of steps they need to complete before sending out a response to the client.

The /signup route verifies the email address through an external service, bcrypts the password, stores the information in the database, retrieves the insert id and sends the information over to the client.

None of those operations are synchronous. I can’t run them in parallel either, since some operations depend on the result of a previous one. It’s pointless to continue if any step fails.

A serial list of functions that need to be processed asynchronously.

Derick posted his solution to the problem and where he felt it could be improved. My solution addresses quite a few of those, so I thought I’d share.

CJS-TASK

available on github and npm

 

  var task = require('cjs-task')(msg.notice.callback);

  task.set('email', msg.notice.email);
  task.set('password', msg.notice.password);

  task.step('validate-email', function(){

    var valid_email = require('email-validator').validate( task.get('email') );

    if(!valid_email){ 
      task.end({message: 'doesn\'t look like that email address is valid'});
    }

    else{ task.next(); }
  });

  task.step('create-passhash', function(){

    var bcrypt_helper = require('./utils/bcrypt-helper-functions.js');

    bcrypt_helper.hash(task.get('password'), function(err, hash){

      if(err){ 
  
        logger.log(err);
        task.end({message: 'could not create user. so sorry. please try again.'});
      }

      else {

        task.set('passhash', hash);
        task.next();
      }
    });
  });

  task.step('create-user', function(){

    var email, password, password_hash;

    email = task.get('email');
    password = task.get('password');
    password_hash = task.get('passhash');

    db_helper.query('INSERT INTO `user` (email, passhash) VALUES ("' + email +'", "'+ password_hash +'")', function(error, result){

      var response = {};

      if(error){

        response.message = 'couldn\'t create an account for ' + email;
        logger.log('[ERROR] CREATE ACCOUNT FAILED:\n' + email + '\n' + error);
        task.end( response );
      }

      else {

        task.end(null, result.insertId);
      }
    });
  });

  // START TASK
    task.start();

Couple points to highlight here.

1. task.end can be used to end a task prematurely. If any of your steps fails, use it to bring the task to a halt instead of wasting more time and resources.

2. task.next is a step control mechanism.

3. task.set /task.get can be used to store and return data relevant to the task at any step. Use it to store the initial data set, keep api responses, set flags … anything really.

4. Under the hood, task.end  nulls the data store, the task list and the callback list after the final callback has been triggered, in an attempt to prevent memory leaks.

5. Derick uses a destructive process on the task steps list. I simply keep track of the current index and increment after each task.next .

6. Under the hood I’m backing cjs-task with a pubsub, so I can trigger events for updates to the data store as well as when steps are triggered. Currently not implemented and probably not necessary, but I feel it’d add tremendous value and make it easy to monitor or modify your task instance. Most likely just an excuse to justify using the pubsub instead of something dead simple like a hashmap.

The longer I look at this, the more they look remarkably different, despite the similar API, job to be done and identical operation loop. Interesting.

Postscript.
Looks like someone just released queuer.js which looks like a hybrid approach. Combines the kind of event hooks I was looking to build into mine and Derick’s approach to handling data. 

Assembling a Practical JavaScript Toolkit

I was motivated by an opportunity to win N1,000,000 (around $5000) in a programming challenge this weekend.

The challenge was hosted on Codility’s platform. I’d never heard of their service before, let alone used it, so I really didn’t know what to expect going in.

On starting the timed test, the most notable constraint was my inability to import packages from github, npm or http. Maybe Codility lets you do it, but I didn’t have time to experiment and there was no clear way to do so in the interface.

No github. No npm. No libraries.

The thought of writing JavaScript without jQuery, Underscore or access to random libraries is very intimidating for many programmers.

It’s not every day you’ll experience such constraints, but you must be prepared to work under peculiar conditions. Your toolkit must be flexible enough that there’s hardly an environment you wouldn’t be productive in.

All the questions in the coding challenge dealt with manipulating collections of data and teasing out results from them. Interestingly enough, I’d written a blog post on Writing Reusable JavaScript a little while ago. The purpose of the reusable code I’d written? Iterating over collections of data and teasing out results from them.

Could I have done the same thing using jQuery, Underscore, Lodash or some other package out there? Yes.

However I wrote my own function to understand what’s going on behind the scenes much better. Because I did, I knew the solution a bit more intimately and could pare down my tool to the bare necessities or add needed bells and whistles on demand. I can perform these operations in JavaScript environments without those libraries or fancy new JavaScript language properties.

When building your code toolkit, don’t be so dependent on ideal environments. You may work in a code base that doesn’t use the lastest ecmascript features. You may not be allowed to use a transpiler. You may not even have access to packages.

You need to understand the principles of what your tool do and how to accomplish it with the barest of necessities: a text editor and your wits.

If you find yourself heavily dependent on something, try to spend a little time creating a dependency-free version of it: a version that you can copy and paste and use without relying on specific libraries environment properties.

When I fell in love with the ability to decouple code with publisher/subscriber pattern, I wrote my own pubsub. As my needs grew, I evolved my pubsub into cjs-noticeboard (pubsub with a noticeboard pattern). Today, my server-side and client-side code use cjs-noticeboard extensively.

Maybe I’m biased towards my own tools because I built them. I think what’s more important is that I understand exactly how they work and that I can take them anywhere. I can put them in IE6, a Meteor.js app or in a Codility challenge and know they’ll work exactly as expected.

You don’t have to build your own tools but you must understand how to carry them with you to all sorts of environments. Not every environ will be ideal, but you need to be prepared to kick ass no matter the conditions.

I’ll conclude with some of my Considerations for Assembling a Practical Javascript Toolkit.

1. Is this dependent on a specific version of JavaScript? (do I need a transpiler or shim to get this working everywhere?)

2. Is this dependent on a library? (Can this work without jQuery or Underscore or a specific framework?)

3. Is this single-purpose? (does it come with baggage I don’t need?)

4. Is it something I can make by myself? (do I understand the principles behind it or is it magic?)

Ps.
Oh yeah … if I win the money I’ll be sure to write more about it 😀

Deal With It

It’s rational to expect a solution to be big if the problem is big, right?

Sound rationale but inherrently flawed. Big things are simply a collection of little things working together. If one little thing stops working as expected, it could bring down the whole thing with it.

You would expect that someone who makes a living assembling little things together to make big things to understand this intuitively, but you would be wrong. So very wrong.

I wrote about a problem I was facing with an app I built. I didn’t have much time to dedicate to investigating and debugging the issue, and since it was non-critical I decided to ignore it til I had time to dedicate to the matter.

When I finally got around to tackling the problem, I was amazed to find out the solution was literally one line of code.

Continue reading Deal With It