Two talks at local Software Development meetup: me and Pavel Knorr

Hello,

Both Pavel Knorr and myself, from Logicify, were having talks at local IT-meetup last Thursday. That was quite fun, thank to DataArt Kherson for organizing people and facility.

Thanks to Pavel for making up his excellent presentation with code samples, and to everyone working on our nodejs-sample application.

Here are the relevant links:

1. https://github.com/Logicify/nodejs-sample-task - source code of the sample app. We also use it as ‘seed’, or an ‘archetype’ for starting new apps of the same structure.
2. https://docs.google.com/presentation/d/1XW8suMG4hpzmccoKtvtLpytVgZLyVPZqZgrqCGnyNUY/present#slide=id.g121b56501_148 Russian language Slides for the Pavel’s talk about basics and caveats of Node.js. It would be best viewed with the accompanying recording of the talk, and if you have some background in client-side javascript and/or serverside programming (like java).
3. https://docs.google.com/presentation/d/1Osy7XQd9LuY7npicwK-r37cNOFUW0No3AwzEbEmRNqQ/present#slide=id.p Russian language Slides for my presentation on Node.js and unittesting. Would be best if the reader woud understand the concepts of the unit testing (mocks, verification, structure, goal of unittesting, level of granularity, usual terminology), thought that’s not really required.

Happy to help if you have questions!

Best,
Alex.

Node.js, elasticsearch, mognodb, ext.js and Heroku sample application - we have it as a work-in-progress

Hello!

We have built a sample application to use as a base (archetype-like) for the forementioned stack. It is accessible on GitHub and we really welcoming everyone trying to interact with it somehow. Push it, copy, use, fork, augment - great. Comments and questions? Totally wonderful. Complaints and ideas? Great as well! We look forward to improving it.

The idea of the app is boring - it’s CRUD+S(earch) app for bookshelf functionality. Nah, boring. Node.js, elasticsearch, mognodb, ext.js and Heroku sample application; it was written in 2 days by 4 people who have never used and even tried any of this tech before. This is a great illustration of quite a fast time-to-dev. The domain itself (books) is quite non-intrusive, which means you can play around with different components easily.

As of now, here’s what we have:

1. Source codes are accessible on https://github.com/Logicify/nodejs-sample-task
2. The app deployed to http://nodejs-sample-task.logicify.com/ and visible in the Internet
3. It has all the stuff necessary for Heroku deployment.
4. Exposes minimal architecture. From the point we’ve been working with this, we’ve came to way better way of handling the configuration, exceptions, logging, etc.
5. It has around half of the testing facilities (supertest, mocha, jscoverage, chaijs) setup, but is missing, for instance, nice usage of Sinon.
6. Has some basic documentation.

Upcoming plans:

1. Add ‘proper’ configuration handling (with several config files and env variable switching them)
2. Proper logger wrapper.
3. Improved testing - showcasing Sinon.js and some better assertions.
4. Growing a little more functionality.

This demo was partly created as a Proof of Concept we’ve been doing for one of our customers back in May, and partly as a demo app for 2 talks me and Pavel Knorr were having on the local Kherson it-talk meetup. Still has huge room for improvement, and that’s what we’ll try to do.

Also, it was created by Logicify team

[ADS MODE ON]
we are doing great custom software development and services. You can reach me for any enquiries or use the website :)
[ADS MODE OFF]

, and at Logicify we really have already used this project twice as an archetype - cloning it, fixing something in it and making it effectively a seed of the new project.

So, if you’d like to interact with it - please do!

Cheers,
Alex.

Node.js express middleware: beware of reading data before the bodyParser()!

Found that yesterday, as a part of implementation of HMAC-like functionality. So, what we did:

/*
THIS IS A CODE WHICH IS ____NOT____ WORKING! DON'T USE IT PLEASE!
*/
app.use(function (req, res, next) {
    var headerValue = req.header(secureAPIHeader);
    req.hasher = crypto.createHmac("sha256", config.secretAdminAPIKey);
    req.setEncoding('utf8');
    req.on('data', function (chunk) {
        req.hasher.update(chunk);
    });
    req.on('end', function () {
        var hash = req.hasher.digest('hex');
        if (hash != headerValue) {
            log.err(util.format("Received wrong Admin request, hashes do not match. Received hash %s want hash %s", headerValue, hash));
            res.json(403, {message: "Not authorized"});
        } else {
            log.warn("Received admin request for url " + req.url);
            req.isSuperSecure = true;
            next();
        }
    });
});

It exposed quite weird (not really) behavior. When the request with the wrong HMAC was coming in, the server gave away the correct 403 error. However, if the HMAC was correct, the server would never return from call chain.

Further investigation showed that it was quite simple. Since we have finished reading all the data from stream (and successfully received ‘end’ event), the underlying bodyParser was forever blocked on ‘end’ event which would never be issued! Actually, bodyParser was only subscribing to this event when the whole data bunch was read.

We didn’t really want to divert from streaming nature of the node.js crypto hasher and read all data into a huge string. So this middleware came into play:

/*
A correct version. The underlying data sourcers are not blocked.
*/
app.use(function (req, res, next) {
    var headerValue = req.header(secureAPIHeader);
    req.hasher = crypto.createHmac("sha256", config.secretAdminAPIKey);
    req.setEncoding('utf8');
    req.on('data', function (chunk) {
        req.hasher.update(chunk);
    });
    req.on('end', function () {
        var hash = req.hasher.digest('hex');
        if (hash != headerValue) {
            log.err(util.format("Received wrong Admin request, hashes do not match. Received hash %s want hash %s", headerValue, hash));
            res.json(403, {message: "Not authorized"});
        } else {
            log.warn("Received admin request for url " + req.url);
            req.isSuperSecure = true;
        }
    });
    // This is the only difference - we execute middlewares in parallel! 
    // => 
    next();
});

Trick here is that we pass execution to the underlying bodyParser straight away. And, when bodyParser receives data so do we - there are 2 listeners for data and end events here. There is an issue with this solution, though. We are not strongly coupling the moments when hmac becomes available in request, and the processing is done. bodyParser does this for us.

Moreover, got an extra-quick reaction by the express team, and now there’s a verify method (in pull request as of 2013-10-18), which does exactly what we do, but w/out streaming (for JSON only), which should be good for 90% users. Here’s discussion link: https://github.com/visionmedia/express/issues/897#issuecomment-26575496

Thanks!

JMeter not sending body with the PUT method

Noticed very, very strange behavior. When we issue a PUT request with jMeter’s HTTP Sampler, it totally ignores what we have in Post Data or in the Raw Post Data section.

That’s kind of crazy, since I would expect PUT method to be an HTTP mutator as well, and where’s the state if we don’t send it?

However HTTP is OK, it’s rather issues of the jMeter or underlying libraries. In order to make PUT method send the POST body, one could simply set the encoding to UTF-8 (that’s the box exactly after the method dropdown. And it works. A kind of magic, but ate another 30 mins of my time.

I had that issue on jMeter 2.9, and it seems to be quite common since that’s the link where I have found the solution.

Chaining calls with Underscore.js

I have came across underscore (http://underscorejs.org/) — a wonderful collection of javascript functions making one’s life easier — a while ago.

However, I have only recently found out that underscore.js has a beautiful interface of chaining. The samples below are quite self-descriptive for anyone familiar with javascipt. But anyway, idea is that with chaining, underscore would automatically pass the result of the previous operation to the next one.

That’s how the work looks w/out chaining:

var someData = [1,4,2,4,5,6];
var results = _.map(
   _.filter(
      _.sortBy(someData, _.identity),
      function(a) {return a>3}),
   function(m) {return m*m}
);
//outputs [16, 16, 25, 36]

and that’s how it looks like when we use method chaining:

var someData = [1,4,2,4,5,6];
var results = _.chain(someData)
   .sortBy(_.identity)
   .filter(function(a){return a>3})
   .map(function(m){return m*m})
   .value(); // that's the piece which 'executes' the chain
//outputs [16, 16, 25, 36]

You judge which one is better!