Two talks at local Software Development meetup: me and Pavel Knorr

Hello,

Both Pavel Knorr and myself, from Logicify, were having talks at local IT-meetup last Thursday. That was quite fun, thank to DataArt Kherson for organizing people and facility.

Thanks to Pavel for making up his excellent presentation with code samples, and to everyone working on our nodejs-sample application.

Here are the relevant links:

1. https://github.com/Logicify/nodejs-sample-task - source code of the sample app. We also use it as ‘seed’, or an ‘archetype’ for starting new apps of the same structure.
2. https://docs.google.com/presentation/d/1XW8suMG4hpzmccoKtvtLpytVgZLyVPZqZgrqCGnyNUY/present#slide=id.g121b56501_148 Russian language Slides for the Pavel’s talk about basics and caveats of Node.js. It would be best viewed with the accompanying recording of the talk, and if you have some background in client-side javascript and/or serverside programming (like java).
3. https://docs.google.com/presentation/d/1Osy7XQd9LuY7npicwK-r37cNOFUW0No3AwzEbEmRNqQ/present#slide=id.p Russian language Slides for my presentation on Node.js and unittesting. Would be best if the reader woud understand the concepts of the unit testing (mocks, verification, structure, goal of unittesting, level of granularity, usual terminology), thought that’s not really required.

Happy to help if you have questions!

Best,
Alex.

Node.js, elasticsearch, mognodb, ext.js and Heroku sample application - we have it as a work-in-progress

Hello!

We have built a sample application to use as a base (archetype-like) for the forementioned stack. It is accessible on GitHub and we really welcoming everyone trying to interact with it somehow. Push it, copy, use, fork, augment - great. Comments and questions? Totally wonderful. Complaints and ideas? Great as well! We look forward to improving it.

The idea of the app is boring - it’s CRUD+S(earch) app for bookshelf functionality. Nah, boring. Node.js, elasticsearch, mognodb, ext.js and Heroku sample application; it was written in 2 days by 4 people who have never used and even tried any of this tech before. This is a great illustration of quite a fast time-to-dev. The domain itself (books) is quite non-intrusive, which means you can play around with different components easily.

As of now, here’s what we have:

1. Source codes are accessible on https://github.com/Logicify/nodejs-sample-task
2. The app deployed to http://nodejs-sample-task.logicify.com/ and visible in the Internet
3. It has all the stuff necessary for Heroku deployment.
4. Exposes minimal architecture. From the point we’ve been working with this, we’ve came to way better way of handling the configuration, exceptions, logging, etc.
5. It has around half of the testing facilities (supertest, mocha, jscoverage, chaijs) setup, but is missing, for instance, nice usage of Sinon.
6. Has some basic documentation.

Upcoming plans:

1. Add ‘proper’ configuration handling (with several config files and env variable switching them)
2. Proper logger wrapper.
3. Improved testing - showcasing Sinon.js and some better assertions.
4. Growing a little more functionality.

This demo was partly created as a Proof of Concept we’ve been doing for one of our customers back in May, and partly as a demo app for 2 talks me and Pavel Knorr were having on the local Kherson it-talk meetup. Still has huge room for improvement, and that’s what we’ll try to do.

Also, it was created by Logicify team

[ADS MODE ON]
we are doing great custom software development and services. You can reach me for any enquiries or use the website :)
[ADS MODE OFF]

, and at Logicify we really have already used this project twice as an archetype - cloning it, fixing something in it and making it effectively a seed of the new project.

So, if you’d like to interact with it - please do!

Cheers,
Alex.

Node.js express middleware: beware of reading data before the bodyParser()!

Found that yesterday, as a part of implementation of HMAC-like functionality. So, what we did:

/*
THIS IS A CODE WHICH IS ____NOT____ WORKING! DON'T USE IT PLEASE!
*/
app.use(function (req, res, next) {
    var headerValue = req.header(secureAPIHeader);
    req.hasher = crypto.createHmac("sha256", config.secretAdminAPIKey);
    req.setEncoding('utf8');
    req.on('data', function (chunk) {
        req.hasher.update(chunk);
    });
    req.on('end', function () {
        var hash = req.hasher.digest('hex');
        if (hash != headerValue) {
            log.err(util.format("Received wrong Admin request, hashes do not match. Received hash %s want hash %s", headerValue, hash));
            res.json(403, {message: "Not authorized"});
        } else {
            log.warn("Received admin request for url " + req.url);
            req.isSuperSecure = true;
            next();
        }
    });
});

It exposed quite weird (not really) behavior. When the request with the wrong HMAC was coming in, the server gave away the correct 403 error. However, if the HMAC was correct, the server would never return from call chain.

Further investigation showed that it was quite simple. Since we have finished reading all the data from stream (and successfully received ‘end’ event), the underlying bodyParser was forever blocked on ‘end’ event which would never be issued! Actually, bodyParser was only subscribing to this event when the whole data bunch was read.

We didn’t really want to divert from streaming nature of the node.js crypto hasher and read all data into a huge string. So this middleware came into play:

/*
A correct version. The underlying data sourcers are not blocked.
*/
app.use(function (req, res, next) {
    var headerValue = req.header(secureAPIHeader);
    req.hasher = crypto.createHmac("sha256", config.secretAdminAPIKey);
    req.setEncoding('utf8');
    req.on('data', function (chunk) {
        req.hasher.update(chunk);
    });
    req.on('end', function () {
        var hash = req.hasher.digest('hex');
        if (hash != headerValue) {
            log.err(util.format("Received wrong Admin request, hashes do not match. Received hash %s want hash %s", headerValue, hash));
            res.json(403, {message: "Not authorized"});
        } else {
            log.warn("Received admin request for url " + req.url);
            req.isSuperSecure = true;
        }
    });
    // This is the only difference - we execute middlewares in parallel! 
    // => 
    next();
});

Trick here is that we pass execution to the underlying bodyParser straight away. And, when bodyParser receives data so do we - there are 2 listeners for data and end events here. There is an issue with this solution, though. We are not strongly coupling the moments when hmac becomes available in request, and the processing is done. bodyParser does this for us.

Moreover, got an extra-quick reaction by the express team, and now there’s a verify method (in pull request as of 2013-10-18), which does exactly what we do, but w/out streaming (for JSON only), which should be good for 90% users. Here’s discussion link: https://github.com/visionmedia/express/issues/897#issuecomment-26575496

Thanks!

JMeter not sending body with the PUT method

Noticed very, very strange behavior. When we issue a PUT request with jMeter’s HTTP Sampler, it totally ignores what we have in Post Data or in the Raw Post Data section.

That’s kind of crazy, since I would expect PUT method to be an HTTP mutator as well, and where’s the state if we don’t send it?

However HTTP is OK, it’s rather issues of the jMeter or underlying libraries. In order to make PUT method send the POST body, one could simply set the encoding to UTF-8 (that’s the box exactly after the method dropdown. And it works. A kind of magic, but ate another 30 mins of my time.

I had that issue on jMeter 2.9, and it seems to be quite common since that’s the link where I have found the solution.

Chaining calls with Underscore.js

I have came across underscore (http://underscorejs.org/) — a wonderful collection of javascript functions making one’s life easier — a while ago.

However, I have only recently found out that underscore.js has a beautiful interface of chaining. The samples below are quite self-descriptive for anyone familiar with javascipt. But anyway, idea is that with chaining, underscore would automatically pass the result of the previous operation to the next one.

That’s how the work looks w/out chaining:

var someData = [1,4,2,4,5,6];
var results = _.map(
   _.filter(
      _.sortBy(someData, _.identity),
      function(a) {return a>3}),
   function(m) {return m*m}
);
//outputs [16, 16, 25, 36]

and that’s how it looks like when we use method chaining:

var someData = [1,4,2,4,5,6];
var results = _.chain(someData)
   .sortBy(_.identity)
   .filter(function(a){return a>3})
   .map(function(m){return m*m})
   .value(); // that's the piece which 'executes' the chain
//outputs [16, 16, 25, 36]

You judge which one is better!

Domain name history search tool - found a non-paid one!

Hi there,

I was fighting the DNS system recently, and GoDaddy to name. Issue was that I was changing the nameservers for a given domain name, but for some reason GoDaddy disallowed to forward that domain. This worked out pretty well afterwards (just takes some time to wait).

However I made a mistake then. I have changed the NS servers, without writing down the old ones. When the change was done, I was unable to recover original IP of the service I was moving (oh sh!).

That was a moment of panic, however, after some searching I have come across (a finally free!) tool, which shows the history of the domain name change over time.

That’s http://whoisrequest.org/history/, and it works just perfectly. Using it, I was able to find out the historical NS records, and then query it for the old IP and it was relief :)

Setting up TeamCity coverage report for node.js test with mocha and jscoverage

Hi!

The mentioned tech stack has been popular for quite a while, leading to many people using it (especially in startups, for quick and robust API implementations, etc.). The question of testing is solved quite well (there is a number of testing, mocking, assert frameworks out there which I find astonishing).

This post is about quite a regular task on how to setup coverage metrics automatically on the build server (TeamCity is my favourite).

Disclaimer: I have a strong belief that coverage metrics should never be given to the business people w/out tech control. When they see _any_ metrics, they become excited with it and we have that overused. I believe, that coverage should be used by developers themselves to see what code paths have not been tested, and what tests can they add.


So, let's start. Prerequisites:

1. You have mocha tests
2. You have a working installation of TeamCity, which is already pulling the sources from repo, building app, running these tests on post-commit basis (and surely sends notifications to the people in charge).
3. You want to add coverage metrics to the party to see what's the progress and what areas need more help. 

Given that your project structure knows nothing of jscoverage, we'll go least-intrusive, scripting way. 

Jscoverage

https://npmjs.org/package/jscoverage

You need a specific jsCoverage build, which knows about node. You can have that installed, as usual, with npm:

 

npm install -g jscoverage

Verify the install, by issuing jscoverage on the commandline. The output is not very helpful, but gives the most interesting parameters you'll need. 

The way all coverage tools work is that either instrument the product of your code (or code itself, for scripting tools), OR plug into the interpreter, and when you apply some interaction to this code (tests, users coming in and out, etc), collect how many times each of the lines was called. After these numbers are obtained, static analysis may plug in to verify how many functions/branches/conditions were covered. 

So, we need to obtain such an 'instrumented' code first from our target code. That's what jscoverage doing:

jscoverage --exclude tests,node_modules,.git,target,.idea  CURRENT_FOLDER OUTPUT_FOLDER

After that you can examine the results in the output folder. All your files should be copied there (except the ones included), and they should have some weird code inside (that's the recording part!), with pieces and bits of your code scattered across the file. 

mocha

As you are already probably using mocha for testing, nothing is really tough here. I assume that you are loading your modules-under-test with the require('../../lib/something'), rather than via absolute paths. 

As we already have a specific folder, which contains jscoverage-processed main app code, w/out all the unnecessary folders, we could now just copy out the tests themselves to that folder, and run mocha. No results yet, we just verify that mocha tests run with our processed code. 

Next, we are good to go to add some html-cov reporting. This reporter is already bundled with mocha, so no need to reference it in package.json. what it does, is that after the mocha tests are ran, it gets the coverage statistics from coverage-augmented files under test, and flushes that (in form of HTML) to the stdout. So, to see what's in the coverage, we do (whilst in the OUTPUT_FOLDER): 

../.././node_modules/mocha/bin/mocha -t 20000 --noinject --reporter html-cov --recursive ./tests/ --coverage

--noinject is required for me, as I am using rewire library for testing, and that somehow clashes with the jscoverage stuff.

You could divert that output to file, or just pipe to w3m.

TeamCity

Quite an easy part now; since we have all working locally on the server, we just need to automate it. I have finally used a html-file-cov mocha reporter, it prints out the tests results as dots to the stdout, and writes the coverage report to coverage.html within working folder; quite an easy and handy tool.  https://npmjs.org/package/html-file-cov

In teamcity, we need to add another build step to the post-commit configuration. I did the easy and ugly script way instead of built-in report processing, since I was quite happy with the html format. 

So, as we have added that build step (of type Custom script), we could now fill it with the actual logic: 

#!/bin/bash

rm -rf ../target-cov/*

rm -rf ./target/target-cov

rm -rf ./target/coverage.html

mkdir -p ../target-cov

jscoverage --exclude tests,node_modules,.git,target,.idea . ../target-cov

mv ../target-cov ./target

cp -r ./tests ./target/target-cov

echo "JS files instrumented correctly"

cd target/target-cov

echo "Running mocha tests with 'dot' reporter"

NODE_ENV=dev ../.././node_modules/mocha/bin/mocha -t 20000 --noinject --reporter html-file-cov --recursive ./tests/ --coverage

mv ./coverage.html ..

echo "Done. Configuration stored to `pwd`/../coverage.html"

As a result of this manipulation (making a dir one level up, storing there the jscoverage-processed files, copying it to the local 'target' folder, copying over the tests non-processed, running the tests, moving the coverage.html to the target dir), we have the artifact - coverage.html - which contains all the stuff we need. 

We just need to publish it - in TeamCity project settings, on the very first page, there's an 'Artifacts' box, which takes paths. Just add +:target/coverage.html, and after each build you'll have a nice drop-down 'Artifacts' with a link to your html with coverage. 

Voila!