Tuesday, August 18, 2009

Metrics - good or bad?

Outside world example
When participating in software projects, people like to speak about metrics.
This is caused by the nature of the software projects. Program develipment oftenly turns to the research task.

You can easily track progress for the bridge construction. It's very easy. You can do 2 things:
  • Get financial documents and reports and see how it's going.
  • Go to the construction place and see how it works.
  • Ask constructors for report. Constructors would tell you, that 'Bridge is complete for 23.56%, first phase is done for 98%, seconds phase - for 33%'
  • Also - there are some numbers about quailty. For example, 4% of weldings need to be redone, 1.23% of surface paintings fails, etc...
Business people are happy. They have numbers. Using these numbers they can see progress, make plans and feel them pushing the thing.

Metrics in software projects - good or bad?
What about software projects?
You can not go to the construction place. You can ask programmers about 'How it's going?', and you'll get the answer 'Doing ok.'

You need numbers to show to business people. Business people will become upset very soon if you will tell them for a month that 'Progress is good!'

Also, having numbers stimulates programmers.
Here are several examples:
  • Code coverage. Percentage of code lines 'covered' with unit tests. Covered means here that when tests execute, this line is reached by execution.
  • Commit percentage. Mr. Alex is committing (putting code to repository) 3 more times than mr. Andrew. We need to punish Andrew!
  • Usually, work breakdown exists in form of tickets - bugs, feature requests, development tasks. People LOVE having metrics on these. For example - 299 work items done, 1020 left for work. Or - this milestone is completed for 50% - 144 of 288 tickets closed.
  • Easiest - count of the code lines produced by a developer.
The Main Thought
Metrics can do both things - indicat process, or hide it. As a business person, you can get positive reports during 2 months, showing that codebase grows speeds up and speeds up. But after those 2 month you'll face real f**ck-up.

Metric should be sane and wise.

You would not decide hospital to go to basing on the average temperature among all patients, would you?


Dmitriy Nagirnyak said...

I sometimes use Use-Case based (story-based) "metric" to measure the overall progress of a project. This gives overall good indication of a project status.
It represents high level status which is suitable for reporting purposes. I can easily tell: the project is done for 50% as we have 200 use-cases implemented out 400 planned ones.

Additionally I should state that code metrics are rarely good status indicator for a project, but is rather good as quality indicator and is the project status for the development team.

In a long run the better results of code metrics are the better overall project status is going to be.

But again, the code metrics can rarely be used to measure the real status of a project at a high level.

Alexander said...

Yeah, UC completion metric is better for businessmen than the lines-of-code metric :)

But - if you have 400 planned use cases, how do you know, which of them are of which value? This may need addition of some coefficient (customer-value?), or 'weight' of another nature to each of the completed use cases.

Well, I totally agree that having 95% covered with unit-tests is good. Even more, if you see that 20% is covered with test, you'll be stimulating yourself for the better results, better coverage, etc.

The main idea I wanted to bring - don't let numbers hide from you the real situation. It's pretty easy - when the project fails, you could always say 'but we had 100% coverage! and good code style!'.

I'm for metrics. At least some :)