Selecting metrics is difficult. It is difficult because you can end up influencing that which you measure.
People tend, not unreasonably, to do whatever presents them in the best light. Measure someone’s performance as the number of widgets they produce, and they produce more widgets, but without a governing quality measure you could reasonably expect quality to suffer. Add in a quality governed and the number if units produced will drop to that which people can produce and just maintain quality. Pay them more for exceeding the quality criterion and they will eventually reach an optimum point where they produce as many units as they can at a quality they believe optimizes their income.
Notice in this last sentence I said ‘that they believe optimizes their income’. This is because as the relationship between the measures and the outcomes becomes more complex, the ability of individuals to work out which combination of measures is optimal is reduced.
The same observation about the complex relationships between metrics and outcomes applies to those who set up metrication programmes. It is often difficult to foresee the outcome of collecting certain metrics. This problem is compounded when metrics are gathered by different parts of an organisation and with different goals.
It can be difficult too predict the consequence of collecting a particular metric, especially when there are many metrics interacting. This situation can be summarised as, ‘the rule of unintended consequences’.
The UK government has made something of a habit of creating metrics, and they have repeatedly fallen foul of the rule of unintended consequences. Schools, for example, are measured in many ways. One measure is used to place schools in a league table. This measure is apparently very simple and, on the face of it, makes perfect sense; the more students who pass their exams, the higher up the league table your school.
The consequence of this league table is simple. Schools now educate students to pass exams.
Nothing wrong with that, you might think. But do we want schools to educate students to pass exams, or make them more able to function in society?
In addition to the complexity of the rule of unintended consequences must be added the cost of collecting metrics. It is often overlooked when setting up metrics systems that every metric costs some resources to collect. It is even more common for people to overlook the costs of analysing those metrics.
The point of all this is that setting up a metrics programme is not a simple matter of picking something to measure. Measuring things cost, analysing data costs, measuring things can have unintended outcomes and measuring something can effect that which you measure.
