IT is part of the business

There is a mantra that resonates strongly: “What cannot be measured cannot be improved”. This phrase, attributed to Peter Drucker, one of the fathers of modern business management, contains a fundamental truth that is still relevant today. On the other hand, measurement alone is not enough; it is important to understand how to measure, what to measure and, above all, how to interpret and act on those measured data, otherwise we may even obtain a result contrary to the intended one.

Of course, we cannot say that measuring is bad, but we can say that it can be a double-edged sword. The implementation of metrics and KPIs has become a standard practice in almost all organizations. From startups to established multinationals, data collection and analysis is a constant activity. However, it is critical to recognize that measurement, if misapplied, can be counterproductive.

The danger of misguided metrics

When metrics become the end instead of the means, you run the risk of falling into what is known as “Goodhart’s Law”: “When a metric becomes a goal, it ceases to be a good metric”. This phenomenon is observed when teams begin to “game the system”, adjusting their behaviors not to improve the system, but to improve the numbers; to look good in the photo. A clear example of this can be seen in incident resolution times. If a customer service team is measured only by the speed of ticket resolution, it could lead to superficial solutions that do not address the root problems. This will cause the quality of service to drop, even if the numbers come out right. Similarly, if a software development team is evaluated solely by the number of lines of code produced, it is likely to generate unnecessarily complex or redundant code.

Transparency as a fundamental pillar

One of the keys to avoid these problems is transparency. Metrics and results must be visible to all members of the organization. This visibility serves several purposes:

• Goal alignment: When everyone can see where the company is headed, it is easier to align individual efforts with overall goals. Key to this is that the company itself knows where it wants to go, which is not always the case.
• Encouraging collaboration: Transparency allows different teams to identify areas of synergy and opportunities for collaboration. Something that is very useful is to have measures that impact two or more teams, so collaboration will be more fluid.
• Accountability: Visibility of results fosters a sense of shared responsibility and motivation to improve. Teams should be aware that their work is reflected in certain metrics.
• Early detection of problems: When data is visible to all, worrying trends or areas for improvement are more likely to be identified before they become major problems.

I remember reading about a case in the Netherlands where they analyzed the electricity consumption of a development of houses, all of which were the same. During the analysis they were able to see that the consumptions could be clearly grouped into two blocks, those that consumed the most and those that consumed the least. Further analysis led to the conclusion. The difference between the two groups was not that they had more or less children or people in the house, it was something simpler. We have said that all the houses were the same, but they were not. One group of houses had the electricity meter in the basement and another had it in the entrance hall. The first group spent the most. The second group, having the meter in sight, was more aware of their spending and reduced their consumption.

Measure what really matters

The true art of measurement lies in identifying which metrics are truly meaningful to the success of the organization. This requires a holistic approach that considers not only the end results, but also the processes and activities that lead to those results.

In this sense, we may find it necessary to combine process metrics (which measure how things are done) and outcome metrics (which measure what is achieved). For example, if we think of software development, in addition to measuring the number of features delivered (result), it is important to measure code quality, user satisfaction and iteration speed (process). Otherwise we may have counterproductive results in the future. Suppose we focus only on the number of features delivered. At the beginning it may go very well, but if we do not take into account the quality of the code, it will deteriorate and it will be more and more difficult to deliver new features, but by then, it will be very difficult to reverse the consequences of having delivered features without the necessary quality. The KPI that we thought was helping, in the end has become the problem.

It is said that “the devil is in the detail”. If measurements are made to check compliance with a standard, we must also be very careful about how to make them. A clear example is speeds. In Spain, the maximum speed on freeways is 120 km/h. The reality is that the speed is much higher and is reduced by passing through the controls, by the radars. Thus, it may seem that everything is correct (the measurement is as expected), but outside the measurement something different happens. The reality escapes from the measurement.

Quantitative metrics

When it comes to metrics based on quantified values, we can adopt different approaches. Two of the two most commonly used frameworks for setting objectives, defining relevant metrics and measuring progress towards those objectives are GSM (Goal, Signal, Metric) and GQM (Goal Question Metric).

Goal, Signal, Metric (GSM)

1. Goal: Define a specific, measurable, achievable, relevant, and time-bound (SMART) goal.
2. Signal: Identify indicators that signal progress towards the goal. This can be information on activities or results..
3. Metric: Establish quantitative metrics to measure the signals, ensuring data availability and reliability.

Goal Question Metric (GQM)

1. Goal: Define a specific, measurable, achievable, relevant, and time-bound (SMART) goal.
2. Question: Ask questions whose answers indicate whether the desired objective is being achieved. These questions should be specific, measurable and relevant.
3. Metric: Develop quantitative metrics to answer the questions, ensuring data availability and reliability.

Although both approaches share similarities, the GQM is more comprehensive and more widely applied. GSM focuses on signals and metrics, and can be used as a simplified and more tactical approach. On the other hand, the GQM incorporates questions to ensure a more comprehensive approach being more suitable for a more strategic framework.

Qualitative metrics

Not everything that is important can be easily quantified. Aspects such as employee satisfaction, organizational culture or innovation are important and should be taken into account, but difficult to measure with numbers. This is where qualitative metrics, such as satisfaction surveys, feedback sessions and case studies, play a vital role.
In general, we can say that while quantitative metrics provide concrete numerical data, qualitative metrics offer deeper insights into perceptions, experiences and satisfaction.

For example, if we want to know how well IT projects and services fit together to support the company’s strategic goals, we can measure this through quarterly sessions with business leaders. Using a scale from “Fully aligned” to “Not aligned” and documenting specific examples of how IT is positively or negatively impacting business goals could help us.

Effective measurement system

There are several aspects to take into account to have an effective and usable measurement system in the short and long term, some of them are:

• Each metric must have a clear connection to the organization’s strategic objectives, either directly or indirectly.
• Seek a balance between short- and long-term metrics, as well as between financial and non-financial indicators.
• Numbers alone do not tell the whole story. It is crucial to provide context and analysis along with the data.
• Metrics should be reviewed regularly to ensure they remain relevant and are not generating undesirable behaviors.
• Metrics should be a starting point for constructive conversations, not a final judgment.
• Make transparent why these metrics are being performed, what their purpose is, as well as how often they are calculated and their method of calculation and application.
• Automate as much as possible the whole process of data collection and KPI elaboration and presentation. We will avoid human errors and tedious tasks.
• Be constant in the calculation and analysis of indicators.
• Do not cheat alone. If a measurement is not as expected, analyze the cause of why it is so. Do not try to cheat the system or simply ignore it.
• Make measurements and results public. Make everyone aware of them. Remember the case of the Dutch electricity meters. No matter how much you measure, if you don’t see the metrics, you are not aware of them.

The purpose of measuring is not simply to collect data or meet arbitrary targets. The purpose should be to foster continuous improvement, adaptation and awareness of what is happening.
You should not only measure to validate what you already know, but also to discover new opportunities, identify areas for improvement and ultimately drive growth.
However, we must never forget that behind every number, every graph and every KPI, there are processes, decisions and above all people, who bring those data to life. A very interesting book on this subject is “Measure what matters” by John Doerr.