Practical and effective metrics

By Geoff Hart

Previously published as: Hart, G.J. 2004. Practical and effective metrics. Intercom February:6ñ8.

Anyone who hangs out in the technical communication community regularly encounters the problem of measuring performance, quality, usability, or some other parameter intended to describe how good a job we’re doing. Collectively, these parameters are known as metrics, derived from the Greek term metrikos, which means "relating to measurement". The primary goal of measuring, of course, is to create a standard against which something can be judged: that is to say, how does the result compare to the work used to create the standard? Does the quality of a product measure up? What’s often forgotten is that metrics can be used not only to measure performance, but also to identify specific problems that are affecting performance.

In this article, I focus on metrics developed by writers and editors in response to management requests, but managers can apply the same reasoning to developing their own metrics.

The perils of performance monitoring

Let me start with two warnings. First, long before Heisenberg developed his uncertainty principle, it was well known that the act of measuring influenced the system being measured. Second, measuring serves little purpose if it provides no means for improvement. Let’s look at both points in turn.

If you must develop metrics, be wary about the statistics you collect. The metrics communicators most often encounter are designed to describe human behavior, and human behavior changes readily. Flexibility in behavior is a good thing when your goal is to encourage improvement. But it also poses a problem: Most of us are more than smart enough to deliver precisely what we’re asked to deliver when we know a metric will be used to judge our work. The classic example of a metric that targets the wrong problem is the one that specifies writing a minimum number of pages per hour. We all know that it’s easier to explain even a simple concept in 20 words than in 10, so that metric will have the unintended consequence of longer documents. Unfortunately, that's not what our managers and our clients really need.

Although this particular example may be trivial, discussions on the techwr-l discussion group (www.techwr-l.com) suggest that this kind of simplistic approach to measuring performance is quite common. In addition, it illustrates the larger problem: When you develop a metric, you define which aspects of quality everyone will work to improve. That means you’d better be defining the right aspects.

Equally important, if you’re going to go to all the effort to develop a metric and measure whether everyone is living up to a standard, you should do more than just measure against a benchmark. A truly useful approach should also identify the problems that lead to failures to meet performance targets. Identifying and solving problems changes the focus from monitoring performance, which inherently involves criticism of those who don’t measure up, to helping workers meet productivity or quality targets. If that’s honestly the primary goal behind developing metrics, you’re more likely to get the results you want (true improvements) because everyone is more likely to accept the approach.

Effective metrics must be objective (measurable), unbiased, and able to provide enough resolution (detail) to assess the factors that need improvement.

Objectivity

Objective and measurable are deceptively simple words. In practical terms, both mean that any two people who set out to calculate the value of a metric must be able to produce comparable results. Subjective metrics are hard to measure because their value depends as much on opinion as on demonstrable fact. The problem with opinion is twofold: Even two experts are likely to disagree on an issue to some extent, and even a single expert can produce different assessments on different days. A truly objective metric is less likely to prompt disagreement between experts, and is more likely to produce consistent results.

Certain metrics are surprisingly easy to quantify. The two most common types of metric involve rates (performance per unit time for productivity or per hundred tries for success) or quantities (something that can be counted). Two good examples of objective performance criteria are the following:

Time: All paperwork must be completed within 24 hours of the final program meeting.
Quantity: Editors will find no spelling or grammatical errors in the document.

These criteria are objectively measurable if there's agreement on what constitutes a grammatical error, and are relevant if they relate directly to the execution and quality of the resultant job. Compare them with two poorly defined criteria:

The department manager is satisfied that the work is complete.
Fewer problems were reported today than yesterday.

In contrast to the first two examples, these are highly subjective. A picky manager might reject work that a laissez-faire manager would accept—perhaps because they have different criteria. Or one manager might reject work today that they would accept 2 days later, when feeling less judgmental, stressed, or rushed. Moreover, neither criterion relates to the work in such a way that you can define problems and suggest improvements.

Biases

Creating metrics can be difficult. You may have to spend considerable time working with the managers who asked you to develop a metric to understand what they really want. Sometimes they can’t define their requirements explicitly, and you’ll have to ask for examples of what they consider good and bad outcomes. With persistence, examining these examples will reveal commonalities in how the managers made their decisions—even if the decisions themselves appear subjective—or will let you ask increasingly pointed questions until you discover the true criteria. Understanding the decision process lets you determine the elements you should use to develop an effective metric.

I’ve made the unusual suggestion that you try to learn the reasons behind the development of a metric for two reasons. First, this is the only way to understand the problem that people are trying to solve. Second, any metric can be abused if someone devotes enough effort to doing so, as fans of Olympic figure skating can attest. Although there are clear guidelines by which skaters should be judged, everyone knows that these guidelines are not universally respected in the judging community. Politics often overrides science when it comes to metrics.

You must work directly with the managers who will be assessing the metrics. In identifying their criteria for success, you should also identify and understand any political agendas or other biases that affected the development of the metrics and interpretation of the results. In an extreme case, for example, managers might be looking for an excuse to fire someone; in other cases, they may be trying to use a productivity metric to support a request for more staff or to demonstrate improvements in their department’s performance to their own managers. In each case, you face a different challenge. For example, in the first case you face the ethical dilemma of becoming responsible for getting someone fired if you develop a metric designed to prove that they are performing poorly. You could instead try developing a metric that identifies the problems that person faces and proposes a means of solving these problems. In doing so, you might thereby satisfy the manager that you know where the problem lies and what you can do to improve the situation—and thus save that person's job.

Resolution

How can you identify the problems that underlie a metric? By creating multiple metrics that assess the various aspects of what you’re measuring rather than striving for a single, all-encompassing metric. This is where the greatest opportunities for improvement lie. For example, consider the previous example of all paperwork being completed via online forms within 24 hours. If this target isn’t being met, managers might suspect that the employees responsible for this work are the source of the problem, and ask you to develop a metric to prove whether this is the case. Simplistically, the process of completing paperwork might have three steps, each of which may pose certain problems. For example:

Waiting for the software to launch: When you have only a limited number of software licenses, workers might have to wait hours before they can even open a copy of the software and begin filling in the forms. In response, they might be tempted to leave the software open on their computer so that it’s constantly available for their use, even if monopolizing the software creates problems for other workers.
Filling in the forms: Studying the work reveals that this step takes only a few minutes once the software is available. So filling in the forms is not a problem at all, despite the original perception that motivated the development of the metrics.
Submitting the forms: A slow server might delay delivery of the form, and if a worker’s computer is tied up for 15 minutes per form waiting for the form to be displayed and the completed form to be processed, employees may wait until the end of the day to complete the forms. They thereby ensure that their other work isn’t disrupted, but as a consequence, everyone ends up vying for the same limited computer resources at the end of the day.

A simple metric with low resolution (the proportion of the paperwork completed within a single day or the total time to complete the paperwork) would not have revealed any of these problems. But by breaking one large metric into separate metrics that describe the individual steps, you discover that the overall problem doesn’t lie with the employees at all. If the goal is faster completion of forms, the solution in this case would be to invest in more software licenses and faster computers. Once you know which parts of a process work well and which ones don't, you can take measures to correct the problems. You can apply this same analytical approach to any process in which quality is important: Once you understand all the steps in the process that contribute to quality (or lack thereof), you can investigate each using a separate metric and identify the problems with each step.

The merits of the metric system

The approach in this article works whenever you can identify biases (such as office politics), the manager’s actual needs, and individual actions to assess. The nicest thing about this approach is that it’s easy to adapt to just about any situation, provided that you can work with managers to identify their true needs and can work with those whose performance will be measured to identify the individual actions you must assess.

An additional benefit of this approach is that it can break down walls between performance-oriented managers and their employees by fostering a collaborative approach to identifying and solving problems. This helps to teach the managers what their staff actually do so they can make it easier for the workers to do their jobs, improving both productivity and quality. Worker satisfaction also improves, since they are being helped to do their jobs better rather than being judged.

Metrics are a fact of life in the modern, fast-paced workplace, and understanding how to use them correctly can turn the potential unpleasantness of performance evaluations into an opportunity to make life easier for everyone. The fact that you'll improve productivity and quality is a bonus from adopting this human-centered approach.