–Geoff-Hart.com:–Editing, Writing, and Translation

—Home —Services —Books —Articles —Resources —Fiction —Contact me —Français

You are here: Resources --> 2002 --> Practical, effective metrics must solve problems
Vous êtes ici : Ressources --> 2002 --> Practical, effective metrics must solve problems

Practical, effective metrics must solve problems

by Geoff Hart

Previously published as: Hart, G. 2002. Practical, effective metrics must solve problems. DocQment, the newsletter of STC's Quality SIG. Sept. 2002. http://www.stcsig.org/quality/newsletters/NL0902/NL0902BASE.html

Writers regularly encounter requests to measure the quality or productivity of our work. The resulting “metrics” serve as performance benchmarks, but often neglect a more important benefit: collaboratively identifying problems that decrease performance so we can try to solve them.

Perils of performance monitoring

Managers must develop metrics cautiously because we humans flexibly change our behavior to deliver exactly what we’re asked to deliver. Metrics that target the wrong problem produce the wrong result; for example, asking writers to produce more pages per hour will generate more pages, but usually at the expense of quality. Other poor metrics abound, each illustrating how managers can demonstrate “on-paper” improvements rather than true improvements. Thus, metrics that measure performance are at best inefficient if they don’t simultaneously identify and help us solve problems.

Effective metrics must be objective (measurable), unbiased, and able to provide enough resolution (detail) to assess the factors that need improvement.

Objectivity

Objective metrics let any two assessors produce comparable estimates; in contrast, subjective assessments produce variable results because they depend on opinion, not fact, and even experts can change their minds. Metrics are easily quantified in terms of time or effort (e.g., rates expressed per unit time, successes per hundred tries). Such easily countable parameters objectively assess the work, in contrast with a poorly defined (subjective) criterion such as “the manager is satisfied the work is complete”. Some managers might reject work that others would accept because of different criteria for completion or might accept identical work 2 days later, when they’re in a better mood. Moreover, the metric fails to reveal problems and thus doesn’t suggest possible improvements.

Biases

Creating metrics is most difficult when a manager’s real goals are unclear. Fortunately, even managers who can’t define explicit goals can often provide examples of good and bad results that reveal a common thread in their decision process or that can answer pointed questions that reveal appropriate criteria. Understanding how they think lets you clarify the basis for a metric, since their purpose reveals important aspects of the problems to be solved.

Even seemingly objective metrics can be abused, as shown by a recent joke about figure skating: “The International Olympic Committee is introducing sets of skating action figures that come with Canadian, Chinese, and Russian dolls, plus a rink, but you must buy the judges separately. (Fortunately they're cheap.)”

Sarcasm aside, biases often overcome science when developing or applying metrics. It can be difficult to succeed if you don’t understand management’s political agendas and other biases that will affect their interpretation of the results. Managers may want to fire someone, support demands for more staff, or demonstrate improvements to their own managers. Each situation poses different challenges. For example, the first could involve you in getting a colleague fired; knowing this, you could propose one or more metrics that identify the problems the person faces, and by solving those problems, save their job.

Improved resolution

Even objective metrics aren’t effective if they don’t identify the problems to be solved. To increase the resolution and focus on real problems, develop separate metrics for each aspect of the work rather than a single, all-encompassing metric. Consider the metric “all online forms must be completed within 24 hours”. Each step might have its own metric:

Launching the software: With too few licenses, workers may have to wait to fill in a form or may monopolize the software to avoid delays, thereby exacerbating the problem for others.
Completing forms: If this step proceeds rapidly once the software is available, it isn’t the real problem.
Submitting forms: If slow computers delay this step, workers may wait until the end of the day to avoid disrupting their other work, forcing workers to compete for limited computer resources at that time.

Studying each step might have revealed that the problems lay in insufficient software licenses and slow computers, not the workers. Breaking one large metric (“total completion time”) into smaller metrics illuminated each phase of the process and thereby uncovered the real problems. Creating metrics for every factor that contributes to quality thus offers considerable power.

The metric system

The approach in this article works whenever you can identify biases (politics), the manager’s actual needs, and individual actions to assess. Better still, it lowers barriers between managers and employees by fostering a collaborative approach that helps managers alleviate the real problems their staff faces. Given that metrics are a fact of life, this suggests we should try to use them more effectively—not just to increase productivity, but to make the work easier and to improve quality.