(False) precision: Part 1: Precision and Accuracy

Previously published as: Hart, G. 2025. (False) precision: Part 1: Precision and Accuracy. https://www.worldts.com/english-writing/eigo-ronbun90/index.html         

Scientists often incorrectly believe that if one decimal place of precision is good, then three decimal places must be better. Although this is true in contexts where high precision is essential and you were able to obtain highly accurate measurements, it’s not true in many other contexts. Indeed, in some cases, presenting many decimal places may actually be misleading. In part 1 of this article, I’ll discuss several aspects of precision and their implications for how you present your data. In part 2, I’ll discuss how these guidelines affect statistical probability.

Note: This article focuses on the results of calculations. Raw data should always be recorded using the full measurement precision of the instrument that you used to obtain the measurements.

Precision and accuracy

First, I’ll start with two definitions to ensure that we’re both using the same terminology. Precision is the ability to repeat a measurement several times and obtain approximately the same result each time; that is, precise results will have a low standard deviation. In contrast, accuracy means that the measurement is correct; that is, there will be a small difference from the actual value, even if there is noticeable variation among consecutive measurements. For example, an accurate value has a low root-mean-square error. Ideal measurements are both precise and accurate. In this article, I will consider both the strict meaning of precision in the context of statistics and the common (but imprecise!) use of the word based on the belief that reporting as many digits as possible for each number represents precision or accuracy.

Consider the example of measuring the thickness of a 1.00-mm-thick leaf using a device that measures thickness to the nearest 0.10 mm:

• A mean value of 1.20 mm ± 0.01 mm (mean ± SD) is inaccurate (it is 20% higher than the actual value) but precise (it has a standard deviation of about 1% of the actual value).

• A mean value of 1.00 mm ± 0.20 mm is accurate (it exactly equals the actual value) but imprecise (it has a standard deviation equivalent to 20% of the actual value).

• A mean value of 1.00 mm ± 0.01 mm is both accurate and precise because the mean value is correct and the standard deviation is low (1% of the actual value).

Precision can be communicated by the number of decimal places in a measurement if those decimal places can actually be measured. If you use a device that can measure a leaf’s thickness to three decimal places, then a mean value such as 1.001 mm is meaningful for consecutive measurements of 1.000, 1.001, and 1.002 mm. Conversely, if you can only accurately measure that thickness to one decimal place, a mean thickness of 1.1 mm is meaningful, but a value of 1.001 mm is meaningless for measurements of 0.8, 1.0, and 1.3 mm. Differences in the first decimal place (values from 0.000 to 0.900 units) overwhelm any values in the third decimal place (from 0.000 to 0.001 units); they are 2 orders of magnitude larger. In this context, reporting values to three decimal places gives a false sense that the third decimal place reveals important differences between your measurements.

Note: Here’s another example: Although the value of the constant π (which equals the ratio of a circle’s circumference to its diameter) can be calculated to any desired number of digits, providing that full “precision” isn’t meaningful in most contexts. Sometimes 3.14 is sufficient; sometimes 3.1415926 will be more suitable.

Significant figures or digits

My description of precision provides an example of why more decimal places isn’t necessarily better. The reason behind this is the mathematical concept of “significant figures”. This relies on the concept that except in specific cases, it’s not plausible to report results to more decimal places than the resolution of the instrument. For example, if the leaf in our example really is 1.0 mm thick, but your instrument can only measure in increments of 1.0 mm, you cannot reliably distinguish between leaves that are 0.9, 1.0, and 1.1 mm thick. Achieving that distinction would require measurements that are reliable for increments of 0.1 mm. These measurements of 1.0 and 0.1 mm, respectively, are significant figures because they include only digits that can be directly and reliably measured.

Note that you will often see a zero added at the right end of a decimal (e.g., 1.0 rather than 1) when numbers are being compared, particularly in a table. This is done for two main reasons. First, if all other numbers are measured reliably to the same number of decimal places, there is no risk of misinterpretation if you add that final zero so that they are all presented with the same number of decimal places. Second, this process aligns numbers when they are being presented in a vertical column in a table, which makes it easier to compare numbers visually. (In my work as a scientific editor, I frequently correct an author’s incorrect interpretation of their own data that resulted from misreading a column of numbers with different numbers of decimal places.)

The rules for determining the appropriate number of significant figures are complex (https://en.wikipedia.org/wiki/Significant_figures). However, if you’re calculating summary statistics such as a mean or median value, a good rule of thumb is that it’s acceptable to provide one digit more than the measurement resolution. Most journals and their reviewers will generally accept this. For example, if you obtain measurements to a resolution of 1 mm (i.e., no decimal places), most readers will accept a mean calculated to one decimal place. Thus, the mean of measured values equal to 1 and 2 mm could be expressed as 1.5 mm. But even if you obtained many measurements, it would generally be inappropriate to calculate the mean to three decimal places. The additional decimal places are not reliable because they cannot be measured with your instrument, and are therefore not significant figures.

[Looking back: In reviewing a Cambridge journal's author guidelines, I found the following commentary on significant figures: "Numerical results should be displayed as means with their relevant standard errors and degrees of freedom. Normally a mean should be rounded to one-tenth of its standard error and the standard error given to one decimal place more than the mean."]

A similar problem arises for percentages. Calculations based on percentages often have very low precision. For example, consider a population of 30 students being assessed in a psychology experiment. The maximum resolution of such measurements is 1 student. But if only 1 student responds in a specific and unusual way, that response is observed in 1/30 students = 3.3333% of the population (here, inappropriately expressed using four decimal places). But is it really appropriate to predict for that group of 30 that one-third of a student (the 0.3333 part of the calculated percentage) will exhibit that response? Probably not. This is why most journals will not accept more than 1 decimal place of precision for percentages unless the measurements are highly precise. For example, if you repeated the same experiment with 3000 students and 100 exhibited the unusual result, it might be reasonable to report this as 3.33% because 3000 × 3.33% = 99.9 students, which is very close to the actual value of 100, without requiring any fractional students.

In part 2 of this article, I’ll explain how those guidelines apply to statistical probability.

Acknowledgment

I thank Dr. Julian Norghauer for a reality check on this article.