Measurement Resolution Is Not Measurement Accuracy
A display reading 23.847 °C can look more authoritative than one reading 23.8 °C. The extra digits may be useful, but they do not establish that the measured environment is actually 23.847 °C, or even that it is within a few tenths of a degree of that value.
This distinction matters wherever measurements support release decisions, process control, safety limits, maintenance intervals or regulatory evidence. A system can record data at fine resolution while its readings remain biased, poorly installed, insufficiently calibrated or unsuitable for the quantity it claims to represent.
Resolution is a property of what the instrument can distinguish or display. Accuracy is commonly used in engineering conversation to describe closeness to a reference value, although the International Vocabulary of Metrology cautions that accuracy is a qualitative concept and is not itself a quantity. Neither is established by adding decimal places.
What resolution actually tells us
Resolution is the smallest change in an indication that an instrument can meaningfully display or register. A digital temperature transmitter may report in 0.001 °C increments. An analogue-to-digital converter may quantise an input into thousands of discrete codes. A balance may show 0.01 g increments.
Those increments describe granularity. They do not account for systematic error, environmental effects, sensor drift, electrical noise, installation effects or data processing.
Consider a pressure transmitter with a displayed resolution of 0.01 bar. If its calibrated error at the operating point is ±0.15 bar, the final two decimal places are not evidence of corresponding knowledge about pressure. They may still be operationally useful for observing relative movement, provided the noise level, repeatability and stability support that use. They should not be interpreted as a statement that the absolute pressure is known to 0.01 bar.
There is a second failure mode: a high-resolution output can make random variation appear as a real process change. If a sensor's short-term noise is larger than the display increment, a trend chart may show continual movement even in a stable system. Filtering may improve readability, but filtering also changes response behaviour and can conceal short-duration events. The choice must follow the decision the measurement is intended to support.
Accuracy, uncertainty and the measurement chain
A defensible measurement concerns the entire measurement system, not merely the sensor datasheet. The relevant chain may include the sensing element, transmitter, cabling, signal conditioning, digitisation, software scaling, timestamps, storage and presentation.
Each element can contribute error or uncertainty. For temperature monitoring, a technically plausible uncertainty budget may need to consider:
- calibration uncertainty at relevant temperature points;
- sensor tolerance, drift and self-heating;
- thermal gradients between the sensor and the material or space of interest;
- installation, including immersion depth, thermal contact and radiation effects;
- transmitter accuracy and digital conversion;
- interpolation, averaging and rounding performed in software.
A well-calibrated probe mounted against an external vessel wall is not necessarily measuring product temperature. It may be measuring a mixture of vessel-wall temperature, ambient conditions and local conduction paths. Increasing the recorded resolution does not correct a poor measurement model.
The Guide to the Expression of Uncertainty in Measurement describes uncertainty as a parameter characterising the dispersion of values that could reasonably be attributed to a measurand. In practical terms, the question is not only “what number did the system produce?” but “within what justified interval is the physical quantity expected to lie, and is that interval acceptable for the intended decision?”
Decimal places can damage decisions
False precision has operational consequences. An alarm threshold set at 8.00 °C may suggest a sharp physical boundary. The underlying control decision may nevertheless be unreliable if the measurement uncertainty near that threshold is ±0.5 °C, the sensor response is slow, or the probe location does not represent the controlled volume.
This does not mean that a limit is unusable. It means the engineering must define how uncertainty, response time and spatial variation are managed. Controls might include an appropriately conservative action threshold, qualified sensor locations, periodic verification, and a defined response when readings approach the limit.
The same issue arises in calibration records. Reporting a result as 100.000 units when the reference standard, method and environmental conditions support only a much wider uncertainty can imply knowledge that the calibration has not demonstrated. Reported digits should be consistent with the uncertainty and decision rules applied. The required reporting convention depends on the method and applicable quality system, but it should not invite an unsupported interpretation.
Designing for the decision, not the display
The starting point is the measurand: the specific quantity intended to be measured under stated conditions. “Room temperature” may be adequate for a general building-management trend. It is not a sufficiently defined measurand for demonstrating conditions in a temperature-controlled storage zone where location, time, load state and airflow materially affect the result.
Once the decision is clear, the system can be specified accordingly. Required range, allowable uncertainty, response time, sampling interval, resolution, calibration interval and installation arrangement should be derived from the use case. Resolution should be fine enough that quantisation does not materially constrain the measurement or control function. It need not be finer than every other source of uncertainty merely because the electronics permit it.
For controlled environments, the evidence must also allow later reconstruction. The organisation should be able to identify which instrument produced a value, its calibration status, the applied configuration and scale, the time basis, and any transformations between acquisition and report. These controls do not improve sensor physics, but they establish whether the recorded number can be interpreted and defended.
A measurement system earns trust when its displayed value is proportionate to what the sensing arrangement, calibration evidence and uncertainty analysis can support. More digits may be appropriate. They are never sufficient.
References
- Joint Committee for Guides in Metrology, International vocabulary of metrology - Basic and general concepts and associated terms (VIM), JCGM 200:2012. Available from BIPM: https://www.bipm.org/en/committees/jc/jcgm/publications
- Joint Committee for Guides in Metrology, Evaluation of measurement data - Guide to the expression of uncertainty in measurement (GUM), JCGM 100:2008. Available from BIPM: https://www.bipm.org/en/committees/jc/jcgm/publications