Reading the Spec
Accuracy, precision and resolution are three separate claims
Instrument specifications use three words that sound interchangeable and describe entirely different properties. Only one of them is expensive to improve, which explains which one gets advertised.

These are listed in the order worth acting on, which with measurement accuracy claims is not the order they are usually presented in.
What matters most
- Resolution is the cheapest of the three to increase.
- Accuracy figures depend on a stated reference.
- Percent of reading and percent of range differ enormously.
What each word means
Resolution is the smallest change a device can display, which is a property of the display and the conversion electronics rather than of the sensor. Precision, sometimes called repeatability, describes how closely repeated measurements of the same thing agree with each other.
Accuracy describes how close a reading is to the true value, which requires a reference the device was compared against. A device can be highly precise and badly inaccurate, returning the same wrong answer every time, and this is the most common combination. Adding decimal places to a display improves resolution alone, which is why a long number is not evidence of a good instrument.
Reading a plus-or-minus figure properly
An accuracy claim expressed as a percentage of the reading scales with the measurement, so a small measurement carries a small absolute error. An accuracy claim expressed as a percentage of full scale does not scale, so at the bottom of the range the absolute error can dominate the reading.
Many specifications combine both, adding a percentage of reading to a fixed number of display counts, which is the honest way to state it. The stated figure almost always applies over a limited part of the range, and performance outside that band is either worse or unspecified. Where an accuracy figure appears without a range and without the conditions it applies to, it is not a usable specification.
The conditions attached to every accuracy figure
Temperature affects most sensing elements, and specifications usually state a reference temperature with a separate coefficient for departures from it. Warm-up time matters because electronics drift while they reach thermal equilibrium, and a reading taken immediately after switch-on is not the specified reading.
Supply voltage, battery condition, orientation, humidity and electrical noise all appear as conditions in a full specification sheet. Consumer instruments frequently publish the headline figure and omit the conditions, which makes the figure a best case rather than a working expectation. The presence of a conditions table is itself a signal that the maker expects the instrument to be used where accuracy matters.
Drift, ageing and why calibration exists
Sensing elements age, and their output at a given input changes slowly over months and years in ways the electronics cannot detect. Mechanical shock shifts some sensors permanently, which is why dropped instruments should be checked rather than assumed to be fine. Calibration compares the instrument against a reference of known quality and either adjusts it or documents the error found.
Traceability means that reference can be linked through a chain of comparisons to a recognised standard, and that chain is what a calibration certificate records. Requirements for calibration in trade, clinical and industrial use are set by regulation and differ substantially between countries.
Checks you can perform yourself
Reference points that occur naturally are the most useful, such as the freezing point of water for thermometers under stated conditions. Known masses check a scale usefully, and a set of certified weights is available at modest cost for anyone who needs one.
Repeating a measurement several times reveals precision immediately, since scatter between readings is visible without any reference at all. Comparing two instruments tells you they disagree without telling you which is wrong, so a third opinion or a reference is required. A device that reads differently after being moved, tilted or warmed is telling you something its specification sheet probably did not.
Manufacturer figures are measured under conditions the manufacturer chose.
Reading an instrument specification
Find the accuracy statement first and check whether it is percent of reading, percent of range, or a combination of both. Look for the range over which that accuracy applies, since ranges outside the specified band are frequently much worse.
Compared like for like, check the resolution separately and treat any surplus digits beyond the accuracy figure as decoration rather than information. Look for a stated calibration interval, because a maker who publishes one is describing how quickly they expect drift to matter. Where the instrument will be used for anything with consequences, the relevant question is whether it can be checked or adjusted at all.
Everything above, in order of what to do first
- What each word means. Resolution is the smallest change a device can display, which is a property of the display and the conversion electronics rather than of the sensor.
- Reading a plus-or-minus figure properly. An accuracy claim expressed as a percentage of the reading scales with the measurement, so a small measurement carries a small absolute error.
- The conditions attached to every accuracy figure. Temperature affects most sensing elements, and specifications usually state a reference temperature with a separate coefficient for departures from it.
- Drift, ageing and why calibration exists. Sensing elements age, and their output at a given input changes slowly over months and years in ways the electronics cannot detect.
- Checks you can perform yourself. Reference points that occur naturally are the most useful, such as the freezing point of water for thermometers under stated conditions.
- Reading an instrument specification. Find the accuracy statement first and check whether it is percent of reading, percent of range, or a combination of both.
The takeaway
Read the accuracy statement and its conditions, and treat spare digits on the display as decoration.
Buy for the failure you can live with, not the feature you will use twice.
Questions readers ask
Does more decimal places mean a better instrument?
No. Extra digits improve resolution, which is the cheapest of the three properties to increase. Digits beyond the accuracy figure carry no information about the true value.
What is the difference between precision and accuracy?
Precision is agreement between repeated readings; accuracy is closeness to the true value. A device can return the same wrong answer consistently, which is precise and inaccurate.
Also by Kavitha Srinivasan
- Lumens are counted at the emitter and light is judged at the wallReading the Spec
- A decibel figure was measured in a room you do not haveReading the Spec
- The cells inside a battery pack decide more than the amp-hour numberReading the Spec
- Up to is a legal phrase and it changes what a number meansReading the Spec





