A report published last week stated that road deaths in the province had fallen by fifteen percent. The figure appeared in every paper, usually in the first sentence, and was treated as straightforward good news. It may well be good news. But almost nothing in the coverage allowed a reader to judge whether it was, because the single number that was printed is the one number that cannot answer the question on its own.
Three pieces of information are missing, and each of them is available in the original document.
The first is the period being compared. Fifteen percent lower than what? The report compares the twelve months to June with the twelve months before, and the earlier period contains the unusual traffic conditions of a year when fuel prices rose sharply and many people drove less. Comparing against an unusual year produces a large difference in either direction, and tells you mainly about the year you chose.
The second is the size of the underlying count. A fall from four hundred and ten deaths to three hundred and fifty is a different kind of event from a fall from twenty to seventeen, although both are close to fifteen percent. Small counts move a great deal by chance. Any percentage quoted without the count behind it should be treated as an opinion rather than a measurement.
The third is what else changed at the same time. The province opened two sections of new road, altered the speed limit on a third, and increased police checks during holidays. All three may have contributed. So may the weather, which was wetter than usual and reduced driving on several weekends. A report that lists one cause when four are available is not lying, but it is choosing, and the reader is not told that a choice occurred.
None of this is difficult, and none of it requires any training in maths. It requires only the habit of asking three questions before accepting a figure: compared with when, out of how many, and what else was going on.
The deeper problem is that the people producing these reports face no cost for leaving the questions unanswered. A press release is judged by whether it is repeated, and a clean number is repeated more readily than a careful one. The paper that prints the full comparison uses more space and appears less confident, which is a bad position in a market where attention is the product being sold.
It would help if public bodies were required to publish figures in a standard form, with the count, the period and the known limits stated next to the headline result. Several health departments already do this, and their releases are clearly more boring and clearly harder to read in the wrong way.
Until that becomes normal practice, the work falls on the reader, which is an unfair place to put it but is where it currently sits. The fifteen percent may be entirely real. The point is that a person who read only the headline has no way to know, and has been left with the comfortable feeling of having been informed.