It was either Mark Twain or former British Prime Minister Benjamin Disraeli who once said, “There are three kinds of lies: Lies, damned lies and statistics.”
Sometimes, statistics do lie or mislead. Yet, used correctly, they are powerful conveyors of reality and can provide new insights into something previously neglected or dismissed.
In global health, it was statistical analyses that first indicated something new was going on with the Zika virus. In the political realm, polls are telling us something new appears to be going on with the U.S. electorate.
Many in the United States are keeping a close eye on election forecasts, especially in light of the latest twists and turns in the race for the White House. It’s been a roller coaster ride, with some polls showing Donald Trump ahead of Hillary Clinton while others claim their statistics show just the opposite.
(It may be worth noting that, in 2012, pollsters showed Mitt Romney in a dead heat with Barack Obama. It turned out not at all close with Obama receiving 5 million popular votes – nearly 4 percentage points ahead – and a total of 332 electoral votes to Romney’s 206 electoral votes.)
Understanding how any of these forecasts are created can also help shed light on why it is important for researchers to use similar methods to measure the world’s health.
When this article went to press, the New York Times estimated that Clinton had an 84 percent chance of winning the presidency, and Trump had a 16 percent chance. The Times created these estimates by using statistical models to analyze the results of multiple state and national polls.
Using findings from many polls instead of a single poll increases the chances of correctly predicting the outcome of an election. For example, see the graphic on the right below, which shows all the different polls used to inform the Times’ election model for Ohio, where they predict that Trump has a 55 percent chance of winning.
New York Times election model for Ohio
Source: Obtained from the New York Times on Nov. 2, 2016.
The researchers behind the Global Burden of Disease (GBD) study use a similar approach to measure the world’s health problems: compile all the data that’s relevant for a particular health problem and analyze it with statistical models to produce rigorous global health estimates.
Reliance on estimates is common in daily life. Weather forecasts, unemployment rates, reports of traffic delays on your GPS, measures of economic growth such as GDP – all are estimates created using statistical models.
I’ve met many people working in global health who are wary of estimates from statistical models, preferring instead to use a single survey or government data source.
But using results from one source of data can give people a skewed picture of reality, as in this example of data on children’s health. Below is a graph that shows all the data sources (represented by dots of different shapes) that GBD researchers used to estimate child deaths in Afghanistan. It’s a complicated graph, but the important takeaway is that the red diamonds in the lower part of the graph represent data from a survey that was conducted in safer areas of the country and painted a rosier picture of children’s health than GBD researchers believe is really the case (the dark green shading represents what GBD researchers believe to be the actual range of estimates for child death rates).
By using all these different data points to measure child death rates in Afghanistan, researchers get more scientifically sound estimates than they would using just one source of data.
Afghanistan: child mortality estimates (1990–2015) and data sources
Source: GBD 2015 Mortality and Causes of Death Collaborators. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980–2015: a systematic analysis for the Global Burden of Disease Study 2015. The Lancet. 2016 Oct 7; 388:1459–1544; access the data visualization tool here: http://ihmeuw.org/3y7w.
And even in countries where data is considered high-quality, such as the U.S., researchers can’t simply accept it as truth.
“In the U.S., physicians reported an estimated one out of every three cardiovascular disease deaths incorrectly on death certificates,” said Mohsen Naghavi, a co-author of the GBD study.
In these cases, physicians listed causes that cannot or should not be considered underlying causes of death, such as “heart failure,” which is a consequence of a cause, not a cause itself (many different diseases can cause heart failure, such as ischemic heart disease, congenital heart defects and respiratory diseases). Using statistical models and other tools, Naghavi and his team adjust the data from the death certificates to address these problems.
While statistical models are critical tools for global health metrics, that doesn’t mean that people shouldn’t strive to collect higher-quality data. Higher-quality data translates to better global health estimates. Take the example of maternal death rates in South Africa versus Botswana. Because South Africa has higher-quality data (such as a nationally representative vital registration system), and data are not available from such a system in Botswana, maternal mortality estimates for the former are stronger than for the latter.
The graph below shows estimates for the two countries. The shading surrounding the solid lines represents the range of possible estimates of maternal disorders. The range is much wider for Botswana than it is for South Africa.
Maternal death rates in Botswana and South Africa, 1990–2015
Source: GBD 2015 Maternal Mortality Collaborators. Global, regional, and national levels of maternal mortality, 1990–2015: a systematic analysis for the Global Burden of Disease Study 2015. The Lancet. 2016 Oct 7; 388:1775–1812; access the data visualization tool here: http://ihmeuw.org/3y7v.
Like election predictions, global health estimates are imperfect. Some of them are right, but others will be proved to be incorrect. Even Nate Silver, the celebrated creator of fivethirtyeight.com, is forthcoming about the shortcomings of his election forecast. That’s why researchers all over the world need to keep working to improve the science, continuing to raise the bar for global health metrics. They must incorporate the latest data as soon as it’s made available. Finally, researchers must keep developing new methods to measure health and adopting the latest breakthroughs in modeling technology.
This article benefited from valuable input from David Phillips and Adrienne Chew at the Institute for Health Metrics and Evaluation.
