A new study has raised questions about the benefits of a long-established health strategy pushed to improve child welfare in poor countries – and in so doing also raised questions about how best to do such impact evaluations.
It’s all about de-worming.
“Thanks to Deworm the World, and the effort of many country governments and foundations, 20 million school-aged children got dewormed in 2009. So this evidence is powerful. It can prompt action,” said MIT Economist Esther Duflo in a 2010 TED talk.
Duflo is widely considered a world leader in pushing for evidence-based health, aid and development projects. The push by experts like Duflo for better ‘metrics and evaluation’ in the development sector is intended to separate the wheat from the chaff, showing clearly what works and what doesn’t.
The global health, and humanitarian sector in general, has a history of sometimes promoting well-intended schemes based on assumptions rather than evidence. Many have celebrated the multiple benefits of ridding children of parasitic worms – from improved school performance to long-term socioeconomic status.
But a recent re-analysis of the claimed child health and education benefits of mass de-worming campaigns has both raised questions about its benefits and the difficulty of doing such complex evaluations. Turns out, not all data is created equal and it’s often the loudest voice that captures the most attention or support.
In late July, the re-analysis of a seminal study on deworming in western Kenya raised doubts about the efficacy of this presumed global health and education best-buy. These investigators concluded that the two billion or so people infected with parasitic worms (AKA ‘soil-transmitted helminths’) are likely no better off than they were before the big de-worming campaigns were launched.
The ensuing debate among academics – see #WormWars on Twitter, for example – revealed, depending upon your point of view, the need for better data or the need not to jump to negative conclusions based on lack of adequate data.
In short, everything in the humanitarian sector these days is under the microscope and the potential impact for the development community could be significant.
A foundational study under attack
The case for de-worming got a boost nearly two decades ago when economists Ted Miguel and Michael Kremer initiated a study in 1998 on the impact of schoolwide deworming programs in western Kenya. They selected the area for its high rates of intestinal helminth infections. Soil-transmitted worms and water-based schistosomiasis were of the greatest concern.
The results, published in Econometrica in 2004, were remarkable. Attendance for students who received the deworming treatment better than the students who did not. Absenteeism fell by one-quarter among students on the treatment. The protection for the students spread an average of 6 km from the school.
Given the cheap cost of the deworming medicines, the lack of side-effects and the significant impact, deworming was held up as one of the best buys for improving both health and education in children. The World Health Organization today recommends all school-age children living in worm-infested regions regularly receive de-worming treatment.
The life-cycle of parasitic worms. (CDC)
A follow up conducted by the authors a decade later showed the long-term impact of school-based deworming. Girls who participated in the program were 25 percent more likely to go to secondary school that those who did not. Gains for boys included more schooling, more hours worked each week, less hunger and a greater likelihood of holding manufacturing jobs. But it is the 20 percent improvement in income as adults for kids treated that stands out. It confirms previous research linking deworming and better long-term income.
This work provided the foundation for mass deworming programs.
However, in 2012 the Cochrane Collaboration, a group organized to independently and systematically review evidence of existing research trials, published its review arguing that there was little evidence to support mass deworming.
“Our interpretation of this data is that it is probably misleading to justify contemporary deworming programmes based on evidence of consistent benefit on nutrition, haemoglobin, school attendance or school performance as there is simply insufficient reliable information to know whether this is so,” concluded the authors.
A meta-analysis is different from primary research. And there are questions about statistical significance and conclusions drawn from those data in this kind of analysis. Much of this is lost when announcements come out that question long-held beliefs about best practices.
WHO
In this case, the original authors defended their work, saying that Cochrane ignored some recent evidence that strongly supports deworming. Both sides dug in, defending their data.
Meanwhile, another group of researchers from the London School of Hygiene and Tropical Medicine, led a team to do a re-analysis of the original Kenya study.
In the re-analysis, the researchers choose to deviate from some of the methods used by the original researchers. The published re-analysis claims that there are serious issues with the original study, and says that deworming doesn’t improve school attendance.
Cochrane published a separate meta-analysis of dozens of studies on deworming, published alongside the re-analysis papers, says that the evidence from dozens of studies overwhelmingly shows that mass deworming is ineffective.
From here all sides began defending conclusions, and suggesting that the other researchers used methods that led to inaccurate conclusions.
To deworm or not to deworm?
22-year-old health extension worker Samrawit Berhe gives a child deworming tablets as part of the 10th round Enhanced Outreach Strategy in Ethiopia’s Endaba Gerima Sub-district.
(UNICEF Ethiopia)
Given the debate and differing interpretation of the findings, what does this mean for mass deworming? Neither side conceded much ground during the debate. Rather both held tighter to their beliefs.
Some independent observers say that the Cochrane report omits important research and its authors come to the wrong conclusions.
Those involved with mass deworming remained steadfast in their disagreement with the re-analysis. A release from NGO Evidence Action raises questions about the science used and questions whether it got so much attention because “controversy sells.”
“The Cochrane Review gets this one wrong when they suggest that we need to test before treatment to get health benefits. Presumptive treatment is an operational, pragmatic approach to treating as many sick kids as possible with a limited amount of money,” according to the news release.
More than 150 million children were treated with deworming medicine as a part of Evidence Action’s deworm the world campaign. It is only part of a growing effort to help the 2 billion people infected with soil-transmitted helminth (aka parasitic worms) each year. At stake is whether or not mass deworming programs, as done in countries like Kenya and India, are all that helpful at improving health, education and income.
There is little evidence that the spread of mass deworming as an intervention is slowing down soon. The re-analysis sparked a debate, but did not change many minds about deworming.
“Overall, it appears that there can be short-term and long-term health benefits to deworming, especially when schistosomiasis is treated together with soil-transmitted worm infections,” concludes the blog of Giving What We Can, an initiative aimed at encouraging impact-based giving. “As a result, both GiveWell and Giving What We Can continue to recommend deworming and schistosomiasis treatment programs such as SCI and Deworm The World Initiative.”
Ben Goldacre, an outside expert who favored the outcomes of the re-analysis and Cochrane review in an article for BuzzFeed, and the re-analysis authors refrained from making public policy recommendations. But the Cochrane review is vocal in its opposition to deworming campaigns. Meanwhile, advocates and the authors of the original study say that the Cochrane review and the re-analysis doesn’t change evidence gathered over nearly two decades.
“What we should be looking at is the actual public policy on the issue of deworming. If you take the evidence you look at as a whole, there is a very strong case that deworming is a highly cost-effective policy,” said Kremer in an interview with Humanosphere. “A number of bodies that have looked at this have come to the conclusion that mass deworming is the appropriate strategy, and are standing behind their policies.”
As evidence-based strategies to alleviating poverty gain more notice, so too will re-analyses of the original research. They too must deal with questions regarding whether or not errors were made in looking back at an existing study. As this episode illustrates, the re-analysis authors are subject to the very same questions that they ask of original researchers. For 3ie and the authors of the deworming reanalysis, they hope that the papers contribute to a growing base of evaluations that closely examine prior findings.
“We anticipate that re-analysis will become more common, will improve transparency, accountability, and strengthen the literature that policymakers use to base decisions that affect the health and happiness of millions of people around the world,” wrote Calum Davey, an author of the re-analysis, and London School colleague James Hargreaves, in The Conversation.
The Worm Wars shows, as just one example of many such debates over the use data in development, the need for accurate research to help definitively show whether or not this particular project does improve school attendance and other social outcomes – beyond the obvious health benefits.
Whether new research or data will change already-entrenched beliefs is another matter.
