Eds note: Updated with rebuttal and comments from representatives of the MVP.
The Millennium Villages Project, an ambitious project launched across Africa by economist Jeffrey Sachs, is once again drawing attention.
A somewhat critical book, The Idealist, by Vanity Fair editor Nina Munk describes Columbia University Professor Sachs’ sometimes brash style in convincing even his strongest opponents to follow his poverty alleviation prescriptions.
The MVP, as the project is known, was launched by Sachs nearly a decade ago to demonstrate that relatively inexpensive targeted interventions can achieve major benefits against disease and poverty. Since then, the project has been repeatedly criticized for lacking adequate measures and data to back up its claims of progress.
The book’s profile of Sachs is augmented by two new reports on the MVP. One explores media attention to the MVP in Africa. Another says that the mistakes made by the MVP provide lessons for future projects and illustrate the importance of transparency in development projects.
Munk’s work describes Sachs’s tireless work to implement solutions to the problems that are killing people today. He implores leaders and his critics that action is needed today to save lives. It is literally a matter of life and death when it comes to the distribution of insecticide treated bednets.
The claims, described by some in the book as bullying, prove to be largely effective. Sachs and his colleagues on the MVP, launched in 2005, raised $120 million to implement in villages across sub-Saharan Africa. The book gives examples of ways in which the author says the program has failed and succeeded. The MVP in northeastern Kenya struggled while one in neighboring Uganda saw improved hospitals and health outcomes.
Researchers Michael Clemens from the Center for Global Development and Gabriel Demombynes from the World Bank are among the chief critics of Sachs and the MVP. The two authored a paper in 2011 arguing that the MVP failed to provide adequate evidence that it actually worked.
Their new working paper says that the controversy surrounding the MVP is evidence of three ‘mini-revolutions’ in development economics: increased standard for impact evidence, open data and the growing role of blogs in research debates.
“A culture of impact evaluation has seeped into the halls of development, and policy now proliferates at least in part on the basis of evidence,” blogged Demombynes.
“All in all, my experience with the Millennium Villages debate makes me highly optimistic that the mini-revolutions of the new transparency are changing development policy very much for the better.”
The pair cite the example of a partially discredited paper published by the MVP researchers in the Lancet in 2012. It claimed that the MVP sites reduced child mortality rate three-times faster than rural trends in the respective countries. The problem was that the data compared were incorrect. When the appropriate periods were held side by side, the MVP sites mostly matched the trends of their countries.
Gains against child mortality were in fact not faster in the MVP sites. Development bloggers pointed out the error and The Lancet retracted some of the statements made in the study report. Unlike previous debates over papers, this happened quickly and publicly.
“The online debate ensured that by the time the Lancet article appeared, there was already a well-developed and transparent conversation about the MVP evaluation and observers who understood the core issues,” they write.
A comparison matters when trying to determine impact. Clemens and Demombynes say that the MVP could have done a better job evaluating its own work. Fortunately, there are now mechanisms that hold the program to account when it publishes results.
Kyu Lee, a spokesman for Sachs and the Earth Institute at Columbia, said Clemens and Demombynes make a number of inaccurate assertions in their new paper which he and his colleagues are seeking to have corrected by the authors.
For one thing, Lee explained to Humanosphere in an email, the researchers continue to refer to an informal assessment of the MVP called “Harvests of Development” a formal evaluation report and point at it as evidence of the poor metrics in the project.
“Clemens and Demombynes continue to misrepresent Harvests of Development to stir up controversies artificially, and that the project evaluation will be completed in 2015 and presented in mid-2016,” Lee said. The MVP team welcomes critical analysis and debate, he said, but they are misrepresenting facts and distorting the debate.
Success for the MVP, in part, means providing lessons for governments and donors. At it’s core, the MVP is based upon the principle of increasing investments and spending for communities. Governments, in theory, will see programs working and then devote budgets to areas like fertilizer subsidies and hospital improvements.
Media coverage, especially in Africa, play a key role in assuring that policymakers learn about the MVP and people can place pressure on their leaders to do more. Columbia University researchers, Michelle Chahine and Anya Schiffrin, evaluated the extent to which African media reports on the MVP.
They found that overall coverage for the MVP is favorable across Africa, but does not provide a significant level of depth. That is mostly due to a lack of resources for the press. Few actually did their reporting from the MVP sites. Additionally, the African press missed the discussions about the controversial study in 2012.
A significant part of the problem is that the journalists do not even know about Sachs nor the MVP. Press releases end up being the only way that many learn about news from the MVP. It is not a problem isolated in African press, say the researchers. Previous work shows that journalists cover ‘manufactured events’ such as press releases and staged events.
“[W]hen journalists do decide to write about development they may not have built up the expertise and contacts that come with beat reporting. For these reasons, journalists may rely on press releases, reports and sources produced by the Earth Institute and Millennium Village Project.”
As the MVP hopes to reach more people the increased external pressures may be helping to improve the way it conducts evaluations and possibly works in practice. Adding further reporting on the MVP within Africa will allow the people who have influence in host countries to apply the lessons learned.
“Diverting money away from alternative valuable uses is most ethically done based on sound, accurate evidence. Sound and transparent evaluation, far from being a spectator sport, is an ethical imperative,” says Clemens.
I will be reporting from the Sauri, Kenya MVP later this month.