Learning from mistakes made: The Millennium Villages Project

Millennium Villages
Millennium Villages

Eds note: Updated with rebuttal and comments from representatives of the MVP.

The Millennium Villages Project, an ambitious project launched across Africa by economist Jeffrey Sachs, is once again drawing attention.

A somewhat critical book, The Idealist, by Vanity Fair editor Nina Munk describes Columbia University Professor Sachs’ sometimes brash style in convincing even his strongest opponents to follow his poverty alleviation prescriptions.

The MVP, as the project is known, was launched by Sachs nearly a decade ago to demonstrate that relatively inexpensive targeted interventions can achieve major benefits against disease and poverty. Since then, the project has been repeatedly criticized for lacking adequate measures and data to back up its claims of progress.

The book’s profile of Sachs is augmented by two new reports on the MVP. One explores media attention to the MVP in Africa. Another says that the mistakes made by the MVP provide lessons for future projects and illustrate the importance of transparency in development projects.

Munk’s work describes Sachs’s tireless work to implement solutions to the problems that are killing people today. He implores leaders and his critics that action is needed today to save lives. It is literally a matter of life and death when it comes to the distribution of insecticide treated bednets.

The claims, described by some in the book as bullying, prove to be largely effective. Sachs and his colleagues on the MVP, launched in 2005, raised $120 million to implement in villages across sub-Saharan Africa. The book gives examples of ways in which the author says the program has failed and succeeded. The MVP in northeastern Kenya struggled while one in neighboring Uganda saw improved hospitals and health outcomes.

A coffee farmer in a Millennium Village in Rwanda, Bugesera District
A coffee farmer in a Millennium Village in Rwanda, Bugesera District

Transparency Revolution

Researchers Michael Clemens from the Center for Global Development and Gabriel Demombynes from the World Bank are among the chief critics of Sachs and the MVP. The two authored a paper in 2011 arguing that the MVP failed to provide adequate evidence that it actually worked.

Their new working paper says that the controversy surrounding the MVP is evidence of three ‘mini-revolutions’ in development economics: increased standard for impact evidence, open data and the growing role of blogs in research debates.

“A culture of impact evaluation has seeped into the halls of development, and policy now proliferates at least in part on the basis of evidence,” blogged Demombynes.

“All in all, my experience with the Millennium Villages debate makes me highly optimistic that the mini-revolutions of the new transparency are changing development policy very much for the better.”

The pair cite the example of a partially discredited paper published by the MVP researchers in the Lancet in 2012. It claimed that the MVP sites reduced child mortality rate three-times faster than rural trends in the respective countries. The problem was that the data compared were incorrect. When the appropriate periods were held side by side, the MVP sites mostly matched the trends of their countries.

Gains against child mortality were in fact not faster in the MVP sites. Development bloggers pointed out the error and The Lancet retracted some of the statements made in the study report. Unlike previous debates over papers, this happened quickly and publicly.

“The online debate ensured that by the time the Lancet article appeared, there was already a well-developed and transparent conversation about the MVP evaluation and observers who understood the core issues,” they write.

A comparison matters when trying to determine impact. Clemens and Demombynes say that the MVP could have done a better job evaluating its own work. Fortunately, there are now mechanisms that hold the program to account when it publishes results.

Kyu Lee, a spokesman for Sachs and the Earth Institute at Columbia, said Clemens and Demombynes make a number of inaccurate assertions in their new paper which he and his colleagues are seeking to have corrected by the authors.

For one thing, Lee explained to Humanosphere in an email, the researchers continue to refer to an informal assessment of the MVP called “Harvests of Development” a formal evaluation report and point at it as evidence of the poor metrics in the project.

“Clemens and Demombynes continue to misrepresent Harvests of Development to stir up controversies artificially, and that the project evaluation will be completed in 2015 and presented in mid-2016,” Lee said. The MVP team welcomes critical analysis and debate, he said, but they are misrepresenting facts and distorting the debate.

Success for the MVP, in part, means providing lessons for governments and donors. At it’s core, the MVP is based upon the principle of increasing investments and spending for communities. Governments, in theory, will see programs working and then devote budgets to areas like fertilizer subsidies and hospital improvements.

Media coverage, especially in Africa, play a key role in assuring that policymakers learn about the MVP and people can place pressure on their leaders to do more. Columbia University researchers, Michelle Chahine and Anya Schiffrin, evaluated the extent to which African media reports on the MVP.

They found that overall coverage for the MVP is favorable across Africa, but does not provide a significant level of depth. That is mostly due to a lack of resources for the press. Few actually did their reporting from the MVP sites. Additionally, the African press missed the discussions about the controversial study in 2012.

A significant part of the problem is that the journalists do not even know about Sachs nor the MVP. Press releases end up being the only way that many learn about news from the MVP. It is not a problem isolated in African press, say the researchers. Previous work shows that journalists cover ‘manufactured events’ such as press releases and staged events.

“[W]hen journalists do decide to write about development they may not have built up the expertise and contacts that come with beat reporting. For these reasons, journalists may rely on press releases, reports and sources produced by the Earth Institute and Millennium Village Project.”

As the MVP hopes to reach more people the increased external pressures may be helping to improve the way it conducts evaluations and possibly works in practice. Adding further reporting on the MVP within Africa will allow the people who have influence in host countries to apply the lessons learned.

“Diverting money away from alternative valuable uses is most ethically done based on sound, accurate evidence. Sound and transparent evaluation, far from being a spectator sport, is an ethical imperative,” says Clemens.

I will be reporting from the Sauri, Kenya MVP later this month.

Share.

About Author

Tom Murphy

Tom Murphy is a Maine-based reporter for Humanosphere. Before joining Humanosphere, Tom found and edited the aid blog A View From the Cave. His work has appeared in Foreign Policy, the Huffington Post, the Guardian, GlobalPost and Christian Science Monitor. He tweets at @viewfromthecave. Contact him at tmurphy(at)humanosphere.org.

  • James Collector

    Mr. Murphy,

    I found your article here thought-provoking in its exploration of the challenges for journalists and evaluators of the MVP. As a journalist and graduate student at UC Berkeley’s Development Practice Program, I have been researching the MVP this semester. One of my assignments is to comment on a blog. In September, I decided to follow the MVP blog for a few months so as to make a more educated comment. However, it seems that commenting is now closed on the MVP blog. Kyu Lee claims that “The MVP team welcomes critical analysis and debate.” Then why are comments closed on their blog?

    On a another note: although it was this article’s title that attracted me, I question what the MVP has learned from its mistakes. A truly ambitious and admirable project, the MVP must certainly have made numerous mistakes and these mistakes represent some of the most important information for a student like myself. The MVP website is mostly self-congratulatory; I wish it explored the process of implementation more. The website authors seem more concerned with results. I would appreciate more details. Am I alone in this sentiment or perhaps missing something?

    • http://aviewfromthecave.com Tom Murphy

      Thanks for the comments, James. This is probably an easier conversation over the phone than in comment section. In short, I think you are heading in the right direction and have raised some important questions.

      Unfortunately, there is a tension between PR and doing research. That is a widespread challenge across the sector. Right now, the form of communicating about programs heavily favors talking about successes and sweeping failure under the rug. It is something that all programs face.

      In Kyu’s slight defense, I do think that there is some space for critical analysis and debate. I’d say that he and I have a very friendly rapport and I make it known that I have serious doubts about the MVP.

      In fact, I was in Sauri in October to look at some of the programs. You can find some of the stories I wrote from my visit here: http://www.humanosphere.org/feature/growing-africa/.

      Shoot me an email tmurphy at humanosphere.org. I’d love to hear more of your thoughts and am more than happy to answer any questions you might have.

      • letai Ray

        Kindly send me ur private email on letairay@gmail.com. I would like to email you a dossier on how millions of dollars have been mis managed under the guidance of Siriri who claim to be taking right path in achieving the Millennium Villages Project in Uganda .
        And nothing has been done yet many are scrambling in poverty

  • Marieme DIAGNE

    Mr. Murphy

    I am a student doing a Master’s degree in Development studies in France and we (some of my classmates and I) would like to do a social and economic assessment of the MVP. Our goal is to determine if, with the Project’s withdrawal, the villages maintain the same dynamism as during the project and witness with our own eyes the benefits of the MVP.

    I would therefore be very grateful if you had any advice for us