Metrics Mania afflicts the fight against poverty | 

Flickr, chrisjohnbeckett

Newton, statue outside the British Library

A lot of people who say they want to help poor people — the aid and development community — have been getting really nasty with each other lately. Why? In part, it’s because fighting poverty is messy and hard to measure.

At the center of this nastiness is a well-known economist, Jeffrey Sachs of Columbia University, who is chronically accused by critics of promoting an anti-poverty strategy –  known as the Millennium Villages Project – which they say is unproven.

The criticism flared again recently, prompting reports like this Forbes piece, which contends Jeff Sachs’ Millennium Villages Showing Zero Results

That’s not quite true. The more thoughtful critics don’t actually say Sachs’ approach doesn’t work. They just say it isn’t clear yet if it’s working.

But boy, does this lack of clarity make some folks angry! To wit:

  • One of Sachs’ leading critics, Michael Clemens at the prestigious Center for Global Development in Washington, DC, recently referred to Sachs on Twitter as “contemptible” for failing to acknowledge his project’s faults in an op-ed he wrote defending the value of foreign aid.
  • Timothy Ogden, editor-in-chief at Philanthropy Action, replied to Clemens (also on Twitter) that Sachs is to economics what Pat Robertson is to Christianity, which I assume was not meant as a compliment.

I have high regard for both Clemens and Ogden and this was, after all, just Twitter. But such hostility directed at Sachs is not that unusual. And it has gotten so intense lately I started wondering if the intensity is overwhelming the content here.

I’m no aid expert, economist or even really that good at math. I’m just a journalist who covers this stuff. And I do love a good argument. But I’m not so sure this qualifies as a good argument anymore.

One way to measure the quality and reliability of a dialogue is by the amount of personal insults or attacks it generates.

As a general rule, the more personal attacks you have the less reliable the information. Some philosopher gave this approach a Latin name – ad hominem – noting that such argumentation constitutes a ‘logical fallacy.’

It can also be quite entertaining, which at least brings more attention to this very important debate over how to improve the effectiveness of aid and development projects.  I certainly don’t think Clemens’ arguments are fallacious. They are important and widely regarded as legitimate critiques of Sachs’ approach — and as the spear point end of a broader push for metrics in aid & development.

But the ad hominem stuff does make you wonder what else is going on. Aren’t all these people basically trying to do the same thing — reduce poverty? Why such hostility?

Metrics Mania! 

Am I arguing that it should be acceptable to do things that can’t be measured? At the risk of sounding goofy, how do you measure happiness or hope? Less goofy: If aid allows those living in a poor community to feel valued, no longer neglected, is that worthwhile even if you can’t put hard numbers on its impact?

Albert Einstein said:  “Not everything that counts can be counted and not everything that can be counted counts.”

Flicky, Beto Ruis Alonso

Metrics Mania

For a series of articles I’m calling Metrics Mania, I start with (well, this post and) a recent interview I did with Jeff Sachs about the criticism of the Millennium Villages and the push for metrics in general. I also asked Sachs to say something agreeable about an economist many see as his arch-enemy, Bill Easterly, which he does.

I intend to follow up tomorrow with a report on a visit I made a few months ago to one of Sachs’ Millennium Village Projects in southern Rwanda. Later stories will feature interviews with others working to improve the lives of the poor and the impact of aid or development.

To be clear, I am not anti-metrics. I have a science background and anyone who reads Humanosphere knows I like evidence and don’t like squishy thinking. And so far as I can tell, everyone in the aid and development community wants to know if their efforts are paying off.

But as Einstein says not everything worth doing can be measured.

And too great an obsession for hard numbers may turn into numerology, mania, and even undermine the very aims the metrics crowd claims to be championing.

  • Calumdavey

    Interesting post, but I cannot agree. People in the developing world have been at the receiving end of well meant ‘programs’ of one sort or another for centuries and it is time to start holding those programs to account. Common sense leads to all sorts of conclusions while science in the west plods along debunking myth after myth.  It’s true that some things are hard to measure (although, have we tried?) but when the sums of money are this great, can we really justify the expense for a few wooly stories? 

    I believe that the time has come to put power back in the hands of the people that Sachs etc are trying to help.  Just because they are so kind and benevolent does not mean that they should be exempt from the standards of evidence that we demand in the west.  We hold government agencies, companies, foundations, and schools to account using watch dogs and statutory bodies.  But only for us, westerners, whereas if someone wants to go wild in Africa, well, then they have free reign. Maybe market mechanisms are not the solution, as they are not always the solutions here in the UK, but there must still be regulation, measurement, accountability, and the option to fire someone who doesn’t deliver what they promised.  

    • http://humanosphere.kplu.org Tom Paulson

      Thanks Calumdavey,

      To be clear, I am not arguing against evaluating these projects or holding them accountable. I am arguing for a little less maniacal devotion to metrics. As I noted, even Einstein said some things that count can’t be counted.

      I think everybody agrees with Albert. So how do we proceed if we want to improve the effectiveness of aid and development?

      To begin with, we need to accept not everything worth doing can be assesed using numbers or precise quantitative comparisons. This is just reality. One of the Millennium Development Goals is to empower women and improve gender equity. It’s very important but not really too easy to measure.

      Trying to make everything subject to precise measures (e.g., requiring that everything be testable by a randomized controlled trial) can distort priorities and challenge common sense. 

      Is Jeff Sachs’ approach working? I don’t know. We’ll see if he can satisfy his critics by bringing in outside experts to monitor and evaluate. In the meantime, Congress is cutting the miniscule amount of money we spend on foreign aid and the gap between the rich and poor is widening.

      I think it is misleading to say we, the rich world, have thrown a ton of aid money at poverty and it hasn’t worked. Look a bit closer and you’ll see that a lot of that money didn’t go to fight poverty; it went to American contractors working overseas, to American farmers for food subsidies or simply to friendly despots (e.g., Egypt) in the form of military aid or other funds serving our ‘national interests.’

      I agree with Sachs on one thing: It really isn’t that big a mystery why poverty exists and what are some basic things people need to help them get out of extreme poverty. The problem is that the solution involves thorny political issues, not to mention the likelihood that we in the rich world might have to make a few sacrifices to balance things out.

      Much easier to argue about numbers.

      Cheers

      • Calumdavey

        Dear Tom

        Thanks for the thorough response.  Unfortuately I still cannot agree.

        First, I am not sure that what Einstein said is completely relevant here.  He was talking about counting, while metrics are about measuring. These are different things.  We cannot count ‘love’ in a single person but we can ask a thousand people if they have ever been in love and get a proportion that measures the amount of love in these people’s pasts.  A similar thing can be done with gender equity and female empowerment. It is a little disingenuous to say that these cannot be measured since there is a huge body of literature doing precisely that (including RCTs).

        But let’s suppose for a second that there are really important things that cannot be measured quantitatively.  Is the inability to measure these outcomes relevant for MVP? I would argue that it is not, at least in the first instance.  And my reason? Because the MVP measured many quantitative outcomes – indeed made under five mortality reduction its principle aim – and yet is still angering its critics.  This is because the quantification of the outcome of interest is distinct from good evaluation design.  The MVP’s problem is the lack of a valid counterfactual control group, not that the outcomes are unquantifiable.  In fact, it would be possible to have either one without the other, and possible that a blinded qualitative outcome measurement in comparable control and intervention groups is more helpful for attributing the change in outcomes to a causal effect of the intervention. Nothing about the MVP precluded random allocation, especially not the lack of quantities to measure.

        The backlash against metrics may hide a darker agenda. MVP failed to show (so far) that they had a meaningful effect *on the things that they identified as outcomes in advance*, so the only option is to say that there has been meaningful change in things that were not measured! There are so many things wrong with this as a scientific approach that it’s maddening to even consider. Think, for a second, how you would react if a politician was defending their record like that.

        I think that what you say about aid is very true and that it actually supports my point.  So much ‘aid’ has been wasted in exactly the way that you describe so why not have a health skepticism of anyone’s claim to be making poorer people’s lives better?  And I am not talking about big faceless agencies, I was also talking about generations of missionaries; there is a whole tattered history of ‘nice’ people going out to Africa and failing to make things better.