Warning: Reductionism can result in distorted vision, poor judgment and difficulty in operating a humanitarian project.
There’s a popular trend today among many humanitarians, aka the aid and development sector, to try to show the benefit of their projects – be it digging a well, feeding kids or improving access to basic health care – with scientific data.
That’s good in principle, if you have a well-designed study that produces meaningful data. But that can be a big if when what you are trying to test is a reduction in poverty, social and economic improvements, healthy behavior change or many of the other aims of aid and development.
It’s much easier for scientists to test a more isolated intervention, like say taking a pill, than it is to even figure out how best to track and attribute the potential impact of many humanitatian efforts. And it’s worth noting that the scientific community is finally acknowledging that even their most refined efforts in reductionist deduction, peer review and attribution often fail.
NY Times Scientific Pride and Prejudice
Economist Trouble at the Lab
Forbes NIH Promises to Make Science Less Wrong
The mainstream scientific community likes to call this a ‘reproducibility’ problem, saying the overall reliability and self-correcting nature of the scientific method(s) remain intact. But when it is noted, as in the NYTimes op-ed, that a team of scientists could only confirm the findings in six of more than 50 ‘landmark’ cancer studies, there is cause for concern.
Meanwhile, the humanitarian sector has a different problem. It tends to suffer from a lack of data or consensus on how best to measure the impact of various initiatives aimed at fighting poverty, diseases of poverty or other kinds of human inequity. The field did not arise, like science, from a desire to know so much as from a desire to help.
So will it help if humanitarians become more like scientists? Maybe. Maybe not. Humanosphere participated in a brief debate that flared up on Twitter over the weekend in between the still-dominant Superbowl Twits (I say Go Hawks! Other say Go Away Already! …). The non-football Twitter debate was prompted by a study done in Ghana that purported to show that eliminating user fees or required out-of-pocket spending in this lower-income country did not result in any ‘overall’ health improvements.
(Here’s a non-paywalled link to download an earlier iteration of this study done mostly by researchers at the London School of Hygiene and Tropical Medicine. Here’s another link that describes their basic methods and conclusions).
Newton, statue outside the British Library
One of the more popular methods promoted by the metrics/evaluation crowd in the aid and development sphere is called the Randomized Controlled Trial (or RCT). It is essentially an attempt to use the standard double-blinded approach used in drug clinical trials to evaluate the effectiveness of various humanitarian efforts.
The British team characterized their study as an RCT aimed at measuring if removing the standard user fees or out-of-pocket payment requirements would produce measurable improvements in health. They found some reduction in anemia for high-risk kids, but said they found no evidence of ‘overall health improvements.’
A number of global health or aid/dev experts responded affirmatively to a Tweet from one leading expert at a DC think tank saying the study showed free health care doesn’t improve health.
This response seemed odd since many other studies have shown that user fees in poor countries do undermine health goals – for obvious reasons. Poor people won’t seek health care at an early stage if it costs too much (and ‘too much’ ain’t much if you live on a few dollars a day) so they don’t do preventive care and early-stage treatment which is usually both cheaper and more effective. They wait until it’s a crisis.
Here are a few links to other reports, studies or advocacy briefs that say free access to basic health care services does improve health in poor countries and financial barriers to accessing health services cause harm:
NYTimes In Sierra Leone, new hope for children and pregnant women
Partners in Health Taking a stand against user fees for health
Social Science and Medicine The Hidden Costs of User Fees
As Rob Yates, a senior economist with Britain’s Department for International Development (DFID), noted in the Twitter debate, the Ghana study did not actually define clearly what it meant by ‘overall health improvements’ and really only measured the rates of childhood anemia (which were reduced when user fees were removed). It was a small study, Yates noted, lacking in clear endpoints.
In short, Yates says it is fair to conclude that the Ghana study found little evidence of a big positive ‘overall’ impact from removal of user fees. But this was arguably due to the study’s limitations; it is not accurate to say the study showed removing the fees did not benefit the poor. The study just wasn’t powerful enough to do anything more than it did.
And absence of evidence, as they say, is not evidence of absence.
So do user fees and making poor people pay even a little for visiting a clinic or a nurse undermine ‘overall health’ or not?
One could argue that we’ve been testing the value of user fees and out-of-pocket payments in poor countries for decades, since this was a scheme promoted many years ago by the World Bank and International Monetary Fund as part of a neoliberal anti-poverty strategy known as ‘structural adjustment.’ Given this, you could argue – in a very loose, broad and non-reductionist way – that the approach has obviously failed since there’s no clear evidence the approach improved health or (as intended) improved financing for health care in poor countries.
But that would probably be going beyond the evidence as well, from a scientific standpoint anyway. Here’s two good analyses, one by WHO and the other by the World Bank, looking at the equivocal evidence for either side of the argument.
Perhaps the most evidence-based position here is the studies can’t yet prove that removing financial barriers to health care services improves the health of the poor. More studies needed. But scientists always say that, don’t they? That’s because they get paid to study things and, well, there is almost always room for debate, for challenging this or that fact or data set.
Meanwhile, a poor mother in Ghana with a very sick child is trying to figure out if she should spend money to feed the family or go to the clinic.
So let’s be careful when we’re pretending to be scientists running a simple plus-minus double-blind trial. There’s much more at stake here if we draw the wrong, or premature, conclusions.
