Update: Humanitarian rankers don’t like getting ranked on

In case you haven’t been following the comment thread on my earlier post regarding the Top 100 NGOs as identified by Global Journal, I wanted to post here a critical look at the rankings by development professional Dave Algoso.

Algoso is an expert on aid and development issues. Here is his post Lies, Damned Lies and Ranking Lists: The Top 100 Best NGOs written in response to my earlier post about Global Journal:

Ranking lists are great publicity for both the rankers and the ranked but they usually involve bad analysis and mislead the readers…. Most of these NGOs are, to the best of my knowledge, quite good. My big disagreement is with GJ‘s ranking methodology. And the fact that they created this list at all.

Meanwhile, the equally well-intentioned folks at Geneva-based Global Journal have expressed, to me by email and in various comment threads, their disappointment at being ‘ranked on’ for publishing their list of the top non-governmental organizations working at making the world a better place.

The editor, Jean-Christophe Nothias, takes special umbrage at being criticized by lowly bloggers and even contends this may involve ‘libel.’ Says Nothias of their rankings:

It is a journalistic approach, not an academic, not a mathematical, one approach that understands a simple fact. Profit has a metric, money. How do you measure solidarity? How do you measure healing, suffering? Do you believe such a ranking has anything to do with the S&P, the NYSE and other financials index?

Right, so how did they do it? How did Global Journal arrive at placing Seattle-based PATH as 6th best NGO in the world — along with ranking a few other local organizations like Mercy Corps and Landesa — and inexplicably exclude other top NGOs like World Vision and the Gates Foundation?

The folks at Global Journal don’t want to go into the details. They appear to be arguing that they didn’t depend solely upon a quantitative methodology that can be checked by others for reliability. They also relied on their journalistic methodology, their own expert judgment, as Nothias says:

Do bloggers have a methodology? Do they make a difference between being a reporter and a rapporteur? Or is journalism, in their eyes, at the cemetery? We have an ethic and a strong belief in the fact that journalism is already part of the methodology.

As a journalist who is also apparently a blogger, I can say with great confidence that the ‘methodology’ and reliability of journalism is highly variable. Ranking, by its very nature, implies some kind of quantitative assessment that should be independent of even the best journalistic judgments.

As far as Algoso is concerned, Global Journal’s list is so arbitrary and subjective it is meaningless:

Ultimately, it sounds like the methodology was: we browsed the web, talked to a couple people, then sat around the conference table arguing among ourselves. Here’s the result. Sorry, guys, but that just doesn’t cut it. That’s not a methodology.

Well, so what? The folks at Global Journal are basically arguing that an imperfect listing is better than no listing.

Algoso disagrees. He notes that many organizations are already using the magazine’s ranking for promotional reasons — for fund-raising, that is. So there’s one obvious downside to Global Journal’s rankings. Should donors not give to World Vision because they aren’t on the list? Says Algoso:

As a development professional, I want to see a more efficient market for funding social causes. That’s an economics-y way of saying that I want funds to flow to those NGOs that can best convert them into positive social impact.

There is a great need to improving the evaluation of impact and effectiveness within the humanitarian, or NGO, community. It’s actually quite difficult to find consensus on the best metrics in this field. Many experts are struggling to come up with the most reliable measures of effectiveness.

In the meantime, people like Algoso think subjective short-cuts to rigorous evaluations may do more harm than good — if only by shifting funding away from those who actually are doing a better job toward organizations that happen to have won a media-sponsored lottery.


About Author

Tom Paulson

Tom Paulson is founder and lead journalist at Humanosphere. Prior to operating this online news site, he reported on science,  medicine, health policy, aid and development for the Seattle Post-Intelligencer. Contact him at tom[at]humanosphere.org or follow him on Twitter @tompaulson.