Data at the Speed of Life

David Graham/iDE

Can pregnant women in Zambia be persuaded to deliver their babies in hospitals or clinics rather than at home? How much are villagers in Cambodia willing to pay for a simple latrine? What qualities predict success for a small-scale entrepreneur who advises farmers?

Governments, foundations, and nonprofits that want to help the world’s poor regularly face questions like these. Answers are elusive. While an estimated $135 billion in government aid and another $15 billion in charitable giving flow annually to developing countries, surprisingly few projects benefit from rigorous evaluations. Those that do get scrutinized in academic studies often don’t see the results for years, long after the projects have ended.

IDinsight puts data-driven research on speed. Its goal is to produce useful, low-cost research results fast enough that nonprofits can use it make midcourse corrections to their programs.

“Everything we do is around helping social-sector organizations make decisions about what works, what doesn’t work, and how to improve,” says Esther Hsu Wang, one of four founding partners at the San Francisco-based nonprofit.

The quartet, who met at Harvard’s John F. Kennedy School of Government, brought a multiplicity of expertise and experience to IDinsight, collectively holding bachelor’s and advanced degrees in economics, business, medicine, and public administration (see sidebar). CEO Neil Buddy Shah is an M.D. and did poverty research at the Massachusetts Institute of Technology, as did co-founder Andrew Fraker. Ms. Wang and her husband, Paul Wang, have Harvard MBAs and experience with major management-consulting firms.

courtesy of IDinsight

They launched IDinsight in 2011 with the help of a $100,000 grant from an anonymous donor. Five years later, demand for their work is robust. With offices in San Francisco, New Delhi, Nairobi, and Lusaka, Zambia, the organization employs about 70 people and has some 40 evaluation projects underway for clients like the World Bank, Unicef, the Bill & Melinda Gates Foundation, the nonprofit venture fund Acumen, the Children’s Investment Fund Foundation, and the Zambian Ministry of Health. IDinsight has analyzed corruption in India, online education in Rwanda, and small-scale agriculture in West Africa.

The Gates Foundation is a big supporter. “They are on the cutting edge,” says Richard Caldwell, senior program officer for measuring, learning, and evaluation at the world’s largest philanthropy. Gates last year gave IDinsight a two-year grant of nearly $1.5 million and asked it to choose three of the foundation’s agricultural grantees and work with them to test their programs.

In India, for example, Gates and its partners are introducing new varieties of stress-tolerant rice by giving farmers free samples — currently, five kilograms per farmer. Is that too much? Too little? No one knows. Giving away smaller samples would mean reaching more farmers; giving away bigger ones might strengthen the case for adopting new varieties. “Without doing the research, you’re guessing,” Mr. Caldwell says.

Rapid Response

IDinsight calls this kind of research “decision-focused evaluation,” which sets it apart from traditional monitoring and evaluation (M&E) and academic research. M&E, experts say, is mostly about accountability and outputs — how many training sessions were held, how much food was distributed, and so on. Usually, it occurs after a program is complete. Academic studies are typically shaped by researchers’ desire to break new ground and publish on topics of broad interest. The IDinsight approach aims instead “for contemporaneous decision-making rather than for publication in the American Economic Review,” says Ruth Levine, who directs the global development program at the William and Flora Hewlett Foundation.

A decade ago, Ms. Levine and William Savedoff, a senior fellow at the Center for Global Development, wrote an influential paper entitled “When Will We Ever Learn? Improving Lives Through Impact Evaluation.” They lamented that an “absence of evidence” for the effectiveness of global development programs “not only wastes money but denies poor people crucial support to improve their lives.”

Since then, impact evaluation has come a “huge distance,” Ms. Levine says. She attributes the growth to investment from foundations like Gates and Hewlett and the pioneering work of economists at M.I.T.’s Abdul Latif Jameel Poverty Action Lab, of which Mr. Shah and Mr. Fraker are veterans; Yale economist Dean Karlan’s Innovations for Poverty Action; and the Center for Effective Global Action at the University of California at Berkeley.

Social scientists have used randomized, controlled trials — the gold standard of evidence, used to evaluate medical interventions — to do systematic reviews of development projects involving cash transfers, education, microfinance, and water and sanitation. “We, as a global community, have a huge amount of new evidence to draw on,” says Ms. Levine.

Testing Markets

IDinsight grew out of that work. Its first major project was aimed at determining the willingness of households in rural Cambodia to pay for simple latrines, which reduce diarrheal diseases. It was commissioned by iDE, a global development organization, with backing from Gates and the Stone Family Foundation.

To test the market, IDinsight ran a set of randomized, controlled trials. Nearly 90 percent of Cambodian consumers, they found, were unwilling to pay the market price for a latrine, which ranged from $35 to $55. But when they were offered a 12-month loan, half of these same consumers agreed to pay $50. Armed with these findings, iDE boosted its budget for latrines and has sold more than 210,000 since.

In Zambia, government officials wanted to know if free “mama kits,” provided to pregnant women on the condition that they deliver their babies in a hospital or health clinic, would drive up an institutional delivery rate that sat at just 43 percent nationally and lower in rural areas. With funding from the British government’s Department for International Development, the Clinton Health Access Initiative, and Unicef, IDinsight tested various mama kits costing $4 to $50.

The government “wanted to know, should they continue with mama kits, and at what price point is it effective?” says Ms. Wang. IDinsight found that a low-cost kit containing a cloth garment, diaper, and blanket was enough to increase institutional deliveries in rural areas by 44 percent. Zambia’s Ministry of Health has since added the kits to its list of essential medicines.

Another iDE/IDinsight experiment in Zambia studied “farm business advisers,” essentially privatized versions of agricultural extension agents. These advisers help small-scale vegetable farmers raise their income by providing access to better seeds and fertilizer, offering loans, and providing marketing advice.

iDE wanted to know how best to select advisers: Should they be elected by their peers, chosen by village elders, or employed based on independent criteria? If the latter, what criteria? IDinsight built a recruitment tool incorporating factors such as social connectedness — determined by how many names farmers had in their cellphones — so that iDE could recruit advisers who exhibited characteristics associated with high performance. “It’s not a game-changing piece of research by any means, but it moves the needle,” says Chris Nicoletti, a research and evaluation specialist at iDE.

IDinsight would like clients to pay for its evaluations to demonstrate the demand for evidence-based evaluations of aid projects. For now, most are financed by donors, the biggest supporters being four grant makers: Gates, the David Weekley Family Foundation, the Mulago Foundation, and the Pershing Square Foundation. IDinsight has also advised a handful of foundations, including Porticus and the ELMA Foundation, about how to measure results, and it has helped GiveWell, the charity evaluator associated with the effective-altruism movement, with its ratings.

Not surprisingly, IDinsight also evaluates itself, to see what work makes the most difference. “The demand for what we’re doing is far bigger than we can service,” Ms. Wang says. “We wish others were doing this work as well.”

Actually, others are. Innovations for Poverty Action recently created the Goldilocks Initiative to do what it calls “right fit” evaluations leading to better policy and programs, according to Thoai Ngo, who leads the effort. Its first clients include GiveDirectly, which facilitates cash transfers to the extreme poor, and Splash, a water charity.

courtesy of IDinsight

Resistance to Data

All this focus on data has generated pushback. Many nonprofits don’t have the resources to do rigorous research, according to Debra Allcock Tyler, chief executive at Directory of Social Change, a British charity that provides training, data, and other resources for social enterprises.

“A great deal of the time, data is pointless,” Allcock Tyler said last year at a London seminar on data and nonprofits. “Very often it is dangerous and can be used against us, and sometimes it takes away precious resources from other things that we might more usefully do.”

A bigger problem may be that the accumulation of knowledge does not necessarily lead to better policies or practices.

“People often trust their experience more than a systematic review,” says Ms. Levine of the Hewlett Foundation. IDinsight’s Esther Wang agrees. “A lot of our frustration is looking at the development world and asking why are we not accountable for the money that we are spending,” she says. “That’s a waste that none of us really feels is justifiable.”