One of the most common challenges facing nonprofits is meeting the expectations of grant makers that demand comprehensive impact evaluations, often without providing adequate resources to conduct those evaluations.
But even given adequate funds and the support of an external evaluator, impact evaluations can place a substantial burden on already-stretched organizational leaders and program staff.
It doesn’t have to be this way.
First, despite grants that call for “demonstrating impact,” many grant makers don’t actually want or need absolute scientific proof of effectiveness of every program. And, second, most programs use strategies that have already been studied by other groups, and thus an enormous amount of research already exists on the effectiveness of similar strategies.
Nonprofits can use those to show strong evidence of the effectiveness of their own programs. In other words, someone else already paid for your research!
For example, tracking the long-term health of a program distributing insecticide-treated bed nets in a given area may not be practical for a small organization. Luckily, a significant amount of rigorous research has been conducted on how and under what circumstances bed-net distribution is likely to affect health, such as reducing rates of malaria.
Once you establish that your program is well designed and executed in strategically distributing the bed nets, it is not necessary to reprove its effectiveness.
Of course, using existing research means that you are relying on someone else’s methods and conclusions. Before linking your program’s outcomes to the existing research, it is crucial to consider the following questions:
Does your program contain the key elements of the research you intend to cite?
Many research studies test the effectiveness of very specific program elements, so it is important to analyze exactly which elements are and are not applicable to your program’s design. A study evaluating the effectiveness of bed-net distribution, for example, is only applicable to programs using a particular type of bed net and method of distribution.
Is the evidence applicable to the conditions in which your program is operating?
The context in which a program operates is an important consideration. To be applicable to your program, the research should test populations that are generally similar to your target population in terms of socioeconomic characteristics (this is known as the “generalizability” of the study).
For instance, a job-placement program for recent immigrants in Cleveland is probably dealing with a different set of factors than a program working with women in rural India who have no formal education.
Is the evidence based on multiple studies?
A greater variety of methodologies and data sets means less opportunity for bias and error. It is best not to rely on one study of a single program or a small number of programs. Reviews that use data from a large number of studies to draw conclusions, such as those done by the International Initiative for Impact Evaluation, provide the most robust evidence.
Is the evidence produced by a credible third party?
Linking your program to studies of questionable rigor risks undermining your credibility. Ensure the validity and impartiality of data by making sure it’s from credible external evaluators or academics.
And because you are using a credible third party, you should make it clear in any reports that you are using external research and link to the original research papers.
Is the evidence representative of the full body of research?
Studying the social sector is messy and complicated, and there are often conflicting findings about a given topic. Avoid using studies that support your preferred conclusion but do not represent the full body of research on a topic.
Of course, many issues are far from settled, so it may not be possible to represent the full body of research through one overarching conclusion; in these cases, it is important to be well informed about the larger debate to speak to why you believe the evidence you are citing is credible and applicable.
Where to Find Evaluations
There is no central repository of nonprofit evaluations, but GiveWell reviews programs in a variety of subject areas. And several other sources offer find high-quality research in specific topics, such as education, health care, international development, and social programs.
Google Scholar is a powerful search engine for published research on programs connected to academic institutions, and sometimes charities publish their research on their Web sites.