The “impact” buzzword flies across emails, event titles, and websites for nonprofits and other organizations striving for positive social change — and that’s not always a good thing.
Don’t get us wrong: We love the concept of impact, too, as in making a real difference for people in need. But the unrelenting focus on impact — from boards, donors, clients, policy makers and others — can lead organizations ultimately to overinvest in data that do not credibly or responsibly answer questions about impact and underinvest in data that help guide management and improve operations. Getting the right data, then, is the challenge.
What’s more, an unrelenting focus on impact often puts too high an expectation on any one study of impact: from one place, at one point in time, from one implementation, and so on. Evidence can and should make a difference in policy making, but it is critical to remember that gathering evidence is like putting a mosaic painting together: No stone on its own tells us much, but piece by piece a larger picture comes into focus.
To help with this growing challenge, we urge nonprofits to focus on four criteria when deciding what data to gather in their evidence strategy. They should ask: Is the evidence credible, actionable, responsible, and transportable?
Making sure each of those ingredients is considered can help any organization develop the right evidence strategy, as we explain in detail in The Goldilocks Challenge: Right-Fit Evidence for the Social Sector.
We named the “Goldilocks Challenge” after the well-known fable: Goldilocks, coming upon the three bears’ house in the woods, struggles to find a chair, bowl, and bed of the “just-right” size; most of the options are too big or small. In the same way, many organizations have developed monitoring and evaluation systems that are too big and cumbersome, generating too much data with too little relevance; others adopt data-collection efforts that are too small, yielding insufficient information about program performance and impact.
To develop just-right data strategies, organizations ought to first have a clear understanding of what they are actually going to do and what that will change in both the short run and the long run. They should also ask what else must happen outside of their control for them to have the impact they desire.
No answer is ever perfectly complete, of course. But with some ideas in place, leaders can develop specific data tactics that will help them demonstrate accountability to their donors and others involved in an issue as well as provide their management with timely data that allows them to learn, improve, and operate efficiently.
Here’s how to consider each of the four essential principles for gathering just the right data:
Credibility. Collect high-quality data and analyze it accurately. Make sure that measures are valid, the data collection is reliable, and the analysis is conducted appropriately. This may seem obvious, but consider how concepts like “health” and “education” will be measured differently in an array of settings.
Reliability and credibility are critical in everything from measurement scales to surveyor training. Poor-quality data can provide misleading evidence that can result in bad decisions and lost opportunities for improvement.
Analysis also needs to be conducted credibly. For example, before-and-after comparisons of a new health-care approach may not yield accurate estimates of impact if there is no way to know what would have happened without the new approach. (Creating a measure for the “what would have happened” is typically the motivation for conducting randomized, controlled trials and is essential to measuring impact.)
Actionable. Collect data you can commit to use. Just because organizations have access to unprecedented volumes of data at low cost today doesn’t mean they can use it all effectively. In fact, gathering a lot of data often leads to poorer decision making. The pressure on all organizations to appear data-driven is part of the problem. In making decisions about what data to collect, we advocate three critical questions: Is there a world in which the specific data will change behavior? Is there a specific action that we will take based on the findings? Do we have the resources necessary to carry out that action? Only a “yes” to all three questions warrants pursuing that data.
Responsibility. Ensure the benefits of data collection outweigh the costs. Analyze the specific expenses and likely rewards of data-collection activities to find the right fit. Costs include the direct expenses of data collection and the real opportunity cost: To what other activities could the time and money spent on data collection have gone?
Responsible data collection also requires considering the time of constituents and minimizing risks to them, such as following proper protocols for research on individuals. At the same time, failure to use money and other resources to collect data can be irresponsible, as it could contribute to the persistence of flawed, ineffective programs. There’s no magic formula to getting the cost-benefit tradeoff right, but organizations need to commit to evaluating it.
Transportability. Collect data that generate knowledge for other programs. Make sure that key lessons gleaned from monitoring and evaluation are transmitted to enable others to build more effective programs. That means thinking about the context in which the findings are most likely to hold true. Of course, organizations must be willing to share their findings so we can test what ideas really do work well in different programs and places.
Applying these principles thoughtfully will help organizations build data-collection systems that are the right fit and provide grant makers and others with credible, actionable data on program performance. With better data, organizations and those that support them can make a bigger impact — and respond appropriately, quickly, and responsibly to ever-increasing pressures to do more with less.
Mary Kay Gugerty is a professor at the Daniel J. Evans School of Public Policy & Governance at the University of Washington. Dean Karlan, is a professor of economics at the Kellogg School of Management at Northwestern University and founder and president of Innovations for Poverty Action, a nonprofit. They are authors of “The Goldilocks Challenge: Right-Fit Evidence for the Social Sector.”
