Grant makers today want to see hard evidence that the charities they fund make a difference.
But assessing long-term change requires time, skill, and money, and grant makers’ growing demand for data has not been matched by greater support for evaluation, experts say.
Plus, foundations often take a short-sighted view of assessment, looking for quick answers to justify their grants rather than helping nonprofits learn from data that can show how to better deliver on their missions, says Jamie Austin, senior director of impact and learning at Tipping Point Community, an antipoverty grant maker in the San Francisco Bay Area that helps nonprofits measure results.
“This idea of outcomes and outputs, and then just demonstrating your impact more generally — it’s a touchy subject,” Austin says. Philanthropists should shoulder some of the responsibility for measuring effectiveness, he adds, noting that figuring out whether a program helps people overcome poverty, for instance, “takes a long time.”
For grant makers seeking to achieve the best possible results, spending time and money to help grantees get the most out of measurement pays off, Austin asserts.
“I think too often the question is typically, ‘How are you going to measure this?’ and, ‘How are you going to prove this to us without our support?’ " he says. “And it really should be, ‘We’re jointly invested in this — how are we going to work together to learn from what we’ve done?” That way, nonprofits can apply lessons about what works and what doesn’t to improve their programs and ultimately get better outcomes, or to inform their fields so other organizations can be more successful.
The Chronicle spoke with foundation and nonprofit officials and philanthropy scholars, who shared insights on steps grant makers can take to help organizations measure — and maximize — their impact.
1. Ask about groups’ plans for evaluation, and also how you can help.
Too often, Austin says, foundations ask nonprofits which outcomes they want to achieve and how they plan to gauge success — and the conversation ends there. While those questions are important, it’s more important to follow up with, “How can we help you do that?” he says. The answers to that question should inform and improve the work.
For example, if a grantee with an employment program wants to measure clients’ job satisfaction, foundations could help the organization identify the method that will garner the most useful information, such as holding focus groups rather than just sending a survey.
2. Don’t run away when data shows outcomes you don’t want.
As groups dive deeper into assessment, they may learn they’re not generating the results they — or you — expected. But that’s an opportunity to learn how to improve, not a signal to pull your support, Austin says. “The matter should not be, ‘Let’s defund’; it should be, ‘What did we learn from this? What do we change about our program? What do we change about our target population? What was, maybe, flawed about the evaluation?’ "
For New Profit, a venture-philanthropy organization in Boston that supports social entrepreneurs, open conversations about obstacles are critical for helping grantees reach their potential. New Profit works to build trust with the leaders and organizations it supports so they’ll open up about where they’re struggling, says Doug Borchard, chief operating officer. “It’s hard to solve problems,” he adds, “if you don’t know what they are.”
3. Build long-term relationships with grantees.
If nonprofits are worried about whether you’ll continue supporting them, they may be reluctant to talk with you about failure. “That’s like, why would you go to your bank and tell them that you’re just about to default?” says Kathleen Kelly Janus, author of the book Social Startup Success, a guide for social entrepreneurs looking to grow. Offering multiyear grants is a good way to foster greater transparency, she says.
But that isn’t the only way. Take Tipping Point, which makes grants for only one year at a time but treats grantees like long-term partners. Tipping Point provides unrestricted funding over long periods of time and forges “personal relationships” with the groups along the way, Austin says. That allows the charities to feel comfortable sharing “more than just the good news.”
4. Don’t just fund programs; invest in organizations.
You can’t separate programs from nonprofits that create them, Austin says. The same goes for impact. “Even if you do get excellent results from a program, then that’s not the end of the story,” he says. “You want to be funding great organizations, since great organizations grow and scale.”
To help grantees get stronger, Austin suggests, give unrestricted support for measurement — on your own or together with other foundations — and let the charity figure out the best way to handle it. Depending on the recipient, that might involve activities like training program staff, hiring an in-house monitoring and evaluation person, or paying for consultants.
5. Don’t rush formal evaluations.
Some grant makers push charities into rigorous assessments “really quickly,” Austin says. This tends to lead to findings that are insignificant, or imply that the program or organization isn’t reaching its goals.
If a program is too new or there isn’t enough data to yield an accurate read of the effects, it may be smarter to hold off on the expense of a formal evaluation, says Elijah Goldberg, co-founder and chief operating officer at ImpactMatters, a nonprofit that conducts impact audits of charities. But in the meantime, he says, it’s important for groups to set their desired outcomes “immediately” and devise some kind of system to track their progress.
For example, some small groups use a Google sheet, coding metrics in green, yellow, or red to track how programs are advancing toward their objectives, Janus says.
6. Broaden your perspective.
More and more foundations want to support only groups with “proven programs,” a fuzzy concept that may mean an organization has done some formal evaluation work, Austin says. Remember, however, that it takes a lot of time and effort to understand if an intervention is effective — and that work doesn’t end with one or two assessments. Organizations with seemingly proven models still need support for measurement.
“Think more deeply about what impact means,” Austin suggests. Try to better understand the issues grantees are tackling and where they are with evaluation so you can set more realistic expectations.
7. Get a better handle on data.
Many grant makers are just happy to see nonprofits’ reporting metrics, but it’s important to be able to put statistics in context, Austin says. For example, learn more about what’s known as the “counterfactual” (what would happen if a program didn’t exist) — “not to be critical,” he says, “but just to better understand the context.”
8. Make room for innovation.
Too often, the emphasis on outcomes means nonprofits can’t experiment; they have to deliver results or risk losing funding, says Aimée Eubanks Davis, founder and chief executive officer at Braven, a charity that helps low-income college students get on career tracks. “One of my biggest suggestions in the world of philanthropy is to actually allow people to test and pilot, and that doesn’t happen very often actually.”
But supporting measurement with a focus on learning may be a way to help change that.
As grantees gather data, give them freedom to explore their curiosity, experiment, and learn, Austin advises. “It’s really whatever the funder can do to change the dynamic so it’s not just saying, ‘Prove to me that you’ve used my money well.’ "
