Data-Driven Program Evaluation Hurts Equity Efforts, Say Critics

In recent years a growing number of foundations have fastidiously articulated new program goals to support people of color, people who are LBGTQ, people with low incomes, and others facing barriers to progress.

Many grant makers have tried to increase the diversity of their staffs and have instituted diversity policies for the vendors they hire. Others have crafted detailed, step-by-step strategies that emphasize the support of advocacy and movement building to change social systems that keep certain groups of people on the margins.

But Jara Dean-Coffey says something huge is missing from all of those equity efforts — a rethinking of the way foundations measure success.

Dean-Coffey, director of the Equitable Evaluation Initiative, and a handful of allies say the traditional approach, with a heavy reliance on surveys and statistical data that boil messy social problems down to numbers, is wholly inadequate. It may be even worse than that, says Dean-Coffey; foundations’ overreliance on an outdated yardstick may actually inflame the disparities they’re trying to address.

Equitable Evaluation Initiative

“When we do strategy work, we’re able to think about relationships, about context, about nuance, and about culture,” she says. “And then we turn to evaluation and we strip all of that away because we like certainty. We like things that are concrete.”

Dean-Coffey’s organization originated in 2017 as a project supported with grants from the Ford and Kellogg foundations and the California Endowment. . The group has worked with grant makers and philanthropy networks to help them consider how their evaluation strategies might be counterproductive to their broader equity strategies.

Other foundations have supported the effort with grants. And Dean-Coffey’s work is at the leading edge of a broader movement to inject equity into nonprofit assessment.

Early Efforts

For now, nonprofits eager for change will have to wait. Attempts to rethink the way foundations evaluate programs with an equity approach are in the very early stages.

Still, Dean-Coffey has big ambitions. She would like to completely reorient notions of truth and knowledge when it comes to the measurement of social programs. With partners from the Center for Evaluation Innovation, the Luminare Group, and the Dorothy A. Johnson Center for Philanthropy, Dean-Coffey used the initial foundation grants to develop a framework for foundations to use when they begin to think about equity and evaluation.

In it, she listed a set of orthodoxies — practices that are taken as a given — that have dominated foundation evaluation for decades. One, for instance, is that evaluators should come to the job with certain credentials, like a Ph.D. Another is that the principle user of the information generated by an evaluation is a foundation itself rather than the group of people it seeks to support. Underlying all of the dominant evaluation practices, Dean-Coffey says, is the idea that data is neutral and doesn’t reflect the biases of the people who collect and analyze it.

She points to the “gold standard” of foundation evaluation, the randomized, controlled trial, a tool that came into widespread use by academics to research the effects of large-scale federal programs. The aggregation of a huge amount of data may help solve some problems and allows researchers to make generalized statements about the impact of a program, she says.

But it can fall short when it comes to taking into account the historical or geographical context of, say, homelessness in a city and what happens at the neighborhood or classroom level can fail to register.

Advocating for Partnerships

An equitable evaluation approach, Dean-Coffey says, would be jointly designed by the foundation and the people it supports, and it would take into consideration the feelings, stories, and concerns of those people, not just an outcome that can be measured numerically.

“Dominant culture and patriarchy have shaped everything we see and do in this country, and that includes knowledge creation,” she says. “That philosophical and academic orientation my not be relevant for the work in which philanthropy and nonprofits are engaged in the 21st century. The reality is that there is no simple truth, and truths in a particular moment in time might be more important than other truths. We’re privileging certain types of information over others because it’s cleaner and we understand it.”

That’s not to say that numbers are bad and stories are good when it comes to measuring foundation work, Dean-Coffey says. Rather than rely on one or the other, equitable evaluation entails pausing periodically during the grant-making cycle and questioning whether a numerical measurement or a person’s self-reported feelings alone are giving an evaluator as complete a picture as possible.

Growing Support

Since Dean-Coffey developed the framework, more foundations, including the Irvine, Packard, S.D. Bechtel Jr., and Walton Family foundations have made grants to the effort, putting her annual budget at about $1.4 million. Among other projects, Dean-Coffey and her staff have led discussions at a Walton Family Foundation retreat and coached the Oregon Community Foundation on the approach.

The idea has made some headway in the foundation world. A group of philanthropy-serving organizations, including Grantmakers for Effective Organizations, the Maryland Philanthropy Network, Peak Grantmaking, and Philanthropy New York, have organized a coalition called “Associations Advancing Equitable Evaluation Practices,” to build interest.

The Evaluation Roundtable, a network of evaluators led by the Center for Evaluation Innovation, which for years has pushed to improve the practice of evaluation generally, is looking into the emerging practice of equitable evaluation. It is considering reorienting the group to advance evaluation as a force for equity and social justice, according to Julia Coffman, director of the center.

While these early adopters are a promising sign for fans of the approach, the practice is new, and it is often difficult for foundation leaders to get a clear sense of how it works.

“The idea of using evaluation and learning as part of strategy is fairly radical for foundations, and so taking it further and saying that the purpose of evaluation is to help advance equity is even more of a paradigm shift,” says Coffman. “For evaluators, it’s basically asking us to do everything differently. So the main question is what does this look like in practice?”

Involving Grantees

For the Missouri Foundation for Health, equitable evaluation begins by asking grantees what they want to learn. For instance, when it began to design a set of grants on health-care access, it didn’t publish a fully baked plan and ask grantees to apply for money. Instead it brought together 12 grantees with health-care expertise and close affiliations to people in the neighborhoods the foundation serves.

The group was asked what sort of approaches it wanted to use regarding access to care and what questions it would ask to help determine success. Traditional metrics a foundation might collect could include how many people a health-care organization treated, how long people waited to be seen, and what health improvements patients experienced.

Sometimes hard numbers are important, says Kristy Klein Davis, the foundation’s vice president for strategy and learning. Klein Davis would like to be able to show state health officials that their contraceptive programs, for instance, reduced unintended pregnancies by a certain percentage and resulted in quantifiable taxpayer savings.

At the same time, an equitable-evaluation approach pushes the foundation and its grantees to get a deeper understanding of patient experience. Were people treated with respect, for instance? Did health professionals steer them to products they didn’t want or couldn’t afford?

“I don’t think it’s acceptable to have better health outcomes but have people say they feel bad about the care they’re getting,” Klein Davis says. “We shouldn’t get those better outcomes at the expense of people’s dignity or self-worth.”

Finding Resistance

Philanthropy New York’s chief operating officer, Kathryn O’Neal-Dunham, believes in the promise of equitable evaluation because it’s a way of forcing a foundation to develop a strong sense of its own values, and it can bring grant makers closer to the people they support. But it isn’t for everyone, O’Neal-Dunham discovered.

Last spring the organization hosted about 20 foundation leaders for an all-day learning session that incorporated some aspects of equitable evaluation. But as the date approached, it became apparent that some of the applicants were expecting something different. They wanted to learn how to incorporate a business approach to their grant making and use evaluation to calculate a social “return on investment.”

A challenge, O’Neal-Dunham says, is that fusing equity with foundation assessment is an attempt to dislodge deeply ingrained attitudes and practices. It’s hard, she says, to describe a set of questions or measurements to use because the practice is just emerging.

“There isn’t an equitable-evaluation tool,” she says. “You can’t just teach people how to do this because we’re re-imagining what it is.”

Providing a Nudge

To move equitable evaluation from an exploration into a broadly accepted practice, Peak Grantmaking, a network of foundation grant managers, plans to offer a guide later this year for foundations that are curious. A little help developing a new mind-set might be necessary, says Melissa Sines, the network’s programs and knowledge director, because a prevailing view among foundations that practice “strategic philanthropy” is that progress is linear; a set of interventions will, step-by-step, produce predictable results.

Foundations that evaluate with an equity focus, she says, need to dive into the “nitty-gritty” complexities of the real world. Rather than use large, aggregated data sets, they should break down their metrics as much as possible to get an idea of what is happening with different populations in different contexts.

She says foundations should make the privacy and security of the data it collects from vulnerable populations paramount. Data, she says, is not “neutral,” and information that might be easy for one group to provide might be difficult for another.

“Systems that have been built up to collect information on applications and reports can either be a wall between you and your community or a way to ease the burden on grant makers’ nonprofit partners,” she says.

Ellie Buteau, vice president for research at the Center for Effective Philanthropy, believes that focusing on equity would be major shift for most foundation evaluation professionals. But she suggests that foundations aren’t necessarily wed to their current practices. More than three-quarters of foundations that responded to a 2016 survey conducted by the center said it is a challenge to produce evaluations that are helpful for the field or that provide useful lessons for grantees or meaningful insights for the grant maker itself.

She adds: “It’s not like foundations are saying the current methods they’re using are overwhelmingly successful and helpful for understanding how their work is going.”