The latest iteration of Charity Navigator’s rating system aspires to bring social impact evaluation to the masses, using percentage- and star-based scores to help potential donors understand the effects of nonprofit programs. In many ways, the new system makes good on the organization’s commitment to offer a “complete picture” of the nonprofits it evaluates.
Measuring Impact
Unfortunately, the system is inherently flawed: Charity Navigator’s impact rating system subjects the nuanced and complex nature of social-impact work to an overly simplistic process of quantification and cost-effectiveness. So while its statements about impact carry an air of scientific rigor — Charity Navigator claims its ratings are “unbiased” and “objective” — this language belies a highly subjective and idiosyncratic approach to determining and evaluating results.
These conclusions are based on our own observations as well as interviews with staff at nonprofits whose evaluations appear on Charity Navigator’s revamped website.
We spoke, for example, to the chief development officer of a San Francisco Bay Area food bank that received a top score for impact based on cost per meal provided. Our conversation, however, quickly revealed how much Charity Navigator’s evaluation missed. (To speak candidly, all nonprofit staff interviewed for this article asked not to be identified.)
We learned that the food bank is far more than a vendor of low-cost food. It also functions as a central node in a broader service network, which includes delivering food to other nonprofits, promoting nutrition awareness, referring individuals to housing providers, and advocating for policies that tackle the root causes of hunger.
What’s more, Charity Navigator’s rating disregards the quality of the meals the organization offers. As far as the cost-per-meal metric is concerned, the officer told us, “It doesn’t matter if you’re serving a hot meal or just bananas.”
Ultimately, these issues weren’t harmful to the food bank, which comes across in the evaluation as highly effective and worthy of donations. As reported previously in the Chronicle of Philanthropy, some observers believe Charity Navigator is more comfortable giving a nonprofit too much credit than providing an inaccurate negative report. But such reasoning seems at odds with the organization’s goal of offering accurate and reliable information to its users. Even worse, a system that mistakenly credits some organizations is liable to erroneously discredit others.
What the Ratings Miss
We found that such errors come in essentially two forms. The first involves overlooking the breadth of a charity’s work, often by zeroing in on just one aspect of the services it provides. For instance, the director of a nonprofit focused on childhood poverty in developing countries told us that her organization was rated solely for its work in the United States, which accounts for less than 15 percent of its total operations.
Second, even when Charity Navigator captures the scope of a nonprofit’s services, another type of error may occur: The metrics and benchmarks Charity Navigator uses to evaluate impact often fail to account for important variations among organizations ostensibly engaged in the same sorts of activities but with differing goals, community needs, or contexts in which they operate.
For instance, we interviewed representatives of a human-service organization whose impact rating was lowered based on the cost per night of shelter it provides to individuals experiencing homelessness. According to staff, however, a night of shelter comes with a whole suite of wraparound services that, unsurprisingly, increase operational costs.
As one staff member put it, “You can’t really separate out the shelter we provide from all the other stuff we do.” Yet because Charity Navigator’s impact assessment disregards this “other stuff,” the organization’s impact score was downgraded.
In some ways, these errors aren’t surprising. It’s hard enough to conduct quality impact evaluations on a case-by-case basis. Doing so requires thoroughly understanding the nonprofit in question and carefully selecting suitable indicators of organizational performance, often through painstaking data collection, analysis, and interpretation. Conducting impact evaluations on a large scale only compounds these already formidable challenges. An evaluation system that exchanges meticulousness for cursory and imprecise analyses will inevitably mis-categorize organizations, misapprehend their work, and misrepresent their actual results.
Amplifying Nonprofit Voices
If oversimplification results when trying to show impact on a large scale, one clear remedy is to incorporate the nuance of nonprofit work into evaluations. To do this, we propose a simple solution: Charity Navigator should give the nonprofits it evaluates as much voice as possible on its platform. This requires more than an appeals process or an advisory group of nonprofit leaders — two laudable steps the organization has already taken. Charity Navigator should also provide nonprofits with a dedicated and publicly visible space on its website to add critical context to the bare-bones metrics the organization imposes.
For example, the food bank officer we spoke with noted that because recent floods had caused the cost of providing meals to increase, it would help if Charity Navigator offered “a little space [for us] to say that ‘this is what happened’ and ‘this is what we did to meet that need in the community.’” Absent that, she worried, donors might see a drop in the organization’s impact rating and doubt the food bank’s integrity.
Given that nonprofit staff frequently dedicate their lives to improving communities — and often face strong headwinds in the process — they deserve at least the chance to contest their ratings and provide an explanation. As one interviewee stated, “We work really hard to accurately share outcomes from our programs. So at least give us the opportunity to show what we do and respond.”
Certainly, permitting nonprofits to comment on their ratings is not without potential complications. One nonprofit director worried that anything her organization might write in response to a low score could come across as defensive. Alternatively, if given the opportunity, organizations may disclose a surfeit of information and evaluation results, creating a situation in which well-resourced organizations have the capacity to carefully manicure their Charity Navigator profiles while the profiles of organizations with fewer resources go unattended.
While these sorts of eventualities deserve consideration, successfully providing space for nonprofit voices strikes us as clearly in linewith Charity Navigator’s values of leadership, collaboration, equity, fairness, and usefulness.
Although our focus here is on Charity Navigator’s evaluation system, our recommendation applies to the social sector more broadly. Impact evaluation is a fraught topic. It demands the inclusion of as many perspectives as possible, especially from those whose effectiveness is under the microscope. The approach we propose both respects the complexity of nonprofit work and empowers these organizations to be part of the conversation about their impact. We hope, in turn, that this fosters a greater appreciation for the complicated and often deeply subjective nature of evaluating nonprofit work.
Ours is not an entirely novel position. Nearly a decade ago, for instance, researchers Karina Kloos and Daniela Papi argued cogently for the need “to help nonprofit management teams reclaim their position in shaping the conversation on evaluation.” Considering Charity Navigator’s visibility and influence, it’s time to reconsider this argument with renewed urgency.