With 200,000 Nonprofits Rated, the New Charity Navigator Aims High, Falls Short

Last year, 11 million people visited the website of Charity Navigator, a leading player in a decades-long drive to help Americans make smart choices when donating. Charity Navigator rates organizations as critics review movies and restaurants, bestowing up to four stars. Its operating principle: The ratings will direct more charitable giving to the most effective groups. With that, more good will get done.

This week, Charity Navigator is introducing the latest tweak in a ratings system that in theory represents the fullest expression of that ideal. Some 50 metrics now inform what it says are comprehensive, holistic evaluations of a large number of the country’s more than 1.5 million nonprofits. Most notably, it promises to rate a group’s impact and determine which groups do the most good for the dollar.

This is a far cry from Charity Navigator’s early days, when it rummaged through tax filings to determine an organization’s fiscal worthiness and assess administrative bloat, or overhead — evaluations that critics said unfairly cast highly effective groups as profligate spenders. The latest changes strip overhead and fundraising-expense metrics from the ratings.

“Now we’re actually offering this opportunity for nonprofits to tell their whole story,” says Laura Andes, chief program officer. “I think that’s kind of exciting.”

Many praise Charity Navigator for broadening its evaluation. “They have been trying to do the right thing for a long time,” says Jacob Harold, former CEO of GuideStar and co-founder of Candid, a clearinghouse for nonprofit information. “And that is something that we should celebrate.”

Some see a ripple effect, with Charity Navigator’s impact rating giving nonprofits incentive to do more to evaluate outcomes — and grant makers to pay for that work. Groups that see others get a rating won’t want to be left behind, says George Mitchell, a City University of New York scholar who has researched charity ratings. “It helps everybody move to this new culture of more data driven-work.”

Yet so far, three years after incorporating impact into its ratings, Charity Navigator’s ambitions outstrip what it can do — indeed, what any ratings service can do. Despite marketing that touts “a comprehensive analysis of charity performance,” its scores remain a work in progress, with stars assigned based on limited data for all but a few groups.

Related Stories

1. Opinion

   Charity Navigator’s New “Impact Score” Tells Us Little About a Nonprofit’s True Value

2. Advice

   6 Tips for Raising Money to Measure Impact

3. Opinion

   Why We Need Better Ways to Help Donors Understand the Cost of Solving Big Problems

Its database now houses more than 200,000 organizations, up from 9,000 just a few years ago, a leap made possible by automated tax-filing collection from the IRS. The vast majority of groups, however, are rated solely on financial and governance data. Only about 6,000 groups are scored in at least three of the four assessment areas: finance, leadership, culture, and impact. Just 549 nonprofits are scored in all four.

Going under the hood of any organization’s rating can reveal a lot of missing parts. Last month, Charity Navigator promoted to its users 26 four-star nonprofits working on relief and recovery following the wildfires in Hawaii. Yet 16 groups earned that top rating based solely on finance and accountability metrics. Only four had been evaluated for impact.

One of those, the American Red Cross, received a top score for impact based solely on 2018-19 data from its blood-donation program. That may not be ideal data, Andes acknowledges. But she says Charity Navigator is more comfortable mistakenly crediting a nonprofit with impact than giving it a negative report that’s wrong. “My foundational thing is that most nonprofits are impactful,” she says. “The scale is tilted in nonprofits’ favor.”

Website visitors who dig deeply into a charity’s profile might discern the limits of the ratings. Yet many are what Ken Berger, a former CEO of the organization, describes as “drive-by donors.”

“Charity Navigator is trying its best to do what it can with the limited tools that are currently available,” says Berger, who led the organization from 2008 to 2015. But, he adds, it should be careful not to promise too much. “There needs to be a more sober, honest conversation as to what we are truly offering and what we can do with the tools that are currently available.”

Others say Charity Navigator ratings, even if backed by full data, present donors with unwarranted certainty about a group’s effectiveness. “It takes many years of analysis looking at different organizations and trying to figure out which are having the most impact,” says Laurie Styron, head of CharityWatch, a ratings group that evaluates nonprofits based on accountants’ analysis of accountability and financial measures.

A Lack of Trust

One big challenge to Charity Navigator’s growth: Nonprofits aren’t lining up to hand over data. Small organizations in particular already struggle to provide what grant makers require in applications and follow-up reports. Big organizations that have enjoyed four-star ratings for many years seem reluctant to contribute, Andes says, perhaps out of concern they’ll lose that top honor.

Courtesy Charity Navigator

Also, Charity Navigator is still remembered — and reviled by some — for its earlier focus on overhead. “There may be some lack of trust,” acknowledged Charity Navigator CEO Michael Thatcher. He says groups may wonder: “What happens when I give you the data? Are you going to hurt me?”

Even if large numbers of groups begin to ante up with information, assessing charity impact at scale has stumped the field for years. “Understanding the impact of nonprofits — that is the Holy Grail,” says Katherina Rosqueta, founding executive director of the Center for High Impact Philanthropy at the University of Pennsylvania. “How do we understand whether nonprofit organizations are making the kind of difference that they and funders seek?”

A few ventures are making headway. GiveWell, started in 2007 by former hedge-fund analysts, does exhaustive analysis to uncover groups whose work has proved effective by gold-standard research. But it focuses only on groups working on global health and poverty, of which it recommends just a handful each year.

The nearly decade-old Impact Genome Registry is developing a common language and standards for measuring the outcomes of social-good projects. The registry so far has developed more than 130 standardized outcomes, which it defines as a measurable change in a condition, a community, or an individual’s behavior over 12 months. Drawing on academic research and data from 2.2 million social programs, it has also come up with an average cost for each outcome — $7,169, for instance, to successfully help a prisoner reenter society, or $345 to provide regular access to food for someone in need.

Using those outcomes, it has verified the impact of more than 4,000 programs, from a mobile food pantry in Marietta, Ga., to a dental care program for low-income patients in Palm Springs, Calif. Institutional donors like the Bill & Melinda Gates Foundation, Google, and Mastercard pay to use its data to analyze their portfolios, aggregate impact, and publish results, according to its founder Jason Saul.

Nonprofits, meanwhile, use the data to study and improve their programs and promote impact to donors. In its marketing to donors, Big Brothers and Big Sisters of Toronto pitches the registry-verified 91 percent efficacy rate of its mentoring program and its below-average cost.

Rating Harvard and Greenpeace

Charity Navigator is the only nonprofit-monitoring group that attempts to measure impact. Candid — which was known as GuideStar when it launched in 1994 — has shied from rating the quality of any group’s work, says former CEO Harold. “The diversity of organizations is so immense,” and there’s not yet a way to equitably measure the impact of both a Harvard and a Greenpeace, an art museum and a local animal shelter.

Still, Candid believes results are valuable and encourages organizations to report their impact. In 2016, it introduced its highest transparency award, the platinum seal, for groups that share quantitative information about their outcomes.

So far, Charity Navigator has outcomes measures for just 2,200 groups. Generally, these focus narrowly on the cost-effectiveness of a program and assessment of a counterfactual — how a community or other set of people would fare without that program. And it only rates organizations that provide a direct service that can be counted — meals for the homeless, say, or eye surgeries for individuals with vision problems. Nonprofits working in less tangible cause areas, like the arts or advocacy, have no impact score.

Charity Navigator is working to grow the pile of data that feeds its algorithms on impact and the other three assessment areas. It has partnerships with the Impact Genome Registry, Giving Green, and other evaluators. It’s also working with Candid, whose brand name rivals that of Charity Navigator among donors. Using data that nonprofits provided Candid, Charity Navigator now rates 30,000 organizations based on “culture and community” measures that reflect whether they seek feedback from constituents and how they pursue diversity, equity, and inclusion.

Within a year, Charity Navigator expects that it will nearly double, to 10,000, the number of groups with scores in at least three of the four assessment areas. Andes predicts it could have impact ratings for 4,000 organizations.

Perhaps most importantly, Charity Navigator is working hard to change its reputation among nonprofits. Whereas Charity Navigator once kept nonprofits at arm’s length to underscore its independence as a rating service, it now runs trainings for charities. When it makes changes to ratings, it invites public, academic, and industry scrutiny and consults with an advisory group of nonprofit leaders.

“The organization now understands that it can have much greater value and impact when it listens to and engages with 501(c)(3)'s and others that are part of this philanthropic world,” says Shannon McCracken, CEO of the Nonprofit Alliance. Once a fierce Charity Navigator critic, McCracken later joined the organization as a fundraiser and now sits on its board.

A Boon to Nonprofits?

Two San Francisco Bay Area charities reflect the mixed response to Charity Navigator’s new ratings. White Pony Express, which collects donations of food, clothing, and toys for food banks and other nonprofits serving families in need, got a four-star rating last year after providing data to be rated in all assessment areas, including impact. It has promoted that endorsement to big donors and on social media, and the group has since seen an influx of gifts from outside the region, says its executive director Eve Birge. “I have seen first-time donations from across the U.S. — especially on the East Coast — that I feel are tied to Charity Navigator.”

Operation Freedom Paws, meanwhile, is frustrated with its two-star rating. The group matches service dogs with veterans and first-responders who have PTSD and other brain-trauma disorders, as well as children with disabilities. It gets low scores on two accountability measures gleaned from its tax filings: Its four-member board includes only two people who work outside the organization, not three, as required to get a high mark. And an independent third party does not audit its finances.

A good impact score would help offset those dings. Charity Navigator, however, doesn’t rate impact for service programs of this kind. “They don’t have a way to measure us,” says founder Mary Cortani, an Army veteran.

Operation Freedom Paws assesses its impact through surveys of participants in its program, which runs a minimum of 48 weeks and provides counseling and other care. Clients with brain-trauma injuries leave with fewer thoughts of suicide and reduced use of medication, the surveys indicate. But unlike Candid — which gives Operation Freedom Paws its highest rating — Charity Navigator doesn’t include self-reported results data in its charity profiles.

“There’s no way for us to put that up there,” says Cortani, who was named a CNN Hero in 2012 for her work.

Charity Navigator CEO Thatcher acknowledges the difficulties that incomplete ratings pose for charities. “We’re still lagging in the more sophisticated elements of the ratings. But we’re playing catch-up as fast as we can.”

The organization, a nonprofit itself, is seeking more funding to accelerate the build-out of its ratings. In the meantime, Thatcher says, “we know we’re half-pregnant or something. And that’s just not a good place to be.”