News and analysis
September 08, 2013

Nonprofits Anxiously Try to Show Results for New Charity Navigator Ratings

PATH/Gabe Bienczycki

PATH, a global-health nonprofit, is trying to keep its top rating by collecting feedback from the people it serves and publishing evaluation reports.

Path, a global-health nonprofit, consistently wins four stars, the top rating, from Charity Navigator. And it wants to keep it that way.

“We’ve experienced many individual donors who have written us checks and said, 'We went to Charity Navigator, and you were four stars,’” says Eric Walker, a senior adviser at the charity.

He says he and two public-relations colleagues work to ensure that Path has posted all the information on its Web site that Charity Navigator, a charity-ratings group, needs to evaluate whether the group has healthy finances and policies in areas like conflict of interest and donor privacy.

But soon Path, like all of the nonprofits that Charity Navigator rates, will have to do more than that to get good marks. It will also have to show that it measures whether its programs actually work.

Charity Navigator, long lambasted for focusing too much on financial criteria like how much an organization spends on overhead, has thrown its weight behind a growing movement to get charities to become more “results-oriented” and “evidence-based.” Its method: Try to get donors who look at its Web site to consider how effective a charity is when deciding where to give.

A Giant Leap for Many Groups

Organizations like Path are gearing up for the change, which the ratings group unveiled earlier this year.

Mr. Walker, a member of a Charity Navigator advisory panel who is “cautiously enthusiastic” about the new approach, says his group will start reviewing how to keep its four stars, for example, by ensuring that fundraising pitches match the way its programs are described in tax filings and that it collects feedback from constituents and publishes evaluation reports.

But it’s already clear from Charity Navigator’s initial forays into this area that most charities will have to make a giant leap to come even close to scoring well on the new tests—and that a one-size-fits-all approach may not work for all groups.

“The state of the nonprofit sector is exactly what our research indicated,” says Ken Berger, Charity Navigator’s chief executive. “The vast number of nonprofits do not report in a meaningful way on their results.”

His organization has so far reviewed more than 200 family and children’s services charities, awarding blue check marks for each of 14 questions that can be answered positively and red Xs for those that cannot. Many groups did not respond to inquiries, leaving analysts to check the groups’ Web sites.

The result, he says: “A sea of red Xs.”

Charity Navigator dubs its new approach “CN 3.0,” reflecting its evolution from a group that at first examined charities’ finances and then in 2011 began looking at how open and accountable they were. It says its results-reporting evaluations will not affect a charity’s rating until at least 2016. But it has begun posting its assessments online as it plows through the more than 6,000 organizations that it now reviews (a number it hopes to increase to 10,000 over the next few years).

Its evaluations do not tell donors whether a charity is effective, just whether the group is trying to find that out.

“The bottom line here is the bar we’re trying to set initially is anybody who’s reporting anything about their results publicly,” Mr. Berger says.

The 14 questions reflect the sometimes jargony language that will be familiar to anyone who has followed the push for more nonprofit measurement: Does the charity have a plausible “causal logic” (a plan for achieving its goals)? Does it indicate how much of a particular action is required to produce a given result? Does it publish evaluation reports that cover the results of its programs at least every five years?

Not everyone agrees that is a good thing.

In a debate with Mr. Berger in the Nonprofit Quarterly, William Schambra, director of the Hudson Institute’s Bradley Center for Philanthropy and Civic Renewal and a Chronicle columnist, rued the spread of metrics, which “long ago conquered and thoroughly colonized the land of philanthropy” to the “untamed and disorderly” nonprofit frontier.

“It’s on that frontier that we learn to be more than clients passively receiving finely calibrated dosages of interventions by the experts,” he wrote. “We learn rather to be citizens able to manage our own affairs according to our own lights—however dim those lights may appear to the professional dosage-dispensers.”

A Good Nudge

Others say CN 3.0 could give charities the nudge they need to do what they should be doing anyway. Most have never gone through the “internal discipline” of finding out whether their programs work, says Paul Brest, former president of the William and Flora Hewlett Foundation, which is a major donor to Charity Navigator.

Some Hewlett grantees used to say, “We can’t take the time out to do that. We’re too busy running the programs,’” he says. “The analogy I came up with is a pilot saying we’re too busy flying the plane. We don’t have time to develop a flight plan.”

But that argument won’t sit well with some hard-pressed charities that are starting to worry about their Charity Navigator ratings. An official at a four-star children’s-services charity that received a heavy dose of red Xs in a CN 3.0 evaluation says the review “caught us off guard.”

Asking not to be named, she adds: “While we have success measures, we would have to hire a full-time staff member to collect the data outlined in results reporting. Our clients are impoverished individuals who are often transient, and so extensive data collection is very difficult. We agree that it’s not good enough to just do good work these days, but this is a cumbersome measurement tool.”

Surveying Clients

Other groups have fared better. Roca, a four-star nonprofit that helps young men who have been involved in gangs or criminal activities get education and jobs, won eight check marks, much higher than most other charities. (See "How Charity Navigator's New Approach Affected a Social-Services Charity."

Families First, a three-star charity in Atlanta that offers parenting workshops for low-income parents, earned check marks on all six questions in the “constituent voice” section, which evaluates whether charities seek feedback from the people they serve.

Christy Winter, who just stepped down as director of continuous quality improvement, says the nonprofit surveys its clients in ways including comment cards; “QR,” or quick-response, bar codes on mobile devices; focus groups; and interviews with current and inactive clients. The e-mail signature of every staff member includes a link to an online SurveyMonkey questionnaire for clients.

Ms. Winter headed a small department set up in 2009 to collect information from parents that could help the charity improve its services, paid for by capital-campaign money, excess program-service revenue, and grant money that covers overhead.

All Charities Not Alike

In developing CN 3.0, Charity Navigator worked with Keystone Accountability, a consulting firm that has been a major proponent of “constituent voice,” arguing that nonprofits can be more effective if they set up a systematic way of listening to the people they are trying to help.

David Bonbright, Keystone’s chief executive, says his firm hopes to get foundation money to set up a Web site to help charities collect feedback and other data that Charity Navigator is seeking.

Mr. Walker of Path says “constituent voice” could be problematic for a group like his, which promotes innovative global-health technologies and practices. Charity Navigator asks for input from “primary constituents,” but, he says: “It’s not like we’re a service-oriented nonprofit taking care of mental-health patients that walk in the door and get feedback from the people who come in.”

That highlights what some critics say could be CN 3.0’s biggest challenge—applying it to a widely diverse set of charities.

Take Voices for Illinois Children, which did poorly when Charity Navigator evaluated it as a family and children’s services group.

“It was rather glaring to look and see all those red Xs,” says Debra Marillo, the development director. But, she says, the charity should be categorized as an advocacy organization, with recognition that such groups face particular challenges when measuring results.

She says they already run into that conundrum with foundations that are wedded to the “logic model,” or a plan for achieving a specific outcome. “They want to see a beginning and an end and the impact of the work,” she says. But, she adds, “how do you know that you’ve been successful? In the case of advocacy, there is a beginning, there is an end, but that end could take years rather than 12 months.”

Holden Karnofsky, co-founder of GiveWell, which recommends charities after conducting in-depth research on their effectiveness, says he hopes CN 3.0 will prompt more nonprofits to share substantive information. But he says it seems geared to groups that that aim to deliver “evidence-backed programs,” and wonders how that would apply, for example, to an arts nonprofit.

Mr. Berger of Charity Navigator says his group is waiting until 2016 to take results reporting into account when awarding stars partly so it can adjust the criteria, if necessary, as it gets feedback from groups working for different causes. But he says that all charities can measure their work in some way, for example an advocacy group can tell whether it has increased attention to an issue among state legislators or an arts group can measure whether it has improved a community’s quality of life.

While CN 3.0 is a move in the right direction, the performance-measurement push will take generations, says George Mitchell, an assistant professor of political science at City College of New York who studies nonprofit governance. He has drafted a paper calling for a major shift in accounting practices to require charities to report their spending in terms of what outcomes the money has produced.

“It’s not going to be fixed with CN 3.0, or 4.0, or 5.0,” he says. “The problem isn’t Charity Navigator. The problem is the nonprofit sector, that we’re willing to give billions of dollars without knowing whether it’s actually helping anybody."


Send an e-mail to Suzanne Perry.