Over the past two decades, concern about a lack of transparency at nonprofits—first, about their finances, and later, their effectiveness—spurred the creation of watchdog organizations that seek to provide potential donors with accurate information.
“The fundamental reason nonprofits exist is to provide a public benefit,” says Ken Berger, chief executive of Charity Navigator, one such watchdog organization. “If you can’t measure in a meaningful way whether you’re doing that, we’re sort of at a loss.”
Here’s a breakdown of how several watchdogs collect and present information, plus advice from their representatives about what makes for good nonprofit measurement.
Charity Navigator
Charity Navigator is a nonprofit that evaluates other nonprofits using a four-star rating system based on 39 measurements in three different areas: finance, governance, and results reporting.
The organization limits its scope to nonprofits with more than $1-million in annual revenue and has reviewed nearly 2,000 charities so far, with the goal of reaching 10,000 by the end of 2016. About a quarter of the charities it reviews receive four stars, while two-thirds of charities receive two or three stars.
Charity Navigator maintains a donor advisory list of nonprofits that have come under investigation, are the subject of a lawsuit, or have filed 990 tax forms that don’t make sense. The group organizes some of its findings into top-10 lists, such as “10 Charities in Deep Financial Trouble” and “10 of the Best Charities Everyone’s Heard Of.”
What it measures. Charity Navigator looks at a nonprofit’s data for basic fiscal management practices. While some organizations argue that watchdogs should focus less attention on the percentage of a nonprofit’s budget that goes toward its overhead expenses, Mr. Berger believes that figure still matters.
“We don’t agree that it’s a meaningless metric,” Mr. Berger says. “When a charity has less than 30 percent of its money in programs and the rest is going to overhead, it’s either a scandal or wildly mismanaged.”
The watchdog places a premium on accountability and transparency. It looks for nonprofits that have policies in place to protect whistleblowers and prevent conflicts of interest, are open on their websites about their proceedings, and run their 990 tax forms past their boards before filing them.
Having a strong board that is truly independent is key, according to Mr. Berger. “We think that’s really a critical area that needs to be looked at,” he says.
Results reporting is the groups newest area of interest—and it’s often the cause of greatest concern among nonprofits, who fear that Charity Navigator is unfairly comparing their results to those of very dissimilar organizations. But Mr. Berger emphasizes that his nonprofit measures the reporting of results, not the results themselves, using criteria that adhere to the “common sense of a reasonable person.”
“We’re looking at how each nonprofit uniquely, individually, in its own way, measures its results,” Mr. Berger says. “We don’t impose a required particular metric. They tell us how they feel they should be measuring their results.”
Collecting information about results reporting has proved difficult so far, Mr. Berger says, because most nonprofits do not publicly report their results. “We have anecdotal information that the reason is that the vast majority of nonprofits do not have a performance-management system in place,” he says. “We’re not blaming the nonprofits alone for this. Part of the reason they don’t have this in place is that the vast majority of funders are not funding the infrastructure required to build a performance-management system.”
In addition to more funds, Mr. Berger believes more training about how to meaningfully report results will be necessary. “If we don’t do that, we will have jargon-eering,” he says. “He or she who does the best marketing wins. That’s the inflection point I think we’re at right now.”
Charity Navigator is following its own advice: The organization’s currently building a performance-management system to evaluate itself.
Best practices. Among the top mistakes nonprofits make, Mr. Berger says, is not taking the 990 tax form seriously enough. Too often that form is filled out by an accountant who specializes in individual taxes and fills it out incorrectly.
“It’s a crime to falsify that document, but a lot of times it’s not given the attention and seriousness it deserves,” he says.
Another problematic practice he observes is nonprofits entering into expensive contracts with for-profit telemarketers who conduct fundraising, then take most of the money raised.
“We find that many of these contracts are onerous,” Mr. Berger said. “When their donors find out about it, they become outraged. It can damage trust in the nonprofit sector as a whole. A lot of charities that get our zero-star rating will have something odd like that going on. Perhaps it’s not intentional. It’s not a scandal. It’s really just poor decision-making.”
GuideStar
For Jacob Harold, chief executive of GuideStar, the question of measurement is a basic—and crucial—one.
“It is fundamental to the future of the nonprofit sector that we wrestle with this question,” he says. “If we’re not being thoughtful about the results we’re getting, then we’re not getting better.”
What it measures. GuideStar, which was founded in the mid-1990s, does not offer ratings, but collects information directly from nonprofits and distinguishes among them by the level of information they provide.
They give a “bronze” label to groups that provide basic qualitative information, “silver” to those that submit quantitative information about finances and operations, and “gold” to nonprofits that offer qualitative information about their programs. The group will soon also award “platinum” labels to groups that provide quantitative information about their results.
“The way we provide the data shows an implicit set of opinions about what sorts of things really matter,” Mr. Harold says. “Qualitative and quantitative are both really important. There should be no numbers without stories and no stories without numbers.”
GuideStar creates a framework for nonprofits to share information about their work. It then feeds that information into databases that other organizations, like foundations, can use to keep track of nonprofits.
Best practices. Mr. Harold recommends that nonprofits use a logic model to think about their work. Also called a program theory, this is a simple flowchart that maps the sequence of events intended to effect a change.
“Most nonprofits don’t have good outcome data,” he says. “Most nonprofits are measuring something, but it’s not always the right thing. We need to help them learn.”
While Mr. Harold acknowledges that many nonprofits lack resources to measure results as thoroughly as would be ideal, he believes there’s usually something even cash-strapped groups can do to improve their impact evaluations, including reporting outcomes, not just outputs, and applying what they do know to improve future programs.
“We should not expect the perfect system,” he says. “What we should expect is that nonprofits are clear about what they’re trying to accomplish. Clarity is a big piece of this.”
GiveWell
GiveWell is a nonprofit that evaluates a relatively small number of charities whose causes the group’s analysis finds most likely to accomplish the greatest good and that it determines would benefit the most from additional donor support.
The organization makes annual recommendations of nonprofits it believes are most deserving of donations. In addition, its Open Philanthropy Project looks for the best ways for major donors and foundations to make a difference both within and outside of traditional nonprofit work, including through political lobbying, research, and startup projects.
What it measures. GiveWell does not focus on nonprofits’ financial reporting, which is often relatively easy to measure, says Jake Marcus, a research analyst at the organization. Instead, it looks for groups that provide the biggest return on donors’ investment, regardless of the percent of funding it uses for administrative purposes. “We have charities doing good work that have a lot of overhead as well,” Mr. Marcus says.
GiveWell looks for groups that can “convincingly and rigorously tell how a donation to that charity will improve the lives of the targets of that program.”
It also analyzes what the effects of additional donations would be. GiveWell argues that some nonprofits’ limiting factors are related to nonfinancial resources. One example is Smile Train, a surgery charity that GiveWell believes needs more doctors, not more cash. Other charities, by contrast, plan to use more money in ways unrelated to donors’ intentions or in programs that GiveWell believes are not its most effective.
Best practices. The organization considers randomized, controlled trials to be the gold standard method of measuring impact, and it recommends that groups work with academics to carry out evaluative research, says Eliza Scheffler, a research analyst at GiveWell.
“Charities are often reporting on a few anecdotes of a few participants in their program,” she says. “The more rigorous the data collection, the better.”
Transparency is important to GiveWell, which looks for charities that report the good with the bad. “The charities that are measuring their work carefully enough to recognize when there are issues, and reporting them and addressing them—that actually builds a lot of confidence in donors,” Ms. Scheffler says.
GiveWell says it strives to follow this advice by publishing its own errors and corrections on its website. “If we saw more charities adopting a mistakes page, we’d be excited about that,” Mr. Marcus says. “People are trying to tell a compelling emotional story but are scared to talk about the things they’re still working on, their setbacks, and their mistakes. But that’s where most of the learning comes from.”
Mr. Marcus recommends nonprofits make clear what the benefit of additional donations would be. “If we increase the budget of a charity by some amount, where would that go?” he says. “What would they do that they weren’t doing before? I think reporting budgets or activities that allow people to say ‘I know where a new donation’s going to go’ would be really helpful. We ask all our charities that question.”