March 07, 2013

Clever Ways to Evaluate Programs Without Spending a Lot

When it comes to well-worn rock-star-as-diva clichés, it doesn’t get much better than the classic story about the rock band Van Halen. The group members, so the story goes, were so focused on trivial matters that they had a special clause in their contract that their backstage bowl of M&Ms had to have all of the brown ones removed. If not, Van Halen would not only demand new candy but reserved the right to cancel the entire show. Classic diva move, right?

Actually, it turns out that Van Halen’s M&M clause was not trivial at all. In fact, buried in the middle of a lengthy document, that clause was a smart way to ensure that the stage crew had read the contract thoroughly and understood the safety requirements for the band’s legendarily over-the-top performances with so much heavy equipment.

In effect, the presence of brown M&Ms served as what we in the measurement and evaluation business call a proxy indicator: It may not measure your intended impact directly, but it tells you very important information about the likelihood that this impact will occur.

At time- and cash-strapped charities, proxy indicators can be a way to get a relatively quick understanding of whether a program is moving in the right direction. Charities can use the results for their own information and share them with grant makers and donors, who like to see evidence that the programs they support are making a difference.

Sometimes these indicators are fairly intuitive. For example, a program that aims to improve health through better nutrition may not be able to give people medical checkups. However, the connection among diet, behavior, and health is well understood, so getting information on how many vegetables people are eating is an easy-to-measure proxy indicator for nutrition and long-term health.

Other proxy indicators are more complex, especially when dealing with more abstract concepts such as poverty. The Grameen Foundation’s Progress Out of Poverty Index is a good example. Collecting data on income is notoriously difficult, especially in the developing world, but the organization found that doing a survey of basic assets in a home was a reliable way to come up with a poverty score.  For example, in Kenya, some of the proxy indicators are the source of lighting fuel, the kind of flooring, and the number of frying pans.

Proxy indicators are useful for several reasons:

  • They’re cheap. They take relatively little effort to set up and relatively little effort to collect the data necessary to draw a conclusion. For example, Progress Out of Poverty Index simply has to arrange for workers to go door to door to determine specific things people have in their homes.
  • They’re easy to analyze. An indicator is only as good as the ability of the person collecting and reporting it. It is always better to have a system that doesn’t require a high degree of specialized training. In the health-care example, measuring blood pressure and cholesterol requires specialization, but doing a survey of vegetable intake does not.
  • You can use the data to dig deeper.  This is especially true with proxies that consist of an index of indicators, such as Progress Out of Poverty, which make it easy to analyze if there are any meaningful geographic or demographic trends. For example, you could compare two provinces or the differences between men and women.
  • They’re reliable. Proxy indicators are useful provided they reliably connect to an important outcome. For example, the Progress Out of Poverty Index is based on years of research in each target country to determine what assets are best to survey.

In the case of Van Halen and the brown M&Ms, there were no in-depth studies connecting candy to concert accidents; however, it turns out that the group had some evidence it had a good proxy indicator one night when there were brown M&Ms in the bowl and a stage later collapsed.

Has your nonprofit had success with a proxy indicator? How does it work?