May 02, 2011

'Three Cups' Controversy Underscores Need for Measurement

It seems fitting that the Three Cups of Tea scandal coincides with the release of the book More Than Good Intentions: How a New Economics Is Helping to Solve Global Poverty, by Dean Karlan and Jacob Appel.

The Central Asia Institute, the nonprofit started by Greg Mortenson, the author of Three Cups of Tea, has been criticized for the apparent lack of any evaluation of the work it did and the schools it constructed.

It is unclear how many schools the group built, how many of these buildings are actually used as schools, and whether the schools performed any better than existing schools.

In a "60 Minutes" exposé on the charity, which aired in April, reporters asked whether the organization had conducted any independent assessments of the effectiveness of its schools in Afghanistan. The Central Asia Institute's governing board replied in a written statement, which "60 Minutes" posted on its Web site:

No. CAI is unaware of any organization qualified to undertake such a study. However, it is clear that the effectiveness of its schools and its programs have been independently assessed by citizens of Pakistan and Afghanistan, in that there are hundreds of requests for new schools in these countries as well as the programs CAI promotes. In addition, the Government of Pakistan has awarded Greg Mortenson the Star of Pakistan related to the promotion of education in that country. Review of test scores of the children at schools built or sponsored by CAI will reveal that the children at these CAI related schools score higher on average than students at other schools. Over the years, many independent observers have visited the schools.

This answer is immediately suspect as it claims that it has increased test scores, even though there are no data to back up the claim.

In addition, there is no shortage of people who are qualified to evaluate its schools. The book More Than Good Intentions describes a number of research projects that provide measures of key factors that lead to student and teacher success. These projects measure the effectiveness of a number of common efforts. They include:

  • Purchasing school uniforms
  • Paying students to attend school
  • Providing deworming pills for students
  • Providing bonuses to teachers who showed up to teach
  • Using cameras to record teacher attendance
  • Breaking students into classes based on their skill level
  • Providing small-group tutoring for struggling students

The book explains in detail how each of these programs was tested to determine if it had its intended impact. The findings can help guide nonprofit and government programs.

These and similar studies would have helped the Central Asia Institute determine which type of assistance had the greatest impact on the educational success of students. But instead of seeking research findings or conducting evaluations of their work, the organization has spent millions of dollars based on assumptions. I'd be curious to find out if there is any study out there that proves that the lack of a building is the biggest impediment to education.

The Three Cups of Tea controversy should be a wake-up call to all nonprofit governing boards. It takes far more than feel-good stories to prove results. The Central Asia Institute is not alone in being unable to prove the impact of its work. Thousands of other nonprofits would find themselves in the same position if their organizations received the same amount of scrutiny.

One of the best things that could come out of this scandal is an increased focus on understanding and measuring the impact of nonprofit projects.