Auditing conservation in an age of accountability

When The Nature Conservancy of California asked Silicon Valley venture capitalist Seth Neiman for a multimillion-dollar contribution to help protect local open space, no one involved had the slightest notion that they were about to step into one of the deepest and most difficult questions in conservation worldwide: how to develop ways of measuring whether their conservation efforts really are achieving their goals. The Conservancy's fundraising team was just trying to raise enough money to buy conservation easements on Mount Hamilton, an island of natural habitat in an encroaching sea of suburbs south of San Jose.

Neiman asked how the Conservancy knew the investment would provide lasting protection for the oak woodlands and the creatures that live there. He wasn't interested in preserving a piece of land for just 30 years. ‘That would be an act of vanity,’ he says. He wanted to know whether it would be protected for hundreds of years.

‘That stumped me,’ says M A Sanjayan, who was called in to answer Neiman's question. In fact, it touched a raw nerve for Sanjayan, the Conservancy’s chief scientist in California at the time and now a lead scientist at the headquarters in Washington. Sanjayan realized that Neiman was probing one of the most important unanswered questions in the science and practice of conservation biology. So he told Neiman the truth. They didn't know. The fundraiser gripped the edge of the table. But Sanjayan pressed on. Maybe with Neiman's help, The Nature Conservancy could find a way to begin answering the question of whether its conservation efforts are really conserving what they say they are for the long run.

That was nearly three years ago. A new ‘measures and audit team’ was formed to design an ambitious programme that will require Conservancy projects to measure whether they are achieving their goals and deploy auditors to verify the results for managers and donors. The idea was field-tested by15 Conservancy projects around the world, and it is now being rolled out across the Conservancy.

Along the way, The Nature Conservancy has rediscovered something that businesses and other non-profits have learned about measuring their performance: in addition to being an important tool for accountability and reporting to donors, measuring and auditing can be powerful tools for continuous learning and improvement. As a result of the audits, some Conservancy projects have abandoned unproductive strategies. Others have realized they need to change strategies to counter new threats.

The Nature Conservancy’s effort is being watched closely in the conservation community and beyond. The Conservancy is under intense scrutiny in the aftermath of a Washington Post series that ‘exposed questionable deals, partnerships with land deals, partnerships with major polluters and other wayward practices,’ according to a nomination the reporters, David Ottaway and Joe Stephens, recently received for a Goldsmith Prize awarded by the Joan Shorenstein Center on the Press, Politics and Public Policy at Harvard University. Those problems centre on governance and tax questions – who makes decisions, who benefits, and how to avoid conflicts of interest – rather than performance and results for conservation. As a result, the efforts described in this article have lately taken a back seat to a wide-ranging internal review of the organization’s governance structure.[1]

But governance and performance are, of course, intimately connected, and The Nature Conservancy’s focus on measurable results is seen by many donors as a hallmark of the organization. If its governance problems are not cleaned up, the organization’s performance could be seriously derailed, but even if it fixes the governance issues, the Conservancy will ultimately be judged on its performance.

The Nature Conservancy is not alone in trying to figure out how to measure its results and open its conservation books to outside review, in much the same way as its financial statements are publicly reported. Four other major conservation organizations – Conservation International, the World Wide Fund for Nature/World Wildlife Fund, the Wildlife Conservation Society and the African Wildlife Foundation – have joined the Conservancy in forming a Conservation Measures Partnership, coordinated by Foundations of Success, a non-profit that helps conservation groups measure success. Together they are working to establish a common framework for auditing their work – in much the same way that publicly traded companies use generally accepted accounting principles (GAAP) to report financial results.

But the conservation organizations promise to go beyond just sharing their financial accounting, like businesses do. They are preparing to share their management accounting, the performance measures that businesses usually keep close to the vest. And that is a brave step beyond business as usual.

There may never be a final answer to the vexing question, ‘Are we conserving what we say we are?’ But measuring and auditing could provide a dynamic method for conservation projects to continue to hone their strategies in a changing world. And that, it turns out, is precisely what Neiman had in mind when he asked his fateful question.

Backing up success stories

Sanjayan chose the Cosumnes River Preserve in California as the first project to audit because it had a documented history of conservation activities and a wealth of data from research and monitoring. It was also well staffed and funded. If this project couldn’t be audited, no project could be.

The measures and audit team was made up of scientists and managers from around the Conservancy, an ecologist from the Cosumnes preserve, and a staff member from Foundations of Success. They arrived at the preserve on a summer morning in 2001 with the goal of answering the simple question: ‘Are we conserving what we say we are here?’ But it soon became apparent that the answer would not be so simple.

For many years The Nature Conservancy has had a simple metric for success: ‘bucks and acres’ – how much money was raised and how much land was protected. By this measure, the Cosumnes River Preserve was a success. It protected one of the last remaining riparian oak forests along the last undammed river in California, home to Sandhill cranes and Chinook salmon.

The Cosumnes also had a great tale to tell about learning and adaptive management. One day, a staff ecologist was studying aerial photographs when he noticed that an oak forest that had grown up in a former farm field was missing in a 1985 photo, taken the year the river broke through a levee during a flood, carrying sand and debris on to the field. The seeds of the forest, he realized, must have been planted by the flood. Here they were painstakingly planting trees around the preserve with mixed results and the river had done the work much more effectively. This soon led to a plan to breach the levee in other spots and use the river to enlarge the riparian forest.

But the measures and audit team wanted to know the numbers behind the story. They dug down into the computer spreadsheet that Conservancy projects are supposed to use to document the health of their ecosystem. Much like a business’s balance sheet, which tracks assets and liabilities, the spreadsheet tracks critical factors such as the size, condition, and landscape context of species populations and habitat, and the threats they face. The team found that many of the entries were just best guesses about whether to classify the health of a species such as salmon or a habitat such as the oak forest as ‘good’, ‘fair’ or ‘poor’. The judgements were entered into the spreadsheet simply by clicking on one of the categories without providing an underlying scientific rationale or data. Dan Salzer, the designer of the system, called this ‘mouse-based monitoring’.

The Cosumnes River Preserve had a lot of data that confirmed the project was making progress on important fronts, such as improving salmon habitat. But the information wasn’t easily accessible in one place where auditors could verify it and managers could use it to compare the effectiveness of their approaches. They asked the project to go back and fill in the books, this time documenting the scientific evidence and rationale behind each of their judgements.

Over the next few months, it took nearly 600 hours for Preserve staff to prepare a full report that could be audited by the rigorous standards the team demanded. Preparing for the audit turned out to be at least as valuable as the audit itself. The project realized it needed to devote more attention to emerging threats, such as groundwater pumping, which is drawing water from the riverbed for nearby farms, vineyards and suburbs. This demonstrated two essential things that a donor like Neiman is looking for. The riparian forest preserve was likely to survive well into the future. And the project was learning from measuring its results and could change. The detailed audit set a new bar for the Conservancy, unfortunately not one likely to be cleared by many projects.

There be dragons

The measures and audit team then decided to test audit a project at the opposite end of the spectrum: Komodo National Park in Indonesia, a classic besieged ‘paper park’. If the audit worked in Komodo, it might work anywhere.

The park is home to the famous Komodo dragons, giant monitor lizards that live on remote islands fringed by some of the most diverse and beautiful coral reefs in the world. While the dragons have done well in the park, the reefs have not. They have been severely damaged by local fishermen and roving bands of pirates, who use cyanide to stun big groupers for the live fish markets in Singapore and Hong Kong and aquarium specimens for the worldwide market. And when the more valuable fish are gone or the fishermen are in too much of a hurry, they simply toss dynamite or homemade bombs overboard and harvest the fish that float to the top, leaving a rubble field of destroyed coral under the waves.

When the team met with Peter Mous, the project ecologist, on a diving boat sailing among the islands in late 2001, he told them he knew what needed to be done. Stop the bombing and overfishing. ‘It’s not rocket science,’ he said.

Mous had little patience for the complex ecological models that were part of the audit. There was urgent work to be done. And the Komodo project could easily demonstrate its success. The number of bombing incidents had dropped dramatically after the Conservancy donated a patrol boat to the park. After two years there was a 4 per cent increase in live coral cover and the decline of large groupers had stopped.

But the team persisted. Mous had simple ecological models for the coral reefs, mangroves and sea grass beds that are the main elements of the marine ecosystem. The team insisted on drawing up a more detailed conceptual model of what the project was trying to protect and the threats it faced. And as Mous worked with the team to fill in the threats and trace them to their sources, they all began to see the value of adding the human element to the model.

One of the causal chains ran right to the heart of the Muslim faith of many of the local communities. At least once in their lives, devout Muslims are expected to make the hajj, a pilgrimage to Mecca. This requires cash. And the quickest way to raise cash is blast fishing. One way to break the chain, the team discussed, might be a micro-loan programme to help pilgrims make the hajj without resorting to destroying the reefs.

This was one small link in a complicated project that involves encouraging local aquaculture; educating villagers about the value of intact reefs, mangroves and sea grass beds as a steady source of fish; and an ambitious long-term goal of improving park management through a concession that will channel increased tourist revenue back to the park rather than to the central government.

Two years later, the Komodo project is still struggling. Offering micro-loans for the hajj, it turns out, may be meddling too much. The patrols have kept blasting down. But there have been pitched battles between park rangers and heavily armed poachers, resulting in many arrests and two deaths.

Tourism in Indonesia has plummeted after September 11, the terrorist attack in Bali and the SARS epidemic. Income for the park has declined. Some in the Conservancy have questioned how much longer the organization can afford to subsidize the park.

That is where the audit proved useful. It didn’t solve the park’s problems, but it documented that the project was making a difference in an extremely difficult situation. And recently the government approved the concession plan that the audit had underscored as a key to a long-term solution for Komodo. This showed that an investment in measuring and auditing results could provide a better basis for donors – and in this case, even management higher up in the organization – to continue to invest in the project.

Trial by fire

The measures and audit team came away convinced that they were on to something worthwhile. So they worked to distill the lessons and simplify the process. Having a conceptual model was essential. It forced people to articulate their intuitive understanding of their projects as an explicit set of causes and effects that could be shared and analysed critically with others. The team knew that the spreadsheet that they provided to projects had to be more user-friendly. At the same time, however, this conservation balance sheet had to be complex enough to be scientifically rigorous.

The balance between simplicity and complexity would remain in creative tension as the team asked 13 other Conservancy projects to test the measures on their own. One of those projects was Lake Wales Ridge, the dry sandy spine of Florida. Fire is an essential part of this highly fragmented ecosystem, which is home to 13 endemic species. The Conservancy’s goal there is to buy subdivided parcels that have not been developed yet and to return fire to the ecosystem.

To that end, the Conservancy had spent US$170,000 a year over five years outfitting a fire strike team to help local agencies manage prescribed burns. Fire crews throughout the area have been trained to set and control the periodic fires that are necessary to maintain the oak shrub patches favoured by native species like the Florida scrub jay.

The area burned had doubled. ‘Sounds like a fabulous success,’ Mary Huffman, the director of the project, told a recent gathering of Conservancy scientists. But it turns out that’s not enough. Even with the Conservancy’s help, the area burned each year fell short. ‘It’s not about capacity,’ Huffman said. The people who are being trained to manage burns are still afraid to set them because they could lose their jobs if a fire got out of control.

The project had missed an important link in the causal chain. Rather than training fire fighters, project managers should have focused on higher-level decision makers, convincing them to provide incentives for their crews to set more prescribed fires.

‘I’m mad as hell,’ Huffman said. But, she added, if they hadn’t gone through the audit, they might not have learned they needed a course correction. Once again, learning was at the centre of the audit. Such failures or shortcomings are not likely to be the kinds of stories that organizations tell donors when they are seeking more funding, but donors are increasingly interested in hearing them. Many understand that if the groups they fund don’t admit to failing some of the time, they are either lying or not trying hard enough.

However, some scientists worry that this is still no better than trial and error learning. ‘It’s what kids do when they touch the stove,’ says Frances James, an ecologist at Florida State University, who is also on the board of governors of the Conservancy. James has advised the audit team to encourage projects to design scientific experiments into their work. Conservation cannot be done everywhere at once. This affords an opportunity for simple, inexpensive, controlled comparisons of conservation techniques.

Open source code for conservation

While The Nature Conservancy was testing these new methods, other conservation groups were also trying to figure out how to measure and audit their own work. By the time the Conservation Measures Partnership came together, they had developed their own ideas, but there was a surprising convergence of thinking. They decided their first task would be developing common ‘open standards for the practice of conservation’ modelled on ‘open source code’ for software developers – programming that can be used and adapted by anyone. These are the steps that any conservation project should complete and could be audited on:

• developing an explicit causal model of ecosystem and threats;
• developing a plan of action and monitoring and evaluation targeted at important threats and information needs;
• implementing actions and monitoring and evaluation;
• analysing their data on an ongoing basis;
• adapting their strategies based on new information;
• communicating what they learn;
• iterating, or continuously repeating the cycle of planning, acting, measuring, learning and improving.

These may seem painfully obvious. But many projects neglect one or more of these important steps. In a recent survey, the William and Flora Hewlett Foundation found that fewer than one out of ten projects it supported had an explicit causal model of how they hoped to affect the systems they were working in. And while many organizations measure what they do, few measure the effects of their work.

There are two obvious reasons for this. People intuitively respond to problems by doing what they know how to do. As the old saw goes: a surgeon will likely recommend surgery. And it is far easier to measure what you do and call it success than it is to take a hard look at whether you actually made a difference.

Activity-based accounting

The partners are also testing activity-based accounting on a variety of projects to see if the cost-effectiveness of different conservation approaches can be compared. Businesses have long used activity-based accounting to break out the costs of different parts of their operations. Rather than simply lumping all of the phone bills or rent or travel costs together, they are assigned to different activities.

Conservation projects have been slow to adopt this powerful accounting tool, in part because it requires more complicated bookkeeping, but also because it is hard to assign costs when one works on multiple strategies at the same time. But at the behest of the Gordon and Betty Moore Foundation, the Wildlife Conservation Society, Conservation International and the World Wildlife Fund are implementing activity-based cost accounting in their multi-partner Gabon Parks Project.

Last year, the president of Gabon announced the creation of 13 new national parks covering close to 10 per cent of the country. The three conservation organizations were already working in Gabon. But the Moore Foundation wanted them to collaborate to help Gabon manage the new parks. And it wanted comparable, measurable results.

This is a key element in measures and auditing. If the partners can generate comparable activity-based accounting, it could provide the basis for estimating a return-on-investment for different conservation actions. Then the circle would be complete and the potential of measures and auditing to drive strategic conservation efforts could be realized. Donors would be able to shop around and invest in those groups that were achieving the best rate of return in terms of results for the money donated. After all, conservation organizations, like other non-profits, are working with other people’s money to achieve their shared goals.

Julie Packard, a prominent donor to conservation as a board member of her parents’ foundation, the David and Lucile Packard Foundation, finds the move toward auditing exciting. In recent years, the Foundation has given millions of dollars to The Nature Conservancy and other groups.

‘The organizations need to feel a responsibility not only to the ecosystems they are seeking to protect,’ Packard says, ‘but to donors and investors who are expecting that the lands will be protected as healthy ecosystems in the 500-year time frame. It’s a lofty goal. And it’s an imperfect science. But it’s the right question to be asking.’

Learning to audit, auditing to learn

This year, the Conservation Measures Partnership has begun sending members to help conduct pilot audits on each other’s projects. The results of these audits have not yet been publicly released. The Partnership is still experimenting with the process. But this is akin to Adobe sending auditors to Microsoft and it could signal the beginning of a new era of transparency, accountability and collaboration among conservation organizations that until now have largely competed on the urgency of their appeals rather than their actual results.

In the wake of the Washington Post series on The Nature Conservancy and increasing scrutiny of governance and accountability in non-profits, many conservation groups and foundations have become less eager to talk in public about their shortcomings, let alone their failures. It will be interesting to see whether the focus on measuring and auditing results and the learning that comes with it has gained enough critical mass to weather the storm and emerge again at the heart of accountability, where it should rightfully be.

As the centre of gravity has shifted from the pioneering efforts of organizations like The Nature Conservancy to the collaboration of different conservation organizations around the world, the truly transformative power of Seth Neiman’s provocative question is just beginning to be realized.

‘I didn’t realize it was such a loaded question,’ Neiman says. But now, he believes, ‘if this is done right, it will change the whole discussion about conservation.’ Instead of seeing conservation just as a good cause, he says, people will start asking, ‘What are your results?’

Lessons from business about measuring and auditing

Auditing is routine but not foolproof. In business, being audited is not a sign of something wrong, although it can uncover problems. Auditing is meant to ensure accurate reporting of results, but it does not guarantee it.

Investors make comparisons. The GAAP or generally accepted accounting principles evolved to allow investors to understand and compare financial results from different companies in different businesses. It provides a common yardstick for evaluating investments. Why shouldn’t conservation investors have the same choice?

Differences don’t disappear. Even within the GAAP there is room for discretion, as there should be. Software businesses are different from furniture makers. Conservation projects are naturally as varied as the biodiversity they seek to protect. But those differences don’t negate the usefulness of common standards and principles for measuring results.

Performance measures are strategic tools. People will perform to what you measure if you reward them for success. But ‘diagnostic measures’, which tell you if you are doing what you set out to do, can be double-edged swords. You need to regularly check that the direction you are going is still the right one.

Pure information is needed too. That’s why you need at least one and perhaps a few ‘interactive measures’ or early warning measures. You don’t reward people for these measures. That would distort the information you gain from them. These typically measure things outside of your activities to give you information about the strategic environment. In business, this could be market share or customer satisfaction. In conservation, this could be something as simple as the weather, which can affect your project but is out of your control. The less confident you are about your strategy, the more you need pure information.

Too many measures spoil the brew. The purpose of performance measures is to conserve management attention by not scattering attention among too many things and to focus attention on the most important stuff. That is why limiting the number of things you measure is crucial. Businesses usually track no more than a dozen different performance measures and sometimes only half a dozen.

Balance measures with other ‘levers of control’. In addition to ‘diagnostic measures’ and ‘interactive measures’, businesses have two other essential ‘levers of control’ – ‘belief systems’ that tell people in your organization what it is you do and ‘boundary systems’ that tell people what it is you don’t do. These are essential because measures inherently create incentives that drive people to perform. Belief and boundary systems guide them in the right direction and stop them from going over the edge.

This isn’t rocket science. Some scientists will object that ecology is even more complicated than rocket science. And they are right. But this is not just about science and understanding ecosystems in more detail. It is about managing conservation, a human enterprise reflecting human values. Science can and must play an important role. But the purpose of measuring and auditing conservation efforts is to help managers and donors make better decisions. And simple business tools can help, if they are kept simple enough, so that they effectively measure the progress conservation efforts are making.

Just do it. Don’t wait for the holy grail of conservation measures. It will not be coming. Get out there with a plan and do the work. Measure, audit and learn.

Notes
1 I have written about this ongoing shakeup on www.ConservationNews.org and continue to cover it there.

Resources
Robert Simons (2000) Performance Measurement & Control Systems for Implementing Strategy: Texts and cases Prentice Hall. Contributors: Antonio Davila & Robert S Kaplan.

Foundations of Success, 4109 Maryland Avenue, Bethesda, MD 20816. Tel +1 301 263 2784 Fax +1 703 764 0179 Email info@FOSonline.org Website www.FOSonline.org

Jon Christensen is editor-at-large for Conservation in Practice, where an earlier version of this article first appeared, and he writes a daily a web log for the magazine at www.ConservationNews.org. He can be contacted at jonchristensen@nasw.org

Christensen was a Knight Professional Journalism Fellow at Stanford University during the 2002-2003 academic year, in which he focused on evaluating environmental conservation projects. He would like to thank Antonio Davila for his incisive crash course on performance measurement and Laura Arrillaga for deeply probing results in her course on philanthropy in the Graduate School of Business at Stanford University and the Chocolate Group for exposure to the challenges and rewards of interdisciplinary rigour.