Outcomes, not implementation, predict conservation success

Abstract

To use more effectively the limited resources available for conservation there is an urgent need to identify which conservation approaches are most likely to succeed. However, measuring conservation success is often difficult, as it is achieved outside the project time frame. Measures of implementation are often reported to donors to demonstrate achievement but it is unclear whether they really predict conservation success. We applied a conceptual framework and score-card developed by the Cambridge Conservation Forum (CCF) to a sample of 60 conservation activities to determine the predictive power of implementation measures versus measures of key outcomes (later steps in the models defined in the CCF tools). We show that assessing key outcomes is often more difficult than quantifying the degree of implementation of a project but that, while implementation is a poor predictor of success, key outcomes provide a feasible and much more reliable proxy for whether a project will deliver real conservation benefits. The CCF framework and evaluation tool provide a powerful basis for synthesizing past experience and, with wider application, will help to identify factors that affect the success of conservation activities.

Introduction

Given the limited resources available for conservation it is imperative that funding is channelled towards those approaches that can be proven to be successful (Sutherland et al., 2004; Ferraro & Pattanayak, 2006). Considerable effort has recently been devoted to devising frameworks that help to monitor the progress and success of conservation projects (Sutherland et al., 2004; Hockings et al., 2006; Pullin & Stewart, 2006; Mace et al., 2007; Kapos et al., 2008; Salafsky et al., 2008) but an issue that still hampers these attempts is the fact that the effects of many conservation projects on target populations or habitats only become measurable well beyond the time frame of the usual project cycle (Kapos et al., 2008). Long-term monitoring of targeted species or habitats would be desirable but is often not practical. An alternative approach is to identify predictors of success that are both easier to measure and serve as reliable surrogates for long-term data on the changing status of conservation targets.

The Cambridge Conservation Forum (CCF), a network of conservation organizations and researchers based in and around Cambridge, UK (CCF, 2009), has developed a project evaluation tool (Kapos et al., 2008). This draws on a conceptual framework (Kapos et al., 2008) that is itself an elaboration of the idea of results chains for conservation interventions (Salafsky et al., 2001, 2008). Together the framework and evaluation tool enable us to investigate the link between intermediate steps (such as activities, outputs and outcomes) and the ultimate success (conservation effects) of a project. Here we use a preliminary dataset to test whether measures of project implementation (as typically reported to donors) are useful predictors of conservation effects, or whether measuring subsequent outcomes is a better alternative in terms of how easily the measures can be derived and how well they predict longer-term effects on conservation targets.

Methods

CCF framework and evaluation tool

The CCF framework and evaluation tool begin with the recognition that conservation projects characteristically involve several different types of activity that may each have different outcomes and appropriate measures of success. Based on existing categorizations of conservation action (Salafsky et al., 2002, 2008) and as a result of consultation within CCF, we defined seven broad categories of conservation activity (Table 1) that together encompass most of the work that CCF members and other conservation organizations undertake. Typically conservation projects include several of these activity types (Wilder & Walpole, 2008; Kapos et al., in press).

Table 1 The seven broad categories of conservation activity adopted by CCF as the basis for its evaluation framework. Together these encompass most of the work that CCF members and other conservation organizations undertake.

For each activity type the CCF framework provides a conceptual model of the likely relationships between its successful implementation in a project and conservation impact, making explicit the linkages that are often assumed (Kapos et al., 2008). The models are simplest for species and site management (Fig. 1) and more complex for the other five activity types (e.g. Figs 2 & 3). All models include implementation stages, leading on through a series of outcomes (how an intervention affects the conservation problem of interest), to conservation effect (project-scale effects on the likelihood of persistence of populations or habitats of conservation concern). In all but the two simplest models we identify key outcomes that provide the platform for reducing threats to and/or improving the responses of conservation targets to those threats. For livelihoods-related projects the key outcome is the abandonment of the relevant damaging practices; for policy work, it is the implementation of the policies or legislation promoted; for education and awareness raising, a change in behaviour by the intended audience; for capacity-building, increases in the quantity and/or quality of conservation action; and for research, the application of research results to conservation practice.

Fig. 1 The conceptual models for species and site management that form the framework for evaluating the success of conservation actions are directly analogous and have the simplest structure, with implementation of management actions leading directly to reduction of threats to the conservation target and/or an improvement in its responses to those threats (outcomes), and a consequent conservation effect. The components corresponding to implementation, outcomes and effects are separated by horizontal lines.

Fig. 2 The conceptual model for enhancing and/or providing alternative livelihoods, in which the links between implementation and conservation effect are more complex, involving more intermediate outcomes: livelihoods-related activities aim to develop sustainable management regimes for important natural resources, encourage the development of alternative sources of income, and/or provide direct incentives. If adequate support for the targeted practices exists or is developed, and the necessary conditions and resources for their implementation are in place, they can bring about a change in practice. Finally, if the new practices have been appropriately designed this may lead to reduction in threats and/or improved responses of the focal ecosystems, habitats, species and/or populations, which may ultimately lead to a conservation effect (a change in status). This may or may not be accompanied by real improvements in livelihoods and/or attitudes but these will not affect conservation status, except by affecting the targeted behaviours or practices. Each of these more complex models includes a key outcome (in this case, change in practice, marked with a star) on which threat reduction and/or improvement of responses depends. The components corresponding to implementation, outcomes and effects are separated by horizontal lines. The hexagons show indicative (but not exhaustive) links to other activity types.

Fig. 3 The CCF framework’s model for capacity building efforts, which aim to improve either individuals’ skills or aspects of organizational capacity, or both. If those improvements are in fact applied in conservation, they should lead to more and/or better conservation action to address the problems of interest (the key outcome), leading through reduced threat and/or improved responses to improved status. The components corresponding to implementation, outcomes and effects are separated by horizontal lines. The remaining models in the CCF framework (Table 1) show analogous sequences (Kapos et al., 2008; CCF Outputs, 2009).

The score-card–style evaluation tool developed by CCF has a single questionnaire for each conservation activity type. Each questionnaire is based on the appropriate model and comprised of carefully worded questions that work progressively through implementation and outputs to outcomes and conservation effects (CCF Outputs, 2009). Each question offers four ordered answers reflecting increasing levels of achievement, which can be scored on a 1–4 scale. There are also further unranked options for when insufficient information is available; these are worded to distinguish between cases where relevant information is anticipated and those where information is not being collected at all.

As an example, the question about the key outcome in the questionnaire on livelihoods-related activities is: ‘Has the target audience reduced their use of the damaging practices addressed by the project?’ and the answers offered are: (a) No; (b) Some individuals have reduced their use; (c) Most individuals have reduced their use and/or some have abandoned them entirely; (d) Most individuals have entirely abandoned damaging practices; (e) Too early to tell but information will be available; (f) Not assessed, necessary information neither available nor anticipated. Answers a–d reflect increasing reduction of damaging practices and e and f deal with missing information.

Trial application

To test which type of measures (implementation vs outcome) offers the most practical approach for assessing conservation performance (in terms of how easily they can be derived and how effectively they predict long-term effects), the CCF questionnaires were used by conservation professionals to self-evaluate a set of 60 activities from 26 projects. A few of these individuals were involved in the development of the framework and questionnaire but the majority were not. Their answers to the questionnaire were based on the best available evidence, which ranged from the experience and opinion of project managers, often supported by anecdotal evidence, to quantitative data (e.g. from surveys of agricultural practice and statistics on trade).

We used the completed questionnaires firstly to compare how easy it was to answer questions about implementation, outcome and conservation effect. Secondly, for each evaluated activity, we calculated a mean score for the answers to questions at each of these three levels, restricting the analysis of the outcome level to the section on key outcomes (as defined above). We then used these average scores to test the extent to which measures at the two lower levels (implementation and outcome) were good predictors of the activity's conservation effect. This analysis reduced the sample to the 22 activities for which full information was available for all three levels. However, this reduction in sample size did not introduce bias to the sample: projects with full information did not differ from incomplete projects in their scores for implementation or outcome (medians (ranges) for implementation 3.3 (2.8–3.5) vs 3.5 (3.0–4.0), Wilcoxon test W = 452.5, n = 22 vs 37, P = 0.473; for outcome 3.0 (2.0–3.4) vs 2.5 (2.0–3.0), W = 91.0, n = 22 vs 11, P = 0.245).

Results

As expected, questions were less likely to be answered the further they were along any given model (and thus closer to conservation effect; see Fig. 4 for a detailed example from capacity-building). This was the case irrespective of the type of conservation activity being undertaken (Fig. 5). Questions on key outcomes, despite being less readily answered than those about implementation, were still much easier to answer than those on actual conservation effects.

Fig. 4 The frequency among 12 completed capacity building questionnaires of substantive responses (i.e. a–d, which reflect the level of achievement rather than lack of information) to key questions. The key outcome, in this case increased quantity and/or quality of conservation action, is marked with a star (as in Fig. 3). This pattern is consistent to a greater or lesser extent across activity types.

Fig. 5 Variation in frequency of substantive responses (i.e. a–d, which reflect the level of achievement rather than lack of information) at different points within the CCF framework. Across different activity types the occurrence of substantive responses to key questions regarding implementation (stippled bars) is always higher than for questions about key outcomes (light grey bars), which usually have more substantive responses than questions about conservation effect (dark bars).

With regard to what best predicts conservation effects, we found that the degree of implementation of a given activity was a poor predictor of its conservation effect (Spearman rank correlation: ρ = 0.077, n = 22, P = 0.734; Fig. 6a). In contrast, the extent of achievement of key outcomes was a much more powerful predictor of the effect of an intervention, even in this small sample of activities (ρ = 0.668, n = 22, P < 0.001; Fig. 6b). This result is not linked to any tendency in the methodology to promote linkage between adjacent levels; we tested for a close link between implementation and outcome scores and found none (ρ =0.060, P = 0.792).

Fig. 6 The relationship between the overall conservation effects of activities in 22 conservation projects and (a) the level of their implementation and (b) the degree to which key outcomes are achieved. Answers from the relevant questions have been given simple numerical values for plotting, with 1 being the lowest level of achievement and 4 being the highest. Symbols are sized according to the number of overlapping points, up to a maximum of four.

Discussion

Our results show that measures of key outcomes can potentially serve as powerful predictors of real conservation effects, and thus project success. While assessing outcomes is more difficult than simply reporting on implementation, measures of implementation were unable to predict success and are thus of little value in tracking conservation impacts. This is important, given that most donors have until now typically only required reporting at the level of project implementation. In contrast, because outcomes are influenced by some of the many external factors that affect the linkages between activity and impact, they provide a more reliable basis for assessing project success. Although in future it would be desirable to validate fully this conclusion using a larger sample size, targeting project monitoring and reporting efforts and resources more strongly towards outcome-level measures is likely to improve the assessment of conservation achievements.

The CCF framework and evaluation tool provide a standardized approach to improving on current reporting practice through the identification and assessment of outcomes for different activity types. The CCF tools build on and complement existing approaches for using conservation experience to inform future conservation actions, including The Nature Conservancy's 5-S Framework (TNC, 2003), logical frameworks and results chains (Salafsky et al., 2001) and tools for implementing them, such as Miradi (Miradi, 2007), conservation audits based on the CMP's Open Standards (CMP, 2004; O'Neil, 2007), threat reduction assessment (Salafsky & Margoluis, 1999), and the scoring and evaluation frameworks developed for assessing agri-environment schemes (Carey et al., 2003, 2005), protected area management effectiveness (Hockings et al., 2006; Stolton et al., 2008), and interventions by zoos (Mace et al., 2007). The CCF tools also complement the growing drive for evidence-based conservation (Sutherland et al., 2004; Sutherland, 2005) by providing both a means to assess the effects of whole projects as distinct from single management interventions and a way of helping to ensure that the data needed for evidence-based conservation are collected.

So far, practitioners who have used the CCF tools report taking no more than a day to assess a project, making it difficult to argue that it is too time-consuming to conduct such assessments (given that projects normally represent several person-years of effort). Although further validation of our conclusion about the usefulness of assessing key outcomes is desirable, we believe that our results provide clear evidence that such measures have the potential to allow practitioners to predict the downstream effects of their projects, even when these will only be realized beyond the project timeframe.

The dataset presented in this paper is relatively small and it was not possible to test the robustness of our general conclusion for each activity type independently. The dataset is also relatively lacking in unsuccessful projects, mirroring the more general patterns in the availability of information about conservation actions that fail (Redford & Taber, 2000). A larger database of assessed projects would not only allow us to address this issue but also to investigate several other interesting questions, such as whether the success of different activity types depends on particular conditions, which combinations of activities provide the best synergies, and which resource allocation strategies are most successful. We therefore strongly encourage practitioners to take advantage of the CCF tools, which are freely available online (CCF Outputs, 2009). Results presented here and feedback from practitioners who have trialled the tool already suggest this will help organizations plan interventions, monitor their progress, and obtain meaningful, early measures of their long-term effectiveness. Expanding the database of evaluated projects would also help us build a much-needed resource with which to start asking precise and quantitative questions about what determines conservation success.