Chapter Four - Generating, collating and using evidence for conservation

Summary

Generating, collating and using scientific evidence is key to effective conservation. We illustrate this with examples from medicine and conservation, and in more depth with an example from our own studies, in which we assess the methods (overpasses and underpasses) used to reduce habitat fragmentation and mortality of bats caused by roads. Evidence is defined as the results of quantitative tests of interventions that directly address conservation effectiveness against clearly stated goals. The results show that methods used in the past were rarely tested quantitatively and when tested were rarely effective. We discuss why it can be difficult to get scientific evidence accepted and used routinely in conservation policy and practice. Finally, we discuss how evidence gathering and use can be improved in mitigation projects.

4.1 Introduction

Does scientific evidence really matter in conservation? In this chapter we will argue that generating, collating and using scientific evidence is key to effective conservation, illustrated by a case study from our own work: how to get bats to safely cross roads. We tell the story of bat ‘gantries’ or bridges, and show what can go wrong in the absence of robust studies that test the effectiveness of conservation interventions. We will also discuss the importance of collating or synthesising multiple strands of evidence to identify the factors that make a conservation measure effective or ineffective, using a case study on underpasses under roads. Finally, we explore a key challenge – getting scientific evidence accepted and used routinely in conservation policy and practice.

Evidence takes a multitude of forms and can be defined in many ways, but in this chapter we will mostly use ‘evidence’ to refer to scientific tests of treatments or interventions, which compare the ‘treatment’ to a ‘control’ in some way and measure the effect quantitatively. We define evidence in this way as it is a broad description that can still address causality for interventions – did treatment X cause reaction Y? For example, it is not enough to know that some bats flew along bat gantries – we need to know, at a minimum, how many, and how many still flew low across the road. But more on that later.

4.2 Why do we need evidence-based conservation?

Modern medicine has many examples illustrating why the discipline needs a robust evidence base. However, basing medical treatments on scientific evidence was not always the norm. The use of randomised controlled trials to test medical treatments was initially considered unnecessary and unethical, and it was hotly contested. A good example comes from an early champion of evidence-based medicine, Archie Cochrane, who demonstrated that randomised controlled trials were necessary and that expert judgement alone could be flawed. In 1971, he presented preliminary results from a trial comparing home care for heart patients with care in the new Coronary Care Units (note that the findings may not be the same now). He had been criticised for risking the lives of patients allocated to the ‘home care’ group. What follows is in his own words:

The results at that stage showed a slight numerical advantage for those who had been treated at home. I rather wickedly compiled two reports: one reversing the number of deaths on the two sides of the trial. As we were going into the committee, in the anteroom, I showed some cardiologists the results. They were vociferous in their abuse: ‘Archie’, they said ‘we always thought you were unethical. You must stop this trial at once’. I let them have their say for some time, then apologized and gave them the true results, challenging them to say as vehemently, that coronary care units should be stopped immediately. There was dead silence and I felt rather sick because they were, after all, my medical colleagues.

(Maynard & Chalmers, 1997)

Results such as these – where the preferred treatment of the time did not work, or actually made things worse – are used to demonstrate why scientific studies of impacts are important when treating people. A growing body of literature suggests that impact studies are also necessary for treating the health of the biosphere, although the ‘gold standard’ of randomised controlled trials is not always possible in this discipline (Pynegar et al., 2018). As we test more and more measures to conserve species and habitats, we find that many do not work. For example, studies have shown that widely used methods to make water voles move prior to building works were ineffective, risking accidental killing of the protected mammals (Gelling et al., 2018); that reintroduction programmes of species from macaws (Volpe et al., 2017) to tamarins (Beck et al., 1991) have resulted in high or total mortality for the released animals; and that artificial bat roosts built to replace those destroyed during building works often failed to attract any bats and, even when occupied, hosted about half the number of bats that the destroyed roost had (Stone et al., 2013). These results underline the need to test conservation solutions and not to simply assume that good intentions will lead to good outcomes.

Our case studies focus on the environmental impacts of roads. Road construction has been shown to harm animals through habitat degradation, loss and fragmentation, direct mortality and barrier effects (Laurance et al., 2009; Benítez-Lόpez et al., 2010; Rytwinski & Fahrig, 2012). Figure 4.1 summarises these cumulative impacts, which are likely to act at different rates and through a long extinction debt. Unfortunately, studies on a wide range of habitats and taxa from grasslands to vertebrates show that many road mitigation options simply do not work. A growing list of papers points to not only poor design of mitigation and monitoring, but a wider context of poor target setting, weak implementation, inadequate reporting and poor or absent enforcement (e.g. Rundcrantz, 2006; Tischew et al., 2010; Beebee, 2013; Drayson & Thompson, 2013; Villarroya & Puig, 2013). We will not address all of these factors in this chapter, but they are important to consider when asking why ineffective measures have persisted for so long.

Figure 4.1 The multiple causes of bat population reduction by road construction and the delayed response (extinction debt).

Adapted from Forman et al. (2003).

4.3 Case study: bats and roads

Why do we need mitigation for bats crossing roads? In the EU, all bat species have been protected under the EUROBATS agreement since 1994, in recognition of the declining populations of many species. As a consequence, whenever populations may be adversely affected by human activity, impact assessment and mitigation are a legal obligation. Over the last 10 years evidence for significant effects of roads on bats has grown and the need for effective mitigation has become increasingly evident (e.g. Altringham, 2008; Russell et al., 2009; Lesiński et al., 2010; Berthinussen & Altringham 2012b, 2015). There are clear specifications within EUROBATS to mitigate against the impacts of roads on bats.

4.3.1 The need to test mitigation: bat gantries

The EUROBATS commitment to mitigate against the impact of roads on bats is very positive, but are the mitigation strategies being used actually working? Early studies assessed the use by bats of underpasses and overpasses primarily built for other purposes, such as to carry minor roads, footpaths or streams. If bats were seen near to these structures it was generally assumed that they were effective mitigation tools (Highways Agency, 2001, 2006; reports reviewed by O’Connor et al., 2011). Underpasses, culverts, footbridges and bridges for vehicles, all of various sizes, were widely adopted as mitigation solutions (Figure 4.2). Many were not subsequently surveyed for use by bats, or qualitative surveys were written up in often confidential reports. Many studies reported ‘use’ – small numbers of bats observed in underpasses or flying over bridges of various kinds, without reference to the number still crossing the road unsafely, or not crossing at all (see Highways Agency, 2006; O’Connor et al., 2011), and many lacked convincing definitions of use. This meant that future projects could not learn from the success or failure of previously built mitigation structures.

Figure 4.2 Two underpasses found to vary in effectiveness in guiding bats safely under roads. (a) An effective underpass on the A590, Cumbria, UK; (b) an ineffective underpass on the A66, Cumbria, UK. Boxplots show the number of bats crossing per survey using the underpass and crossing over the road above at safe and unsafe heights (above and below 5 m, traffic height). The variable success of underpasses underlines the need to understand the details of conservation interventions; in this example, the location of the underpasses impacted on how effective they were.

From Berthinussen and Altringham (2012b).

In addition to multi-use structures, some ‘bespoke’ structures were built and ‘bat gantries’ or ‘wire bridges’ (Figure 4.3) were widely adopted. Bat gantries were assumed to act as navigational aids to echolocating bats, encouraging them to continue using existing ‘commuting routes’ from roosts to feeding areas (which often follow linear features such as hedgerows) after road construction, but lifting them above the traffic. Ideally, crossing points should be built on known bat commuting routes determined by pre-construction surveys, as bats tend to be faithful to particular routes. However, many were built away from known bat commuting routes for engineering reasons, to fit in with landscape topography, to combine bat routes with minor roads or footpaths, or simply to reduce cost. It was assumed bats would find the new crossing points (Highways Agency, 2001), and in some cases new hedge planting was designed to guide them to these structures. In many guidance documents, environmental statements and mitigation plans it was implicit, or even explicit, that the bats would respond as predicted (Highways Agency, 2001; Limpens et al., 2005).

Figure 4.3 Two bat gantry designs: (a) wire mesh design on the A11, Norfolk, UK; (b) wire and ball design on the A590, Cumbria, UK. Boxplots show the results of surveys carried out to test the effectiveness of the gantries in guiding bats safely over the road. Data were recorded for the total number of bats crossing per survey, the numbers crossing at unsafe heights (below 5 m, traffic height) and the numbers using the gantry according to two definitions of ‘use’ (flying within either 2 m or 5 m of the wires above traffic height). The bat gantry story neatly demonstrates the need to test conservation interventions before rolling them out on a wide scale.

From Berthinussen and Altringham (2012b, 2015).

In 2008, JDA was asked to provide evidence to a public inquiry for the effectiveness of these strategies (Altringham, 2008). No quantitative evidence was found to suggest that any of the strategies implemented were effective in protecting bats, particularly at the population level. However, neither was there evidence to suggest that they were ineffective. This prompted us to conduct our own research to determine the effects of roads on bats and the effectiveness of mitigation (Berthinussen & Altringham, 2012a, 2012b, 2015).

In our research we emphasised the difference between qualitative assessments of the ‘use’ of a structure by a small number of bats and measures of effective protection at the population level. We also stressed that the number of bats present pre-construction is rarely assessed, meaning that post-construction bat numbers may already be a fraction of what was there before. We proposed two broad measures of effectiveness: (1) measurements of local bat activity and of the movement of bats along severed commuting routes before and after road construction, to assess landscape-scale impact and the permeability of new roads; and (2) measurements of the effectiveness of the crossing structures – the proportions of bats that use them to cross safely. Our research was limited by logistics to the second measure – do mitigation structures guide bats safely across roads?

The headline result was that ‘wire and ball’ bat gantries did not alter the behaviour of bats crossing roads – they were wholly ineffective (Berthinussen & Altringham, 2012a; Figure 4.3b). This was a disturbing finding, as over the previous decade about 15 gantries had been built in the UK and continental Europe. Although our study showed that one design of bat gantry was ineffective, it was suggested that other designs would have greater success in guiding bats to fly at safe heights above roads. Our next study found that ‘wire mesh’ gantries (Figure 4.3a, of a different design to the ‘wire and ball’ structures) were equally ineffective (Berthinussen & Altringham, 2015).

In summary, a mitigation measure widely used for over a decade was essentially untested and subsequently shown to be ineffective. This underscores the need for rigorous testing of the measures that we implement in the name of conservation. We also found (albeit based on a small sample size) that building all types of crossing away from known commuting routes, even with new planting to guide bats to them, was unsuccessful (Berthinussen & Altringham, 2012a). This is important, as it shows that the location of mitigation measures is as important as the measure itself – effective measures need to be implemented with a good understanding of the local context. Furthermore, we found evidence that some underpasses were used by a high proportion of bats, and that the one green bridge tested in the UK – a large structure planted with trees, shrubs and ground cover – was used by over 90% of bats crossing the road in that area, suggesting that effective ways to allow bats to safely cross roads do exist.

4.4 Synthesising evidence

The bat gantry case study provides some insight into why we need to rigorously test the effectiveness of measures aiming to protect the natural world. However, this is just the first step towards implementing a truly evidence-based approach to conservation. The next step is to systematically bring together all the evidence, from many studies, on particular conservation measures. This approach is also borrowed from evidence-based medicine, where it has proven to be a lifesaver.

One of the most important developments in evidence-based medicine was the Cochrane Collaboration, an organisation set up to conduct systematic reviews of the scientific evidence on topics such as how well different treatments worked. In medicine – as in conservation – natural variation in populations means that it often takes large numbers of replicates for beneficial or detrimental effects to become apparent. Modern doctors, making potentially life-changing decisions, want to have the information on every study on a particular treatment to hand, not just the results from a single trial that may not be representative. The goal should be the same in conservation: to bring together all the evidence for an intervention to assess whether it works, whether it does harm, or whether it only works in certain situations or with certain variations of the intervention.

There are many examples of the importance of collating evidence in medicine. For example, a systematic review on cot death or sudden infant death syndrome (SIDS), using the studies already available in the 1970s, could have saved the lives of an estimated 60,000 babies. Due to a lack of evidence synthesis and an overreliance on expert opinion, medical practitioners advised parents to put children to sleep on their fronts until the 1990s, when studies and reviews led to the realisation that this sleeping position increased the risk of SIDS (Gilbert et al., 2005).

In conservation, collating or synthesising the data is as critical as it is in medicine (Sutherland et al., 2004). While the most rigorous method, systematic review, is very important (see Chapter 7, the Collaboration for Environmental Evidence and Mistra EviEM), more cost-effective methods of collating the evidence may also be desirable in this underfunded discipline, where the evidence itself can be scarce and variable in quality (Sutherland & Wordley, 2018). ‘Synopses’ published by Conservation Evidence (www.conservationevidence.com) follow one such method, known as subject-wide evidence synthesis (Sutherland & Wordley, 2018). Researchers draw up lists of all the interventions that could benefit a given taxa or habitat, classified according to potential threats based on IUCN criteria (Threats Classification Scheme Version 3.2); the scientific studies for the effectiveness of each intervention are then collated and summarised. For example, we produced the Bat Synopsis (Berthinussen et al., 2013, updated 2019), which provides key messages and summaries of the relevant studies, to help conservationists see which interventions for bat conservation are likely to be the most effective, and under which circumstances. The summary of this synopsis in What Works in Conservation (Berthinussen et al., 2018) takes this a step further, by using expert scoring to categorise the interventions based on levels of effectiveness, certainty in the evidence available and potential harms.

The first Bat Synopsis (Berthinussen et al., 2013) listed 78 interventions that could be implemented to conserve bats, covering areas as diverse as logging, roost provision and wind turbine operation regimes. No evidence for effectiveness was found for 48 of the 78 interventions, many of which are used routinely in the UK and elsewhere. This does not mean they are ineffective, but simply that they had not been tested quantitatively when we checked the literature. For a further 12 interventions the evidence was too limited for assessment. This demonstrates the scarcity of experimental evidence for many possible management actions, severely limiting the ability of conservationists, ecological consultants, developers and government agencies to undertake evidence-based conservation or mitigation.

Of the 18 remaining interventions, 14 had some proven value as conservation tools for bats. These included using selective logging instead of conventional logging, turning off wind turbines at low wind speeds and minimising light pollution. An update to this synopsis was published in 2019 (Sutherland et al., 2019), expanding the list of interventions to 190 and adding new studies that were published in the intervening years. There are many interventions which have had valuable evidence added in this update, but we have not yet seen a shift to a majority of interventions being tested via multiple high-quality experiments.

4.4.1 Example of evidence synthesis: road underpasses

For many of the interventions addressed in the Bat Synopsis, our greatest contribution was to demonstrate that no evidence existed for the efficacy of these measures – hopefully spurring more research and a more critical eye towards choosing conservation measures. But for a handful, we could begin to tease out what made an intervention effective in some circumstances but not others – one of the many benefits of summarising multiple studies. One such intervention is the use of underpasses to get bats to cross roads safely.

In the 2013 Bat Synopsis we found four studies, from Germany, Ireland and the UK, which between them showed that at least nine bat species used underpasses (none purpose-built for bats), with up to 96% of the bats crossing through underpasses rather than the road above (although this varied greatly) (Berthinussen et al., 2013). By summarising the key details of each study, we can see that some species use underpasses frequently while others do not appear to use underpasses at all, and that only a few species appear to use small underpasses, such as drainage pipes of diameter less than 1.5 m. There are indications that effectiveness increases with diameter and when underpasses are placed on known bat commuting routes – conclusions supported by ongoing studies (Davies, 2019). The 2019 update of the Bat Synopsis added two further studies, which tested much larger underpasses and still found the largest structures to be the most effective, but also explored the differing responses of various functional guilds of bats. These details are critical. Further testing and refining of underpasses, followed by evidence synthesis, should help to ensure that future underpasses are as effective as they can be.

4.5 Getting the evidence used

We are trying to bring this work, demonstrating the importance of generating and using evidence on the effectiveness of interventions, to as large an audience as possible, to ensure that those responsible for commissioning, designing, approving and testing mitigation structures are aware of it. The bat gantry studies have been reported in national newspapers, radio and television. This was achieved through press releases, by approaching media contacts directly and by being approachable when contacted. The work has also been reported in several books and papers (Altringham, 2011; Abbott et al., 2015; Altringham & Kerth, 2016; Sutherland & Wordley, 2017). JDA and AB have run seven workshops for practitioners on road mitigation measures for bats and talked at over 10 conferences in the UK and abroad. CFRW has mentioned this study in around 50 talks to conservationists and government agencies and used it as an example in an opinion piece on evidence use in conservation (Sutherland & Wordley, 2017). The Bat Synopsis and What Works in Conservation, which contains a summary of the Bat Synopsis, have also been widely promoted.

This awareness resulted in tens of thousands of views of the paper and relevant parts of the Bat Synopsis, and this exposure has translated into further successes. The impact of early work (Berthinussen & Altringham, 2012a, 2012b) led to a Defra-funded project to develop better mitigation monitoring protocols (Berthinussen & Altringham, 2015) and a statutory conservation agency guidance note summarising the protocols. The approximately £1 M spent on bat gantries in the UK as of 2017 (Sutherland & Wordley, 2017) was brought up in the House of Lords by Lord John Krebs in January 2018, who used it to demonstrate why the UK government’s 25-year environment plan needed to explicitly commit to being evidence-based. Some road-building projects have taken heed of the evidence. The A40 Penblewin to Slebech Park Improvement in Wales opted to mitigate impacts on bats using underpasses of varying sizes, many built on known commuting routes, and funded more rigorous monitoring (Davies, 2019).

However, not everyone is listening. Despite widespread reporting of the ineffectiveness of bat gantries in 2012, six gantries of a ‘wire-mesh’ design were built in Norfolk, England in 2014 at a reported cost of £350,000. These were probably planned before the 2012 paper was published, but plans were not modified in light of this study. In 2015 these gantries were also shown not to work (Berthinussen & Altringham, 2015). Nevertheless, seven more gantries are under construction (as of 2018) at a cost of over £1 M on the North Norfolk Distributor Road (MacDonald, 2014). In another example, the environmental statement for the proposed and controversial extension to the M4 across the Gwent Levels in Wales (Welsh Government, 2016) draws extensively on our 2015 Defra report (Berthinussen & Altringham, 2015). However, it proposes numerous culverts for bats which, by the authors’ own admission, are almost all too small to be used by the target bat species. In addition, most will not be on known commuting routes.

Furthermore, there are still inadequate mechanisms in place to assess the effectiveness of mitigation measures. A feature of many environmental statements and mitigation plans is the absence of a monitoring plan capable of assessing mitigation success or failure. There is frequently no monitoring plan at all. This appears to be due to a reluctance or inability of government agencies to enforce effective monitoring, a reluctance on the part of many developers to pay for monitoring and a lack of understanding about how to design and conduct monitoring that is fit for purpose. As a result, developers and taxpayers spend money on unproven mitigation with no prospect of improved understanding.

4.5.1 Why is evidence ignored?

Why are proven methods rejected, often in favour of methods that have been demonstrated not to work? Why is there an apparent reluctance to seek out, use or accept evidence, or to collect it, among some decision-makers, including some government agencies and ecological consultants? Sutherland and Wordley (2017) explored a few general psychological and structural reasons for this phenomenon, and more detail on this topic is given in Chapter 9 of this book. Here we share some of our own experiences of the failure to use evidence in road mitigation.

The real or perceived higher financial cost of effective mitigation solutions is one concern. Mitigation consumes a very small part of the total cost of a road-building project, but mitigation and monitoring are obvious targets when budgets are tight or overrun. Effective mitigation may or may not be more expensive than ineffective options, but ineffective mitigation is simply a waste of resources.

A desire to simplify the planning and implementation of mitigation is another reason why some parties are reluctant to challenge or change accepted approaches. Road building is complex, making off-the-shelf, approved mitigation solutions an attractive option. Being able to implement development projects as quickly and cheaply as possible can make mitigation a tick-box exercise – complying with regulation at minimal cost may be more important than implementing effective mitigation. Mitigation solutions such as bat gantries can be designed and built relatively cheaply and, if experts say they will work, then they fulfil all legal requirements and may be assumed to require little or no ‘expensive’ monitoring. To question their effectiveness can put in jeopardy budgets, work schedules, building specifications, even the project itself. A reluctance to listen to objections is understandable, but not excusable. Consultant ecologists can be placed in a difficult position. Their scientific training, personal concern about nature and professional standards all demand unbiased assessment. However, their livelihoods depend upon contracts from developers who are frequently not obliged to commit to effective mitigation and monitoring.

Finding, evaluating and applying the evidence on mitigation strategies can also be a challenge. With the existence of freely available downloadable material (such as the Bat Synopsis) in a concise, jargon-free form, decision-makers should be more aware of what works and what does not. However, ecological consultants and statutory agency staff still need time to find, read and digest the information, and require some scientific training to evaluate the evidence. The difference between quantitative evidence and anecdote is not always understood and ‘professional judgement’ may be relied on even when it runs counter to the evidence. However, it does not have to be painful for developers, consultants or planners to improve on current practice. Adoption of good mitigation practices early in a project can avoid the problems of making corrections during the project, and investment in effective technologies may lower the costs of solutions such as large underpasses and green bridges.

4.6 How can evidence use in mitigation projects be improved?

First, there should be a key requirement that mitigation structures are tested for effectiveness, not just use, and a quantitative bar set for effectiveness (Berthinussen & Altringham, 2015). Ecologists employed to assess mitigation effectiveness must be prepared to shun options proven to be ineffective. Professional bodies must fully support ecological consultants in implementing those measures shown to be effective, and sanction members who use methods known to be ineffective. Improvements may be much more evident if the enforcing statutory agencies are willing and able to deny planning permission to development projects that have poor mitigation strategies. There should be real commitment from governments to pledge to conserve species and habitats using evidence-based measures and discarding measures proven to be ineffective. This may require additional resources to assess existing and proposed legislation against evidence syntheses.

To identify effective and ineffective solutions there is a clear need for dedicated funding for rigorous tests of interventions. Monitoring interventions often requires long-term commitment which, in turn, requires adequate long-term funding. This could come from developers and government agencies, but a greater recognition by academic funding bodies of the value of applied questions would also have a huge impact. PhD projects could be encouraged to have applied components, testing interventions. Research council funding for academics to address applied questions of conservation importance and communicate them to practitioners would have a huge impact.

Greater power for statutory agencies to enforce existing laws, check up on implementation and demand replacements for ineffective solutions would dramatically improve mitigation effectiveness. A framework with greater incentives for developers to show that their mitigation has been effective would be beneficial. These could include a requirement to make the results of mitigation monitoring for effectiveness public, penalties for failures to do so and awards for new, proven effective solutions.

While many of these goals may not be realised in the near future, we can all promote approaches to conservation that are evidence-based and effective. If enough of us do it, it might just change the world.