Comparative evaluation of three surveillance systems for infectious equine diseases in France and implications for future synergies

Summary

It is necessary to assess surveillance systems for infectious animal diseases to ensure they meet their objectives and provide high-quality health information. Each system is generally dedicated to one disease and often comprises various components. In many animal industries, several surveillance systems are implemented separately even if they are based on similar components. This lack of synergy may prevent optimal surveillance. The purpose of this study was to assess several surveillance systems within the same industry using the semi-quantitative OASIS method and to compare the results of the assessments in order to propose improvements, including future synergies. We have focused on the surveillance of three major equine diseases in France. We have identified the mutual and specific strengths and weaknesses of each surveillance system. Furthermore, the comparative assessment has highlighted many possible synergies that could improve the effectiveness and efficiency of surveillance as a whole, including the implementation of new joint tools or the pooling of existing teams, tools or skills. Our approach is an original application of the OASIS method, which requires minimal financial resources and is not very time-consuming. Such a comparative evaluation could conceivably be applied to other surveillance systems, other industries and other countries. This approach would be especially relevant to enhance the efficiency of surveillance activities when resources are limited.

INTRODUCTION

The monitoring of infectious animal diseases is necessary to describe their occurrence and help the planning, implementation and evaluation of risk mitigation actions. Surveillance systems must primarily be implemented for diseases that could have a huge impact on public health, animal health and welfare, and the national or international economy and trade [1–3]. For equine diseases, equine infectious anaemia (EIA), equine viral arteritis (EVA) and contagious equine metritis (CEM) are some of the most frequently monitored hazards because they can lead to significant financial losses for the horse industry. EIA is caused by a virus of the family Retroviridae. Infection usually results in a chronic disease with intermittent acute episodes which can end in death. The EIA virus is mainly transmitted by the transfer of blood from an infected horse, either by bloodsucking horseflies or the use of contaminated needles (iatrogenic transfer) [4]. Horses generally remain lifelong carriers for EIA and so represent a potential hazard for other horses. EVA is a respiratory and reproductive disease caused by a virus of the Arteriviridae family. Some strains of EVA virus lead to abortions or mortality in foals [5]. EVA can be transmitted horizontally by aerosols or venereal contact, vertically through infection in utero or, occasionally, through indirect contact by fomites. The virus is extremely resistant and can even be transmitted through frozen semen. Stallions can remain lifelong carriers for EVA and can potentially transmit the disease to mares during breeding [6–8]. CEM is due to Taylorella equigenitalis, a Gram-negative coccobacillus belonging to the Alcaligenaceae family. CEM is characterized in mares by a variable degree of vaginitis, endometritis, cervicitis and temporary infertility. In stallions, no clinical signs are observed. It is a sexually transmitted disease, but indirect transmission by fomites is possible. Stallions and mares can remain carriers for CEM for several years and can transmit the disease during breeding [9, 10].

In France, EIA, EVA and CEM are endemic, but over the last 10 years, only a few sporadic cases have been detected, and none at all some years [11–13]. They have different regulatory statuses: in view of its serious economic impact (possible death, euthanasia of infected horses, farm placed under quarantine, movement restrictions, etc.), the risk mitigation actions against EIA are more severe than those for EVA or CEM. Most of the EIA monitoring and control measures are implemented by public authorities, whereas measures against EVA and CEM are implemented either by authorities or private partners.

As defined during the workshop held prior to the International Conference on Animal Health Surveillance (ICAHS) in 2011, a ‘surveillance system’ or ‘network’ is a range of surveillance components used to investigate the occurrence of a single hazard in a specified population [2]. A ‘component’ was defined as a single surveillance activity for investigating the occurrence of one or more hazards in a specified population [2].

In most developed countries, the surveillance networks monitoring equine diseases comprise various components, including compulsory notification of suspicions, testing prior to export or sale, and surveillance of breeding stock. In France, there is also a voluntary passive surveillance run by the RESPE (French network for epidemiological surveillance of equine diseases) veterinary association [14, 15]. Unfortunately, these various components are not well connected and do not share all the data collected.

Surveillance systems have to be evaluated to ensure they meet their goals and provide high-quality health information in a timely and efficient way [16-18]. Furthermore, multi-component networks need to be assessed to highlight possible synergies that could be implemented between existing components in order to improve the whole system's performance. To date, the French surveillance systems for equine diseases have never been evaluated. Moreover, there are very few published evaluations of equine disease surveillance systems worldwide, and most of these publications focus on assessing only one component [19-21].

The objectives of this study were to (i) assess the surveillance systems for equine diseases in France by means of the OASIS method [18], taking EIA, EVA and CEM as examples; (ii) compare the results of the assessments and highlight their mutual and individual strengths and weaknesses in order to propose improvements, especially to increase the links and synergies between the different components and systems.

MATERIALS AND METHODS

Surveillance networks and components

The French surveillance networks for EIA, EVA and CEM are quite separate and each includes several independent components (Table 1). Some components are managed by the animal health authorities, including the French Ministry of Agriculture's Food Directorate (DGAL), the national reference laboratories (NRLs), the French agency for food, environmental and occupational health and safety (ANSES) and the French horse and riding institute (IFCE). These components are the compulsory notification of suspected cases (comprising clinical and post-mortem suspicions by veterinarians and necropsy centres), the mandatory surveillance of stallions used for semen collection and pre-export testing. Another component is directed by private partners, i.e. pre-sale testing, generally managed by sales companies. There is also the voluntary surveillance run by RESPE, an association of more than 500 veterinarians throughout the country who report clinical suspicions and collect samples from all cases regarded as suspect when compared to a list of syndromes, including abortions and respiratory disorders that may be due to the EVA virus [14]. Last, one component dealing with breeding stock is managed by both public authorities (IFCE) and private partners (stud books).

Table 1. Surveillance components for EIA, EVA and CEM currently implemented in France

CEM, Contagious equine metritis; EIA, equine infectious anaemia; EVA, equine viral arteritis; RESPE, French network for epidemiological surveillance of equine diseases.

* Two types of surveillance are considered in this table: (i) passive surveillance, which is defined as an ‘observer-initiated provision of animal health-related data (e.g. voluntary notification of suspect disease) or the use of existing data for surveillance’ [2] and which is used to detect infected animals with clinical signs or that have died from the disease; (ii) active surveillance, which is an ‘investigator-initiated collection of animal health-related data using a defined protocol to perform actions that are scheduled in advance’ [2] and which is used to detect asymptomatic infected animals.

† EIA is often tested before a sale in France by an agar gel immunodiffusion test (serological test), especially before auctions, because the purchase may be cancelled if the horse is discovered to be infected by the EIA virus within 30 days of the transaction.

‡ The tests to be performed depend on the destination country and the type of export: temporary or permanent.

Assessment method

The semi-quantitative OASIS method [18] was chosen to evaluate surveillance of the three diseases. OASIS provides a detailed and standardized framework. It was designed to evaluate surveillance networks as a whole by assessing many criteria (n = 78) relating to the goals, organization and operation [22]. It can assess the level of satisfaction for each criterion globally so as to consider all the components at the same time, which is more relevant than evaluating each component one after another. The method has already been used to assess animal diseases and food safety surveillance networks in different countries [23–25].

We used the OASIS ‘flash’ variant, which is faster than the conventional OASIS method, especially for the first step related to data collection [26]. With the conventional method, information is collected by an assessment team who meets several of the system's stakeholders, including field workers, central institutional organizations and intermediate units. With the flash method, data were collected by the same person, by reviewing the literature and regulations, and through phone discussions and individual meetings with people from central institutional organizations. The required data relate to objectives, structure, targeted population, tools, data management, etc. All data were collected through the OASIS detailed questionnaire, for each component and disease.

For each disease, 8–9 experts were selected and brought face-to-face to assess the surveillance system. The chosen experts were the managers of each component, specialists in the disease, epidemiologists and animal health surveillance experts used to implementing OASIS. Most of the experts took part in all three expert panels. Three 1-day meetings were held, one for each surveillance system. During the meetings, the panel rated each criterion on the OASIS scoring grid in keeping with the questionnaire and with the help of the detailed OASIS scoring guide. The expert panel then reviewed and validated the results for each network as a whole.

The results are displayed using the three OASIS outputs [18], which present the assessment of ten functional sections of the surveillance systems (e.g. central institutional organization, laboratory, surveillance tools), seven critical control points (CCPs, e.g. sampling, coordination and data analysis) and ten attributes (e.g. timeliness, flexibility, simplicity). A criterion's level of satisfaction is measured by a percentage that represents the score obtained for the criterion out of the maximum possible score. The panel highlighted the main strengths and weaknesses of each network. First, an item was considered a strength if its level of satisfaction was higher than 66% of the maximum score and a weakness was an item for which the level of satisfaction was lower than 50%. Nevertheless, according to the principle that the inner conviction of the evaluators prevails, the experts slightly refined the categorization according to their interpretation of the performance of the surveillance systems.

Finally, the experts compared the results of the three assessments. They identified the mutual and specific strengths and weaknesses of each, in addition to identical needs, and they drew up recommendations.

RESULTS

Although the three surveillance systems were assessed independently, the evaluations reveal many similar strong and weak points as well as some differences (Figs 1–3).

Strengths

The laboratory section is the greatest strength for all three systems (Table 2). For each disease, there is a network of accredited laboratories and an NRL. This section includes many criteria that are highly satisfactory, including the quality of diagnostic tests (sensitivity, specificity, timeliness and relevance), standardization of methods used in all the laboratories, implementation of annual proficiency tests, quality assurance and sufficient resources to perform analyses.

Table 2. Main strengths and weaknesses of the French surveillance systems for EIA, EVA and CEM

CEM, Contagious equine metritis; EIA, equine infectious anaemia; EVA, equine viral arteritis.

* The disease for which the surveillance system has a specific strength or weakness is given in parentheses.

The surveillance tools are quite good (Figs 1 and 2), especially regarding the quality and relevance of samples, for which collection is easy and standardized. Reporting procedures are quick and easy and diagnostic techniques are well established.

Fig. 1. OASIS output 1 for the French EIA, EVA and CEM surveillance systems assessed (the level of satisfaction for each section is displayed as a percentage score).

Fig. 2. OASIS output 2 for the French EIA, EVA and CEM surveillance systems assessed (the level of satisfaction for each critical control point is displayed as a percentage score).

For the three surveillance systems, data collection is also quite good (Fig. 2) due to the fast and simple notification procedures, the quality of samples, and the speed of laboratory analyses. The other criteria related to timeliness (Fig. 3) also return a high level of satisfaction, e.g. the transmission of samples to laboratories and the feedback on individual results to field workers.

Fig. 3. OASIS output 3 for the French EIA, EVA and CEM surveillance systems assessed (the level of satisfaction for each attribute is displayed as a percentage score).

Weaknesses

Many mutual weaknesses and needs have been identified (Tables 2 and 3). The main needs for all three systems concern the current absence of central institutional organizations (Fig. 1), there being a necessity for a steering committee, management team, and a scientific and technical committee (STC). Regarding each network individually, there is no steering committee bringing together representatives of all partners involved in monitoring. Hence, the expectations of certain stakeholders – private owners, laboratories, and the semen production industry – are neither identified nor taken into account when defining surveillance objectives. Therefore, system acceptability is low (Fig. 3). Within each surveillance system, the absence of a management team common to all the components substantially hinders flexibility (Fig. 3), data analysis, communication and coordination between all the parties, which is the lowest-rated CCP (Fig. 2). Moreover, none of the systems has any joint STC for existing components.

Table 3. Main mutual needs for improving the surveillance of EIA, EVA and CEM in France

CEM, Contagious equine metritis; EIA, equine infectious anaemia; EVA, equine viral arteritis.

Data management is poor for all three diseases (Fig. 1). Indeed, each component has its own independent database and only data about confirmed cases are centralized by the public authorities.

Another weak point is data analysis (Fig. 2). The data are not analysed as a whole and most of the databases are not well suited to epidemiological analysis. Furthermore, the lack of a joint STC prevents a satisfactory scientific validation of results.

Opportunities and threats

This comparison of the three surveillance systems has highlighted several strengths (tools, teams or skills) specific to a component or system which could be shared and used by the other components or systems. They are opportunities to create synergies and improve the monitoring of all three diseases (Table 4).

Table 4. Opportunities: main tools, teams and skills which should be pooled to improve the surveillance of EIA, EVA and CEM in France

CEM, Contagious equine metritis; EIA, equine infectious anaemia; EVA, equine viral arteritis; IFCE, French horse and riding institute; RESPE, French network for epidemiological surveillance of equine diseases.

For instance, communication (Fig. 1) and information distribution (Fig. 2) are better for the EIA surveillance system due to the better external communication and a systematic annual report published jointly by the national animal health authority and the NRL. Similar annual reports should be implemented regarding EVA and CEM, based on the EIA model (Table 4). The regular reporting improves motivation and awareness of all the partners, as well as the reputation and acceptability of the surveillance networks.

Several specific strengths have also been pointed out regarding surveillance tools. For example, data collection forms have been created by the RESPE regarding EVA and the IFCE regarding EIA, EVA and CEM. It would be advisable to draft a standardized form based on existing models for use by all the surveillance components for all three diseases.

Regarding data management, automated data transfers (ADTs) are used by accredited laboratories and the IFCE for breeding stock. These transfers are more reliable and faster than manual entries. Given that all analyses are performed by a network of accredited laboratories (or the NRL), ADTs should be generalized to every component. Furthermore, it is necessary to implement a single relational database or to network the existing ones.

Moreover, several investigation teams can be activated either by the animal health authorities, stud books or the RESPE when an EIA, EVA or CEM case is detected (Table 4). However, depending on the breed and use of the infected animal, certain EVA or CEM cases are not currently investigated. These teams should be united to form a single investigation unit. If not, several teams could continue to exist but need to be better coordinated and activated according to formal, standardized procedures.

By contrast, some threats were identified, especially the poor sampling (Fig. 2) and surveillance procedures (Fig. 1) regarding EIA. Although EIA, EVA and CEM are all notifiable diseases, the populations targeted for EVA and CEM surveillance are breeding stock, exported horses and auctioned animals, whereas EIA has a different regulatory status in France and the entire equine population appears to be monitored. While surveillance activities seem appropriate for breeding stock, exported and auctioned animals, the current procedures are not as complete or representative as necessary for other equids, e.g. horses raised for meat or leisure, donkeys, etc. Similarly, the sensitivity, representativeness and usefulness scores (Fig. 3) are worse for the EIA surveillance network than for the EVA and CEM networks. In order to improve these features, it will be necessary to define clear and detailed objectives and to ensure that current surveillance activities are consistent with these objectives, with the help of the STC. It may be necessary to revise sampling procedures, to amend or end certain activities and create others.

DISCUSSION

The objectives of this study were to assess the surveillance of three equine diseases in France, and to compare these evaluations in order to identify possible improvements. Despite several weaknesses and poor coordination between components, many significant strengths have been identified and the quality of the overall surveillance systems for EIA, EVA and CEM is quite good. OASIS was relevant and flexible enough to allow the comparative assessment of such complex multi-component systems. All the assessors approved the final conclusions and agreed that the method is easy to use and raised no major difficulties.

Several quantitative methods have been designed to assess the surveillance system in veterinary and public health. They may provide an accurate evaluation, but are based on only a few criteria such as sensitivity, costs or performance indicators [17]. Some qualitative or semi-quantitative methods have also been developed [3, 17, 18, 27–29]. Some of them [30] include attributes from the US guidelines published by the Centers for Disease Control and Prevention (CDC) [27], in the same way as the OASIS tool. However, OASIS uses many more assessment criteria, enabling the system to be assessed in depth. In our study, consensus was reached quite easily both on the score for each criterion and ensuing recommendations.

A major difference between OASIS and other frameworks is the level of detail of the scoring grid, which simplifies the evaluation and also guides the evaluators. Other authors have drawn up a scoring guide to implement a collective semi-quantitative evaluation, using a hierarchical analysis method [28]. However, they have assessed only a few criteria (n = 13) and focused only on the ‘credibility’ of the surveillance. In their study, the weighting of each criterion is variable and subjective because it was decided by the assessors themselves, depending on their own area of expertise. Other assessments are based on comparison with a standard surveillance document but do not use a scoring grid [31].

Some authors developed the SERVAL method, which consists of a tailored collective qualitative assessment [29]. The assessors select 5–10 evaluation attributes from a list of 22 depending on the surveillance network's objectives. To perform a comparative assessment of surveillance networks by means of SERVAL, it would be necessary to select the same attributes for each network. The SERVAL data collection questionnaire is less detailed than the OASIS one and there is currently no scoring guide. Nevertheless, SERVAL, which may include other analyses such as cost analysis, helps assessors to define the purpose and scope of the evaluation (time, resources, etc.), and to further identify the target audiences and how the outcomes of the evaluation will be communicated. Finally, SERVAL and OASIS appear complementary. It would be useful in the future to assess surveillance systems by combining the two methods through a semi-quantitative evaluation using OASIS within the SERVAL framework.

OASIS is a standardized tool, which can assess in the same way the monitoring of any disease, whatever and however many the components. The OASIS method facilitates comparison and readily highlights the individual strengths of each component as opportunities that could be extended to other components, especially through its different results and graphical outputs which are complementary. Output 3 highlights the system's quality (for instance poor flexibility), whereas output 2 indicates which CCPs could explain this situation and where there is a room for improvement (e.g. the lack of coordination), while output 1 shows which part of the network has to be improved (e.g. the central institutional organization).

This research used the ‘OASIS Flash’ method which reduces the time required for data collection. However, the information collected is less comprehensive than with the conventional method and the recommendations could be less detailed than recommendations produced with the conventional method. In the same way, the compliance of field workers could be more difficult to obtain because they were not directly questioned during the data collection step. Nevertheless, the information collected before the meetings was validated by component managers. The overall results and recommendations would probably be similar whichever OASIS method had been used, as was shown by the use of both OASIS variants to assess the surveillance of bovine tuberculosis in France [32]. Moreover, this faster variant is easier to implement with the active participation of the various component managers because it is not very time-consuming, as their presence is requested only for the 1-day assessment meetings. The number of person-days needed for data collection is limited, depending on the systems being evaluated, unless numerous interviews, additional economic analyses and/or quantitative assessments are required.

All the component managers should ideally participate in assessing a surveillance system; this study brought all of them together except representatives of auction companies. Despite their absence, the prior data collection and the presence of other experts allowed the team to evaluate each criterion. On the one hand, the presence of network managers is necessary because they know the surveillance systems best. On the other, their presence could bias the assessment because they could regard certain criteria as more important (or less important) than is the case. The detailed scoring guide and presence of external experts used to evaluate other surveillance systems counteracted this. Although the choice of assessors may influence some results, unavoidable individual subjectivity is mostly neutralized due to the collegial evaluation procedure. The fact that the experts reached a consensus quite easily for all the criteria suggests that managers were fairly objective and that assessment is reliable.

Nevertheless, some limitations should be mentioned. Economic analysis could have been performed in order to assess the cost-effectiveness of each recommendation. Additionally, the OASIS method does not include quantitative evaluation (especially of sensitivity and specificity), even if results are presented as percentages of achievement of objectives. Quantitative assessments could be useful for measuring the potential impact of the proposed improvements. However, the overall assessment and main recommendations would probably have been quite similar.

Considering that many recommendations are the same for all three networks, it will be more useful to create joint teams, tools and operations rather than implement them separately three times. For instance, it would be more efficient to form a single steering committee for all three systems because there is extensive overlap in the people involved. Similarly, it would be suitable to create a joint STC and management unit uniting the managers of all the components which could then pool their tools and skills – including data analysis. Such joint bodies would strengthen the relationships and cohesion between partners while fostering the stability and flexibility of surveillance as a whole. A single innovation like this can improve several sections, CCPs and attributes, and enhance the efficiency of all three systems simultaneously. However, it is legitimate to wonder whether there is sufficient willingness and indeed sufficient resources available for implementing such synergies. Fears may be expressed about issues such as a lack of cohesion between stakeholders within the horse industry, refusal to share data, a loss of self-reliance or increased short-term costs. Nevertheless, pooling tools would lead to long-term savings.

In order to perform regular, complete and relevant data analysis, the epidemiologists of the French epidemiological surveillance platform for animal health (ESA Platform) should take part in data analysis. Indeed, this platform was created in 2011 to strengthen the monitoring of the main animal health hazards by helping improve surveillance systems and fostering the pooling, analysis and dissemination of health data.

The different transmission modes and the various regulatory statuses of EIA, EVA and CEM have led to the concurrent implementation of several separate surveillance components. Some of them address other diseases. Therefore, although we focused on three diseases, our evaluation covered all the perennial surveillance components for equine diseases in France. Consequently, if the previously mentioned teams and tools were created and pooled, they could be used to improve surveillance of the other threats too. Furthermore, a single national surveillance system for equine infectious diseases could be implemented. In addition, new components could be created and added if relevant, whether for syndromic surveillance (e.g. data collected by renderers), sampling in abattoirs, surveillance of sentinel herds, surveys or surveillance of vectors, for example. Moreover, the equine population should be better covered by necropsy centres, and current moves to improve the networking of these centres consolidated. In the UK, improving the livestock population's access to post-mortem facilities was the first core recommendation of the surveillance advisory group tasked to recommend a future delivery model for surveillance in England and Wales in 2012 [33]. The usefulness of necropsy centres for the timely detection of fatal infectious diseases has also been highlighted in France, especially during the EVA outbreak in Normandy in 2007 [34].

Unlike most published assessments, which focus on a single component or system, this study entailed a comparative evaluation of several different networks. This approach reveals possible synergies, which is especially relevant when resources are limited. In view of the existence of multi-component surveillance systems for equine diseases in several developed countries [35-38], comparative assessments using the same model may be conducted to evaluate the surveillance of equine hazards in other countries. Furthermore, this approach may be applied to other industries, other kinds of surveillance systems, and other types of diseases, including exotic diseases, zoonoses, systems that include surveillance of vectors (e.g. West Nile fever) or of susceptible wildlife, even plant health or environmental health surveillance systems. It could improve the surveillance activities within one industry or across several different industries in the case of diseases affecting several industries simultaneously.

This study highlights the need to link every new surveillance component to existing components to make the surveillance more effective and more efficient. It is advisable to consider which tools could be pooled, to anticipate how to integrate and analyse the data (for instance, if capture–recapture analyses are expected), etc. prior to implementation of a new component. There is probably a need to develop a method for linking up existing or new components that are monitoring the same disease. Such a method could learn from the United States' experience, where a federal initiative has been designed to combine the existing animal health surveillance activities into a comprehensive system [31, 39].

CONCLUSIONS

This comparative assessment of three surveillance systems by means of OASIS has highlighted many possible synergies that could improve the effectiveness and efficiency of surveillance as a whole. In the future, additional economic analyses and a quantitative assessment should be planned in order to measure the potential impact of proposed improvements, and to prioritize them according to their cost-effectiveness. Such a comparative assessment could conceivably be applied to other surveillance networks and other animal species. This could be a preliminary step towards merging or better connecting several surveillance systems either in the same animal industry or in several animal industries within the same country.