Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2024-12-28T14:12:33.253Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing assessors: proposal for a guidance for evaluating the scientific performance of a pesticide regulatory authority

Published online by Cambridge University Press:  16 April 2021

Sari AUTIO Open the ORCID record for Sari AUTIO [Opens in a new window] ,
Herman KOËTER ,
Mar CARRETERO ,
Anthony HARDY  and
Alberto MANTOVANI
Sari AUTIO
Affiliation:
Safety and Chemicals Agency Tukes, Helsinki, Finland; email: [email protected].
Herman KOËTER
Affiliation:
Orange House Partnership, Brussels, Belgium.
Mar CARRETERO
Affiliation:
Regional Public Health Laboratory, Madrid, Spain.
Anthony HARDY
Affiliation:
Grayshott, UK.
Alberto MANTOVANI
Affiliation:
Istituto Superiore di Sanità, Rome, Italy.
Rights & Permissions [Opens in a new window]

Abstract

External evaluations by an international committee of the scientific risk assessment and decision-making processes of the Netherlands Competent Authority for the authorisation of plant protection products and biocides (Ctgb) were conducted in 2013 and 2018. Based on the experience drawn from them, a general guidance for such visitations is suggested. An international visitation committee examined the structure and management of the Authority, its human resources and staff policy, the scientific processes and output, the documentation and communication of its decisions and the mechanisms for keeping up to date with international scientific developments. Attention was paid to the degree of openness and transparency throughout the organisation and in particular when dealing with confidential information. From the experience gained it can be concluded that visitations not aiming at finding mistakes and omissions but instead focusing on recommendations and constructive suggestions will result in cooperation, mutual trust and acceptance of the recommendations made. A follow-up visitation after a few years can be effective in maintaining a traceable, high-level scientific output. In view of the strong drive towards the European Union-wide harmonisation of the regulatory practices of hazardous chemicals, a voluntary evaluation of regulatory authorities’ scientific performance is recommended as a means for organisational learning.

Type
Articles
Information
European Journal of Risk Regulation , Volume 14 , Issue 1 , March 2023 , pp. 178 - 190
Copyright
© The Author(s), 2021. Published by Cambridge University Press

I. Introduction

The accountability and reliability of regulatory authorities involved in the multidisciplinary risk assessment of hazardous chemicals is an extremely important prerequisite for public trust.Footnote 1 When citizens receive up-to-date information from regulatory authorities, whose risk communication is open, transparent and credible, it creates confidence and trust in their institutional performance.Footnote 2 Pesticides, including plant protection products and biocides, are intentionally designed to be toxic to living target organisms or to influence their fundamental life processes and thus have the potential to kill or control harmful organisms and pests. At the same time, they can cause unwanted or unintended adverse effects to other non-target organisms, human health and the environment,Footnote 3 and may thus increasingly create concerns amongst citizens.Footnote 4 Therefore, the release onto the market of these hazardous chemicals is strictly regulated in the European Union (EU), and their approval and authorisation are based on scientific risk assessments conducted by Competent Authorities of the Member States.Footnote 5 Based on European Regulations, active substances are approved at the Community level, and products containing those active substances are authorised at the national level in each Member State.Footnote 6 The Competent Authorities are involved in risk assessments for European-wide approvals of active substances and are responsible for product authorisations nationally.

Risk assessment is a process of assigning magnitudes and probabilities to the adverse effects of human activities or natural events.Footnote 7 The concept of risk is inherently a social construct, and the perception of it varies between individuals and different groups of people. Professionals tend to stick to regulatory definitions, which may create tensions and debates between experts and civil society if the process of risk assessment and risk management is not communicated to society in a proper and transparent manner.Footnote 8 While harmonised risk assessment methodologies for plant protection products and biocides exist in the EU, differences between countries in regulating environmental health risks are revealed and explained by Clahsen et al.Footnote 9 Such cultural differences in risk perceptionFootnote 10 and applications of local knowledge (eg specific agricultural practices and local conditions where a pesticide may cause harm to non-target organisms, such as in the case of neonicotinoids and pollinators)Footnote 11 can become obstacles for effective international cooperation within the risk assessment and management of hazardous chemicals unless the same principles are applied across different countries. The EU regulatory framework represents a major effort in this direction: the implementation of common criteria is particularly advanced for plant protection productsFootnote 12 and biocides,Footnote 13 whereas risk management is a Member State issue, and its practices may vary in specific cases (eg when introducing risk mitigation measures to prevent neonicotinoid-induced bee mortalities in EU Member States).Footnote 14 Furthermore, in wider international contexts, joint procedures are developed and recommended, for instance, by the Organisation for Economic Co-operation and Development.Footnote 15

Concerns about the risks of hazardous chemicals in European civil societies are increasingFootnote 16 and question the competence and scientific performance of authorities involved in assessment and decision-makingFootnote 17 to reduce the environmental impact and risks for human health of pesticides.Footnote 18 Such a strong EU-orientated integration of the legislation and risk assessment practices requires a high level of competence and expertise in the national authorities involved, ensuring that their performance is efficient, objective and adequate.Footnote 19 The Food and Veterinary Office of the European CommissionFootnote 20 carries out audits in the Member States in order to verify their implementation measures of pesticide legislation. However, there is no legally binding requirement for the evaluation of the scientific performance of national regulatory authorities; instead, such undertakings are on a voluntary basis. Nevertheless, scientific performance evaluation enables institutional learning and innovation of authorities.Footnote 21

In the Netherlands, the Board for the Authorisation of Plant Protection Products and Biocides (Ctgb) is the Competent Authority for the authorisation of pesticides.Footnote 22 At its request, an evaluation of its scientific processes, scientific output and decision-making processes was undertaken by an external international visitation committee (IVC) in 2013 and repeated in 2018 to follow the progress made during the intervening five years. Both visitations were carried out by five experienced senior experts on pesticide risk assessment and risk management from five different EU Member States, jointly representing broad expertise on pesticide risk assessment and risk management in the public sector. The individual experts were endorsed by the board of the Ctgb and the size of the committee was limited to five. The core of the committee remained the same in the two evaluations.

Following the definition of Donaldson and Scriven,Footnote 23 evaluation of research is a transdisciplinary activity that produces tools for knowledge users, such as risk managers, regulators or policy decision-makers. Much has been published on the different approaches, practices and methodologies of evaluation depending on the specific needs of the orderer, evaluee and evaluator.Footnote 24 There is a large volume of literature concerning the project, programme and practice evaluation of different fields in society,Footnote 25 but still there is not much practical guidance and there are few suggestions available on how to conduct an evaluation of a pesticide regulatory authority. Therefore, our aim in this paper is to provide future evaluation researchers with practical insights about details of the methodologies and information mining approaches that could be applied in such visitations. While details of the observations and related findings as well as specific conclusions and recommendations can be found in the reports of the Ctgb visitations,Footnote 26 this paper offers some general recommendations that arose from our experience.

II. Approach

1. International visitation committee

An important element of any visitation is the stature of the experts selected as candidate members of the committee.Footnote 27 Generally, the Chair is selected by the management of the Competent Authority or organisation. Potential committee members are generally identified by the appointed Committee Chair for endorsement by the Competent Authority. Drawing from the Ctgb experience, the selection of potential candidates is recommended based on the criteria as presented in Box 1 and supported by detailed curriculum vitae (CVs) and declarations of interest (DoIs).

Box 1: Criteria for selecting the international visitation committee members.

  • All members should have at least fifteen years of experience in life sciences in the public sector and/or as independent consultants.

  • Members should not have any direct or indirect interest in the organisation or in any of its staff, compatible with the guidelines of European authorities.

  • All members should have sufficient knowledge of European Union regulations and other international developments relevant for the project.

  • As a group, the committee must have adequate knowledge and experience in the risk assessment and risk management of chemicals in general, and specifically in pesticides.

  • All three European Union zones should be represented.

2. Terms of reference and action plan

Evidence-based policymaking at all levels of government, resulting in better public services, necessitates transparency in their evaluation practices.Footnote 28 Therefore, prior to the start of a visitation project, it is advisable to draft a terms of reference document describing the objective(s), the composition of the visitation committee, a brief explanation of the methodology of the evaluations, the timetable and the agreement on the confidentiality required for conducting an evaluation.

To ensure transparency, an agreement on the remuneration of the Committee members should be included in the terms of reference. An action plan for a visitation comprises a roadmap for the visitation and is essential as a tool to keep track of the progress of the project and to avoid omissions. The objective of the international visitation in the Ctgb was to assess the independence, the scientific quality and the legal compliance of the formal decision-making process and decision content following requests for the authorisation of plant protection products and biocides in the Netherlands, and it was agreed that the roadmap should provide the information presented in Box 2.

Box 2: Roadmap of a visitation.

  • Indication of data and information sources such as specific documents, scientific reports, minutes of meetings, communication notes and Internet searches to be made available to the evaluators.

  • Planning of interviews with senior management and a selection of scientific staff.

  • Assessment and evaluation plan of all relevant information gathered.

  • Draft compilation of all observations, conclusions and recommendations.

  • Providing the draft report of the visitation to the management of the organisation or Competent Authority (including the chief scientific officer) for scrutiny of the report for misunderstandings, errors and, as appropriate, unresolved disagreements.

  • Presenting the final report to the management board for endorsement and publication of the report.

3. Access to information and meetings

To serve best the users of an evaluation, it is essential to define which data are critical and should be available for scrutiny by the evaluators.Footnote 29 Taking into account the enormous number of documents to be considered, amongst which are summaries of discussions, minutes of meetings, internal notes and staff information, it is advisable to request the appointment of a staff member with the task of supporting the IVC with procedures for accessing databases, dossiers, documents and personnel files. This arrangement proved to be successful and saved a lot of work in the Ctgb evaluation. This person could also arrange for translation into English of essential documents that are only available in the language of the member country of concern.

Documents of interest for the visitation committee include those covering:

  1. (1) General management and decision-making processes;

  2. (2) Scientific staff policies;

  3. (3) Dossiers of authorisation requests; and

  4. (4) General informative documents.Footnote 30

Neither the authorisation fees nor comparisons to other Competent Authorities were within the scope of this evaluation.

The IVC should organise an appropriate number of meetings with the management and scientific staff of the Authority to be evaluated in order to promote a balanced approach towards learning and accountability.Footnote 31 During the visits, individual interviews with staff members can be held. In addition, the IVC members should communicate regularly between themselves via video calls and emails.

4. Staff policy

The quality of the scientific staff is the ultimate determinant of the reliability of the scientific output of the institution,Footnote 32 and therefore an evaluation should focus on the multitude of different disciplines necessary for conducting scientific risk assessments. Hence, in order to conduct a proper evaluation of the expert staff competences, we conclude that access is required to the annually updated, accurate CVs and DoIs of all scientific staff. As the visitation deals with personal data, the evaluators should always adhere to evaluators’ ethical guiding principlesFootnote 33 and keep the information confidential. Using the EU CV format is preferable, but if another format is used, the information must include the information presented in Box 3.

Box 3: Personal information required to evaluate staff competences.

  • Highest level of education (attach diploma/certificate).

  • Area(s) of expertise.

  • Years of experience in the field(s) of expertise in the public sector.

  • Years of relevant experience prior to current employer.

  • List of first author publications relevant to the work programme.

  • List of relevant presentations at conferences during the last five years.

  • Contributions to relevant advancements in the regulatory field (eg preparation of guidance documents, working group memberships, etc.).

  • Specific training in the relevant field(s) of expertise.

Interviewing of scientific staff members, individually and/or as teams, is an essential part of evaluation methodology.Footnote 34 In the case of the Ctgb evaluation, the open-ended interviews were formulated jointly by the IVC members, and topics covered included workload, teamwork, level of professional challenge, level of appreciation, appropriate level of responsibility, career options and awareness amongst the staff of the importance and relevance of their responsibilities for the health and safety of society and the environment. The senior management and human resources staff should also be interviewed separately from the staff members as their respective views affect the scientific output of the organisation.

Based on the experience of the IVC, a general recommendation can be made: the human resources staff could regularly stimulate and support the scientists to broaden and improve their knowledge and scientific competency,Footnote 35 such as by applying for the EU Registered Toxicologist (ERT)Footnote 36 status and participating in relevant international conferences.

5. Management of the risk assessment/risk management framework

National authorities have to deal with risk assessment and risk management according to the requirements of their respective national regulatory systems. While the EU regulationsFootnote 37 define the tasks of the Competent Authorities of the Member States, they do not state how the risk assessment and risk management tasks should be organised in each country. In some EU Member States, these two fundamental tasks are carried out by separate organisations, while in others they are carried out as distinct functions within one Competent Authority. In either case, the EU principles require that risk assessment activities be science-based, transparent and independent from risk management.Footnote 38 This is a critical point for the attention of a visitation committee. Risk assessment is based on the dossier of scientific studies according to internationally agreed study guidelines and guidance documents based on the Uniform Principles.Footnote 39 Risk management considers the outcome of the risk assessment as the starting point of its reasoned decision-making process, considering aspects such as local economic, agricultural, ethical and political consequences, and its necessary separation from risk assessment is emphasised by many authors.Footnote 40

Evaluation of how risk management is separated effectively from risk assessment is not easy to define and communicate. In different Member States there are several models on how risk assessment and risk management are separated within the Competent Authorities. The most transparent approach would be outsourcing of risk management to another public, independent and knowledgeable sister organisation in the same country and ensuring a robust and iterative science-based communication framework between the two organisations. Within the EU framework, a good example of separated tasks is the European Food Safety Authority (EFSA), with its risk assessment task and the Commission dealing with risk management (CHAFEA).Footnote 41

When the same organisation is responsible for both functions, its management should be responsible for ensuring that scientific risk assessment conclusions will not be blurred by risk management arguments. Ideally, the organisation will have a risk management unit that is physically separated from the risk assessment department. Therefore, the visitation should consider the organisational structure of the evaluated authority in detail. It is also necessary to scrutinise whether in its outputs there is a clear and understandable separation between the scientific risk assessment and the risk management decision, taken with full supporting arguments, and that this is presented in a transparent and consistent format.

6. Dossier evaluation and contribution to the implementation of the EU regulatory framework

Dossier evaluation is the cornerstone of a Competent Authority’s scientific activity, and therefore major effort is expected from an IVC to scrutinise dossier evaluations. The purpose of the scrutiny of dossier evaluations should not be to review the validity of the conclusions of specific risk assessments, but rather to get an overview of the scientific process of producing such documents in general. National Competent Authorities conduct several risk assessment activities. The same authorities responsible for the risk assessment and authorisation of plant protection productsFootnote 42 may also conduct similar tasks for biocides.Footnote 43 In addition to decision-making on the national authorisations of biocidal and plant protection products containing those active substances, these competent authorities may also act as Rapporteur Member States (RMSs) at the community level to produce the risk assessments of active ingredients of both pesticide categories for the EU. The authorities also extensively review assessments made by other Member States. Based on the expertise from the Ctgb evaluation, the authors developed a set of evaluation criteria for scrutinising the scientific processes and the quality of documents produced by the evaluee. To get an overview of these scientific processes, the criteria given in Table 1 are recommended for evaluating the quality of the respective documents.

Table 1: Recommended criteria for the evaluation of the quality of risk assessments of active substances intended for the European Union (EU) and for plant protection products and biocidal products intended for national authorisation decisions.

Because the risk assessment and risk management processes take time and involve several experts in the workflow, it is important to keep track of all actions during the scientific process. This internal traceability is usually enabled by the document management systems (DMSs) of regulatory organisations. Internal traceability is a requisite for the transparency and trustworthiness of the whole risk assessment and decision-making process of any Competent Authority.Footnote 44 In order to trace the full assessment and evaluation activities trail, evaluators’ access to the internal electronic DMS of the Competent Authority is indispensable. When used properly, all documents will be recorded in the system in a timely fashion. A DMS greatly increases the availability of the documents and thus contributes to the efficiency of the whole scientific risk assessment process.

Visitation committees are advised to encourage the Competent Authority to contribute proactively to the harmonisation of the EU regulatory framework. For example, the forum of authorities within an EU zone is recognised as an important platform for discussing and solving long-running zonal issues in the field of pesticide legislations.Footnote 45 Actions to establish such fora are valuable steps for strengthening zonal cooperation.Footnote 46 It is also important that the scientific experts have opportunities to participate actively and contribute to the development of international guidance documents and risk assessment methodologies.Footnote 47 Therefore, we recommend that such activities should be included in the evaluations of the scientific performance of Competent Authorities.

III. Openness, transparency and challenges of communication

In Europe, Competent Authorities are leading regulatory forces whose activities could inspire other national bodies as well. A well-documented and transparent strategy would help to distinguish between the risk assessment and subsequent risk management decisions, enhance communication with EFSA, other national authorities and the wider society and promote greater public trust.Footnote 48

To this purpose, visitation committees should evaluate whether the conceptual framework of the Competent Authority to distinguish risk assessment from risk management is in line with the European general food legislation and shows an appropriate level of openness and transparency throughout the work processes within the organisation.Footnote 49 Such a framework would provide the necessary insight into the interactions between the scientific risk assessment conclusions and risk management arguments in order to finalise the regulatory decisions. Specifically, we recommend that the minutes and records of the discussions by the decision-making body of the organisation at hand should clearly identify any changes made by it in the scientific assessment reports submitted for its consideration. Furthermore, as discussed previously, all staff involved should be aware that internal traceability of activities is a requisite for the transparency and trustworthiness of the whole process.

Openness and transparency are recommended in communication with other similar regulatory authorities. The development of public trust requires clear, effective and open communication with the outside world, including all stakeholders and the wider international regulatory community.Footnote 50 This is also the overarching principle of the European General Food Law Regulation.Footnote 51

We recognise the primary need to communicate with stakeholders as well as the general public of the particular country using the national language. However, the translations into English of publicly available documents will increase the transparency to the external world, primarily the other Competent Authorities of other EU Member States and other international organisations.

The balance between confidentiality and data protection issues on the one hand and openness and transparency on the other is fundamental for a scientific authority, since both are key values underpinning risk assessment and risk management, respectively.Footnote 52 Industry tends towards increased confidentiality and protection of data, backed up by fairness of competition and the enterprise’s right to defend its own invention.Footnote 53 . From the non-governmental organisations (NGOs), there is pressure for increased transparency and free access to the dossiers with the aim of minimising human and environmental exposure and risks.Footnote 54 One recent example of clashing views of stakeholders is the case of neonicotinoids, with arguments by the industry for refusing access to the requested information about possible adverse environmental effects claimed by the NGOs.Footnote 55 A legislative stream from the Aarhus ConventionFootnote 56 through to the recent decision of the EU General CourtFootnote 57 gives priority to the right to demand access to safety-relevant information. In the Netherlands, stakeholders’ engagement in environmental policy has a long tradition, called the Polder Model, which calls for open knowledge-sharing between authorities and citizens.Footnote 58 Developing a balanced approach between confidentiality and transparency is an ongoing process involving the whole EU regulatory framework, where scientific authorities are important stakeholders who can apply reflective practices in their risk communication.Footnote 59

In line with the general approach of the EFSA Conference 2018, we emphasise that trust cannot be requested, it can only be given.Footnote 60 Without a policy of transparency wherever possible, a regulatory body may in time become isolated amongst its peers at the international level.

IV. General conclusions based on the Ctgb evaluations

Considering the evaluation policy of the United Nations Economic Commission for Europe,Footnote 61 we find the scientific performance evaluation to be a valuable means for organisational learning, strengthening the accountability of public bodies and providing societal benefits.Footnote 62 To our knowledge, beyond the mandatory Food and Veterinary Office of the European Commission (FVO) audits,Footnote 63 a voluntary scientific performance evaluation, as performed in the Netherlands, is unique amongst the Competent Authorities for authorising hazardous chemicals. Drawing on our experience, we brought insights and advice and suggested a general guidance for conducting evaluations of similar organisations in this paper. The evaluation was carried out successfully, not only with respect to its approach, action plan and execution, but also in dealing constructively with its observations and subsequent recommendations. Expert panels carrying out voluntary reviews should be perceived as, and behave as, advisors rather than examiners. We were very pleased with the assistance provided by the Ctgb staff in helping to find the necessary information we required to evaluate scientific output. During the on-site visits, the IVC was facilitated by the organisation’s staff to achieve a clear and comprehensive understanding of the organisation’s structure and functioning and to gain access to relevant and adequately detailed information. This unimpeded access and allowance to review all science-related information signalled self-confidence with respect to the scientific approach and output of the organisation as well as confidence in the added value provided by the IVC assessment. The progress made by the Ctgb in five years, reported in detail in the visitation report of 2018,Footnote 64 indicates that periodic external evaluation by international expert panels could assist the Competent Authorities in many countries to develop their scientific capacities and streamline their processes for gaining trust amongst society and achieving a high level of protection of human health and the environment.

Footnotes

The international visitation committee acknowledge the support and assistance offered by all Board for the Authorisation of Plant Protection Products and Biocides (Ctgb) staff members and would like to thank them all for their patience and willingness to spend their precious time with us for interviews and technical help. In particular, we would like to thank the chairman and the members of the board, as well as the director of the Ctgb, for their trust in the capacity, capability and expertise of the international visitation committee membership. We are also grateful to the project manager for providing us with all necessary documents and information. He has been our beacon, always patient and tireless, when some of us got lost in the sea of documents. Finally, we thank the unknown peer reviewer for suggesting significant improvements to this article. The authors declare that they have no competing interests.

References

1 T Tweedale, A Lysimachou and H Muilerman, “Missed & dismissed, pesticide regulators ignore the legal obligation to use independent science for deriving safe exposure levels” (2014) Report published by Pesticide Action Network Europe and Generation Futures, 29 pp.

2 O Renn, Risk Governance. Coping with Uncertainty in a Complex World (London, Earthscan, 2008) pp 222–27.

3 O Nesheim, F Frederick Fishel and M Mossler, “Toxicity of pesticides” (2019) EDIS 2005 (8). <https://journals.flvc.org/edis/article/view/114995> (last accessed 19 March 2021).

4 See, eg, VL Zikankuba, G Mwanyika, JE Ntwenya and A James, “Pesticide regulations and their malpractice implications on food and environment safety” (2019) 5 Cogent Food & Agriculture 1601544.

5 T Hardy, S Bopp, M Egsmose, H Fontier, L Mohimont, H Steinkellner and F Streissl, “Risk assessment of plant protection products” (2012) 10(10) EFSA Journal S1010.

6 Regulation (EC) No 1107/2009 of the European Parliament and of the Council of 21 October 2009 concerning the placing of plant protection products on the market and repealing Council Directives 79/117/EEC and 91/414/EEC. OJ L 309; Regulation (EU) No 528/2012 of the European Parliament and of the Council of 22 May 2012 concerning the making available on the market and use of biocidal products. OJ L 167.

7 GW Suter II (ed.), Ecological Risk Assessment (Boca Raton, FL, Lewis Publishers 1993).

8 Renn, supra, note 2, pp 239–41.

9 SCS Clahsen, I van Kemp, BC Hakkert, TG Vermeire, AH Piersma and E Lebret, “Why do countries regulate environmental health risks differently? A theoretical perspective” (2019) 39(2) Risk Analysis 439.

10 BB Johnson and B Swedlow, “Cultural theory’s contributions to risk analysis: a thematic review with directions and resources for further research” (2021) 41(3) Risk Analysis 429.

11 RL Anjum and E Rocca, “From ideal to real risk: philosophy of causation meets risk analysis” (2019) 39(3) Risk Analysis 729.

12 European Commission, “Pesticides” <https://ec.europa.eu/food/plant/pesticides_en> (last accessed 19 March 2021).

13 European Commission, “Biocides” <https://ec.europa.eu/health/biocides/overview_en> (last accessed 19 March 2021).

14 D Wintermantel, JF Odoux, A Decourtye, M Henry, F Allier and V Bretagnolle, “Neonicotinoid-induced mortality risk for bees foraging on oilseed rape nectar persists despite EU moratorium” (2020) 704 Science of the Total Environment 135400.

15 OECD, “Guidance Document on the Planning and Implementation of Joint Reviews of Pesticides” (2011) Series on Pesticides No. 60 ENV/JM/MONO (2011)11. Organisation for Economic Co-Operation and Development. Environment Directorate, Joint Meeting of the Chemicals Committee and The Working Party on Chemicals, Pesticides and Biotechnology, 54 pp.

16 S Koch, A Epp, M Lohmann & G-F Böl, “Pesticide residues in food: attitudes, beliefs, and misconceptions among conventional and organic consumers” (2017) 80(12) Journal of Food Protection 2083.

17 A Mie, C Rudén and P Grandjean, “Safety of safety evaluation of pesticides: developmental neurotoxicity of chlorpyrifos and chlorpyrifos-methyl” (2018) 17 Environmental Health 77.

18 European Commission, “Farm to Fork Strategy for a fair, healthy and environmentally-friendly food system” (2020) <https://ec.europa.eu/food/sites/food/files/safety/docs/f2f_action-plan_2020_strategy-info_en.pdf> (last accessed 19 March 2021).

19 H Deluyker, A Rodrıguez Pena, M Scannell, J Tarazona and B Url, “What does the future hold for assessment science?” (2016) 14(S1) EFSA Journal S0501.

20 Food and Veterinary Office of the European Commission (2020) <https://ec.europa.eu/food/audits-analysis/audit_reports/index.cfm> (last accessed 19 March 2021).

21 S Kuhlmann and J Bogumil, “Performance measurement and benchmarking as ‘reflexive institutions’ for local governments: Germany, Sweden and England compared” (2018) 31(4) International Journal of Public Sector Management 543; C Luederitz, N Schapke, A Wiek, DJ Lang, M Bergmann, JJ Bos et al, “Learning through evaluation – a tentative evaluative scheme for sustainability transition experiments” (2017) 169 Journal of Cleaner Production 61.

22 The Decree on Mandate, Authorization and Representation by the Ctgb, 3 March 2011, Government Gazette No 4789, the Netherlands, 18 March 2011.

23 SI Donaldson and M Scriven (eds.), Evaluating Social Programmes and Problems. Visions for the New Millennium. The Stauffer Symposium on Applied Psychology at the Claremont Colleges (Mahwah, NJ, Lawrence Erlbaum Associates Publishers, 2003).

24 IF Shaw, JC Greene and MM Mark (eds.), Handbook of Evaluation: Policies, Programmes and Practices (Thousand Oaks, CA, SAGE Publications 2006).

25 DW Compton and ML Baizerman, “The evolution of a philosophy and practice of evaluation advice” (2012) 136 New Directions for Evaluation 67.

26 H Koëter, S Autio, U Banasiak, M Lynch, V Silano and A Cuvillier, Report on the international visitation of the Board for the authorization of plant protection products and biocides (Ctgb) in the Netherlands addressing the scientific process, the scientific output and the decision-making process (Lucca, Orange House Partnership, 2013); H Koëter, S Autio, M Carretero, T Hardy and A Mantovani, Report of the second visitation of the Netherlands Board for the Authorization of Plant Protection Products and Biocides (Ctgb) addressing the scientific process, the scientific output and the decision-making process. (Lucca, Orange House Partnership, 2018).

27 E Desmedt, D Morin, V Pattyn and M Brans, “Impact of performance audit on the administration: a Belgian study (2005–2010)” (2017) 32(3) Managerial Auditing Journal 251.

28 J Lyytimäki, T Assmuth and M Hildén, “Communicating chemical risks for social learning: findings from an expert opinion survey” (2009) 8(3–4) Applied Environmental Education & Communication 174.

29 S Eckhard and V Jankauskas, “The politics of evaluation in international organizations: a comparative study of stakeholder influence potential” (2019) 25(1) Evaluation 62.

30 C Donovan, “State of the art in assessing research impact: introduction to a special issue” (2011) 20(3) Research Evaluation 175.

31 R Lahey and S Nielsen, “Rethinking the relationship among monitoring, evaluation, and results-based management: observations from Canada” (2013) 137 New Directions for Evaluation 45.

32 B Wolf, T Lindenthal, M Szerencsits, JB Holbrook and J Heß, “Evaluating research beyond scientific impact. How to include criteria for productive interactions and impact on practice and society” (2013) 22(2) GAIA 104.

33 See, eg, American Evaluation Association, AEA Guiding Principles. Public Statement on Cultural Competence in Evaluation (Washington, DC, American Evaluation Association 2011).

34 S Bell, B Shaw and A Boaz, “Real-world approaches to assessing the impact of environmental research on policy” (2011) 20(3) Research Evaluation 227.

35 H Selck, PB Adamsen, T Backhaus, GT Banta, PKH Bruce, G Allen Burton Jr et al, “Assessing and managing multiple risks in a changing world – the Roskilde recommendations” (2017) 36 Environmental Toxicology and Chemistry 7.

36 J Fowler and CL Galli, “EUROTOX’s view regarding the role and training of certified European registered toxicologists (ERT)” (2007) 168 Toxicology Letters 192.

37 Supra, note 6.

38 European Food Safety Authority, “Risk Assessment vs. Risk Management: What’s the Difference?” (Food Safety News, 23 April 2014) <https://www.foodsafetynews.com/2014/04/risk-assessment-vs-risk-management-whats-the-difference/> (last accessed 19 March 2021).

39 Commission Regulation (EU) No. 546/2011 implementing Regulation (EC) No 1107/2009 of the European Parliament and of the Council as regards uniform principles for evaluation and authorization of plant protection products. [2011] OJ L 155/127.

40 See, eg, E Ingre-Khans, M Agerstrand, A Beronius and C Ruden, “Transparency of chemical risk assessment data under REACH” (2016) 18 Environmental Science. Processes & Impacts 1508.

41 European Commission, Consumers, Health, Agriculture and Food Executive Agency <https://ec.europa.eu/chafea/about/who-we-are_en.htm> (last accessed 19 March 2021).

42 Regulation (EC) No 1107/2009, supra, note 6.

43 Regulation (EU) No 528/2012, supra, note 6.

44 European Parliament resolution of 16 January 2019 on the Union’s authorization procedure for pesticides (2018/2153(INI)).

45 European Commission, “Guidance document on zonal evaluation and mutual recognition under Regulation (EC) No 1107/2009.” SANCO/13169/2010 rev. 11 (January 2021).

46 M Stenrød, M Almvik, OM Eklo, AL Gimsing, R Holten, K Künnis-Beres et al, “Pesticide regulatory risk assessment, monitoring, and fate studies in the northern zone: recommendations from a Nordic–Baltic workshop” (2016) 23 Environmental Science and Pollution Research 15779.

47 See, eg, OECD, “Guidance Document on the Planning and Implementation of Joint Reviews of Pesticides” (2011) Series on Pesticides No. 60 ENV/JM/MONO (2011)11. Organisation for Economic Co-Operation and Development. Environment Directorate, Joint Meeting of the Chemicals Committee and The Working Party on Chemicals, Pesticides and Biotechnology, 54 pp.

48 A Smith, L Parrino, D Vrbos, G Nicolini, M Bucchi, M Carr et al, “Communicating and engaging with the public in regulatory science” (2019) 17(S1) EFSA Journal e170717.

49 Regulation (EU) 2019/1381 of the European Parliament and of the Council of 20 June 2019 on the transparency and sustainability of the EU risk assessment in the food chain and amending Regulations (EC) No 178/2002, (EC) No 1829/2003, (EC) No 1831/2003, (EC) No 2065/2003, (EC) No 1935/2004, (EC) No 1331/2008, (EC) No 1107/2009, (EU) 2015/2283 and Directive 2001/18/EC (text with EEA relevance). [2019] OJ L 231.

50 European Chemicals Agency, “Review of the Policy of avoiding potential conflicts of interest”. 51st Meeting of the Management Board 0-21 September 2018. MB/40/2018 final, 21.9.2018. Public, 5 pp. <https://echa.europa.eu/documents/10162/3430273/FINAL_MB_40_2018_%281%29_Revision_CoI_policy_MB51.pdf/4a14e991-89ab-52ec-67b6-e9e93181e738> (last accessed 19 March 2021); European Food Safety Authority, “EFSA’s policy on independence. How the European Food Safety Authority assures the impartiality of professionals contributing to its operations.” Adopted in Parma on 21 June 2017. mb170621-a2, 9 pp. <https://www.efsa.europa.eu/sites/default/files/corporate_publications/files/policy_independence.pdf> (last accessed 19 March 2021).

51 Regulation (EC) No 178/2002 of the European Parliament and of the Council laying down the general principles and requirements of food law, establishing the European Food Safety Authority and laying down procedures in matters of food safety of 28 January 2002, OJ L 031.

52 M Blastland, ALJ Freeman, S van der Linden, TM Marteau and D Spiegelhalter, “Five rules for evidence communication” (2020) 587 Nature 362.

53 Directive (EU) 2016/943 of the European Parliament and of the Council on the Protection of Undisclosed Know-How and Business Information (Trade Secrets) Against Their Unlawful Acquisition, Use and Disclosure. [2016] OJ L 157/1.

54 See, eg, C Robinson, CJ Portier, A Čavoški, R Mesnage, A Roger, P Clausing et al, “Achieving a high level of protection from pesticides in Europe: problems with the current risk assessment procedure and solutions” (2020) 11 European Journal of Risk Regulation 450.

55 P McGrath, “Politics meets Science: The case of neonicotinoid insecticides in Europe” (2014) 7(1) S.A.P.I.EN.S Surveys and Perspectives Integrating Environment and Society <http://journals.openedition.org/sapiens/1648> (last accessed 19 March 2021).

56 The United Nations Economic Commission for Europe (UNECE) Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters (The Aarhus Convention) [1998] <https://ec.europa.eu/environment/aarhus/index.htm> (last accessed 19 March 2021).

57 General Court of the European Union, “EFSA’s decisions refusing access to the toxicity and carcinogenicity studies on the active substance glyphosate are annulled.” PRESS RELEASE No 25/19, Luxembourg, 7 March 2019 <https://curia.europa.eu/jcms/upload/docs/application/pdf/2019-03/cp190025en.pdf> (last accessed 19 March 2021).

58 Y Schreuder, “The Polder Model in Dutch economic and environmental planning” (2001) 21(4) Bulletin of Science, Technology & Society 237.

59 W Ulrich, “Reflective practice in the civil society: the contribution of critically systemic thinking” (2000) 1(2) Reflective Practice 247.

60 I Devos, KC Elliott and A Hardy (eds.), “Proceedings of the 3rd EFSA International Conference 2018: Science, Food, Society” (2019). 17 EFSA Journal Special Issue S1 <https://efsa.onlinelibrary.wiley.com/toc/18314732/2019/17/S1> (last accessed 19 March 2021).

61 UNECE, “United Nations Economic Commission for Europe Evaluation Policy” (2014) <http://www.unece.org/fileadmin/DAM/OPEN_UNECE/03_Evaluation_and_Audit/UNECE_Evaluation_Policy_October_2014.pdf> (last accessed 19 March 2021).

62 MM Mark and GT Henry, “The mechanisms and outcomes of evaluation influence” (2004) 10(1) Evaluation 35.

63 Food and Veterinary Office of the European Commission <https://ec.europa.eu/food/audits-analysis/audit_reports/index.cfm> (last accessed 19 March 2021).

64 Koëter et al., Report of the second visitation, supra, note 26, Annex 7, pp 98–103.

Figure 0

Table 1: Recommended criteria for the evaluation of the quality of risk assessments of active substances intended for the European Union (EU) and for plant protection products and biocidal products intended for national authorisation decisions.