The development process of economic evaluation guidelines in low- and middle-income countries: a systematic review

Abstract

Objectives

To review the development of economic evaluation guidelines (EEGs) in low- and middle-income countries (LMICs), with the goal of assisting those developing EEGs in LMICs.

Methods

We conducted a systematic search in MEDLINE (Ovid), PubMed, EconLit, Embase (Ovid), the Cochrane Library, and the gray literature until March 2021. We extracted data on the methods used in the EEG development process, the responsible party engaged, and the development team’s composition. We conducted a quality assessment, using the Appraisal of Guidelines for Research and Evaluation-Health Systems tool, and then carried out a relative comparison.

Results

Fourteen EEGs and nine studies were identified. In ten countries, the Ministry of Health was responsible for handling the development process. The majority of LMICs who developed EEGs did not explicitly report the discipline of those involved in the process. The developers of EEGs followed four main steps: conducting a review on national guidelines, organizing workshops, and getting support from international experts or from organizations. One-third of the identified EEGs failed to engage multisectoral or multidisciplinary developers, and approximately 14 percent did not follow or report any recommended step.

Conclusions

This study identified a scarcity of published information related to the development process and the suboptimal quality of included studies. It provides relevant material to support international organizations and developers of guidelines in LMICs in developing EEGs that fit their national context. In addition, this paper recommends a transparent approach to the design of guidelines and to reporting on the methods for developing them.

Whatever the economic setting, health systems are struggling with healthcare resources allocation and the implementation of universal health coverage (UHC) (1). The World Health Assembly held in 2014 marked health technology assessment (HTA) as a crucial tool for UHC (1). HTA supports policymakers in dealing with their budgetary constraints by facilitating an equitable, efficient, and high-quality health system (1;2). Economic evaluation (EE) is an essential component of HTA, requested in many countries to support reimbursement decisions by providing information on the cost-effectiveness of health technologies (3;4). For harmonization and comparability purposes, economic evaluation guidelines (EEG) have been developed and applied in many countries, particularly European and North American countries, where HTA is well established (5–7). Along with other health systems guidance (HSG), EEGs provide critical input and support for decision-making for effective health system programs; accordingly, a well-described process is essential to ensure the transparency and objectivity of these guidelines (8;9).

Recently, the interest in and use of HTA has increased in low- and middle-income countries (LMICs), leading to the development of EEGs in these countries, such as Brazil, Cuba, Egypt, and Indonesia (7). Although EEGs are emerging in LMICs, several countries do not yet have an established EEG (7). Developing national EEGs is crucial to providing researchers with practical steps for conducting their EEs (10). International organizations have demonstrated interest through their contribution to this quality domain (8;11;12). The World Health Organization (WHO), in collaboration with the Swiss Center for International Health, developed a handbook for supporting the development of HSG (8). Similarly, in 2015 the International Network of Agencies for HTA (INAHTA) created the LMICs working group, Guidelines International Network (GIN), to explore methods for promoting guideline development, adaptation, dissemination, implementation, and research within developing countries (12). Furthermore, the Appraisal of Guidelines for Research and Evaluation (AGREE) enterprise developed a new appraisal tool for assessing the quality of HSG, the Appraisal of Guidelines for Research and Evaluation-Health Systems (AGREE-HS) (11). At the 2020 international HTA global policy forum, researchers proposed the adoption of transparency, inclusivity, and impartiality as starting principles in developing a deliberative process in HTA (13;14), bringing attention to the process of developing guidelines.

Despite the importance and the value of the development process of EEGs, there is a lack of published comprehensive literature on this subject. Understanding how previous EEGs have been developed could provide relevant insights for the next countries to develop EEGs in their context. Accordingly, this systematic review aimed to review the process and sources of evidence in developing guidelines for EEGs in LMICs, with the ultimate goal of assisting and supporting developers from LMICs, in addition to identifying gaps in research, and providing recommendations about each significant step in the development of guidelines for LMICs aiming to design their own EEGs.

Methods

A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA-S) guidelines (15). The protocol of this review was registered with the Open Science framework platform (Registration DOI: https://doi.org/10.17605/OSF.IO/DHRYF).

Systematic review

Literature Search Strategies

We searched MEDLINE (Ovid), PubMed, EconLit, Embase (Ovid), and the Cochrane Library from the inception date to 9 February 2020, and performed a literature search update on March 2021.

In our search strategy, we combined three key concepts: “economic evaluation” AND “guidelines,” AND “low and middle-income countries.” For the last concept, we used the Cochrane filter 2012 (16), which we adapted to the 2020–21 World Bank classification. For the three concepts, we mapped controlled vocabulary, such as Medical Subject Headings (MeSH) terms and keywords. A medical information specialist validated the strategy. The full search strategies for MEDLINE, PubMed, and Embase are provided in Supplementary Material 1.

Moreover, the gray literature was searched, including the Web sites of the World Bank, WHO, the Professional Society for Health Economics and Outcomes Research (ISPOR), pharmacoeconomics guidelines around the world, the international Decision Support Initiative (iDSI), the Guide to Health Economic Analysis and Research (GEAR), as sources of eligible documents, and Epistemonikos as a source of systematic reviews. In addition, the Web sites of country-specific HTA agencies or Ministries of Health were reviewed (provided in Supplementary Material 2). A backward citation tracking for relevant systematic reviews and included guidelines was conducted. The search was neither limited to a specific language nor to a publication date.

We performed these literature searches in preparation for conducting two systematic reviews; the first was to identify and review the key features of EEGs in LMICs, and the second one to understand and to review the development process of these EEGs (7).

Eligibility Criteria

The official versions of EEGs, including pharmacoeconomics guidelines (PEGs) and drugs guidelines, developed by the national agencies of relevant LMICs, and publications related to their development process were included.

EEGs from high-income countries, EEs studies and reports regarding diseases, interventions, and other topics not relevant to guidelines and their development process were excluded. Nonoriginal research documents and studies or expert opinions regarding EEGs, systematic reviews, and unofficial published guidelines were also excluded. An article is considered original research if it is the report of a study written by the researchers who actually did the study, if the researchers describe their hypothesis or research question and the purpose of the study, if the researchers detail their research methods, if the results of the research are reported, and if the researchers interpret their results and discuss possible implications (17).

Study Identification and Screening

All records identified by the search were retrieved. To determine eligibility, titles and abstracts were screened by one reviewer (C.D.) and full texts of selected references were assessed by two pairs of reviewers (C.D./R.K. and C.D./R.R.). A calibration exercise was conducted prior to proceeding. Documents in languages other than English, French, and Arabic were translated via the online.doc.translator Web site. Discrepancies were resolved through discussions between reviewers.

Data Extraction

Data related to the methods and steps used in the EEG development process, the responsible party engaged in setting the guidelines, and the composition of the development team if reported, were extracted. A calibration exercise was conducted before data extraction. Two pairs of authors (C.D./R.K. and C.D./R.R.) extracted relevant data from the included studies independently and in duplicate. Disagreements between reviewers were solved through discussions to reach a consensus.

Data Analysis

A narrative synthesis of collected data was done. Then, a comparative summary table was presented.

Quality Assessment

We conducted a quality assessment of the development process of included EEGs using the AGREE-HS tool, which was adapted to the scope of our study.

While the AGREE-HS tool represents five domains (topic, participants, methods, recommendations, and implementability), we selected two quality domains that answer our research questions in identifying the steps in developing EEGs in LMICs and the composition of the development team. Accordingly, we adopted the term process to define participants and methods domains. “Participants” is the item that addresses the composition of the HSG development team. “Methods” addresses the use of systematic methods and transparency in reporting, along with use of the best available and up-to-date evidence (11). We developed a quality assessment document for each country, where appraisers reported their comments and scoring (Supplementary Material 3). Each item was assessed on whether its criteria have been met, whereby appraisers took into consideration whether the item-related content was well reported, easy to find, and easy to understand. For consensus, we performed a practical appraiser’s exercise (AGREE-HS appraisal of an eligible EEG).

Appraisers rated each item using a 7-point response scale. A score of 1 (lowest quality) was given if there was no information relevant to the AGREE-HS item, if the criteria were very poorly reported in the HSG document, or if the authors explicitly stated that it was not done; whereas a score of 7 (highest quality) was given if the information related to the AGREE-HS item was exceptionally well reported, all criteria related to the item were considered during the development of the guidance, and the information related to the item was applicable in its context. Scores between 2 and 6 were assigned when not all criteria of the AGREE-HS item were met, depending on the completeness and quality of reporting. Scores increased as more criteria were met (11). Two appraisers (C.D./R.K. and C.D./R.R.) independently assessed the process adopted by each country in developing its EEGs, using the accompanying eligible studies related to the development process. The mean AGREE-HS item scores were calculated and presented for each document.

In the absence of any empirical basis for deriving thresholds and defining high- and low-quality HSG, appraisers assessed the process based on a relative comparison between LMICs of each quality domain scores.

Results

The flowchart of study selection is detailed in Figure 1. In February 2020, twenty-one records were included; of these, thirteen were EEGs and eight were discussing and describing the development process. In addition, with the literature update in March 2021, one guideline reporting on the first edition of the Philippine HTA methods guide published in September 2020, new Chinese guidelines developed in 2020 to replace the previous ones, one article published in 2021, entitled: “Current Development and Practice of Pharmacoeconomic Evaluation Guidelines for Universal Health Coverage in China,” and one Indian guideline entitled: “Health Technology Assessment in India: A manual” were included.

Figure 1. PRISMA diagram of study selection. BIA, budget impact analysis; EE, economic evaluation; EEG, economic evaluation guidelines; GEAR, guide to health economic analysis and research; HICs, high-income countries; ISPOR, Professional Society for Health Economics and Outcomes Research; LMICs, low- and middle-income countries.

EEGs in LMICs

Based on the 2020–21 World Bank classification, fourteen LMICs (10.4 percent) had EEGs, including none of the twenty-nine low-income countries (LICs), five out of fifty lower-middle-income countries (Bhutan, Egypt, India, Indonesia, and the Philippines) and nine out of fifty-six upper-middle-income countries (Brazil, China, Columbia, Cuba, Malaysia, Mexico, the Russian Federation, South Africa, and Thailand). Moreover, Mercosur, a union of countries that includes Argentina, Brazil, Paraguay, Venezuela (upper-middle-income countries), Bolivia (a lower-middle-income country), and Uruguay (a high-income country, which is not covered by this review), has developed a common guideline (18–37).

Table 1 includes the country’s name, its guidelines, and articles describing the development process, responsible party, publication year, and type of document.

Table 1. Summary of Countries’ Eligible Documents

EMTD, Essential Medicines and Technology Division; HTA, health technology assessment; HTAP, health technology assessment panel; IETS, Instituto de Evaluación Tecnológica en Salud; InaHTAC, Indonesian Health Technology Assessment Committee; NA, not applicable.

Responsible Party and Team Composition

The Ministry of Health was the responsible party handling the development of three-fourths of the identified guidelines (n = 10) (Bhutan, Brazil, Cuba, Egypt, India, Indonesia, Malaysia, the Philippines, the Russian Federation, and South Africa), while nonprofit independent institutes were commissioned to develop 28.6 percent (n = 4) (China, Colombia, Mexico, and Thailand) of the identified guidelines.

The majority of LMICs who have developed EEGs did not explicitly report the discipline of those who were involved in the development process. Elsisi et al. and the Indonesian Health Technology Assessment Committee (InaHTAC) clearly reported the make-up of the teams involved in developing guidelines in Egypt and Indonesia (20;37), respectively, while others specified the working position (19–21;28;32) or the sector (22;25;34;35) or did not report it at all (18;23;24;26;27). Heterogeneity in extracted data was observed; the team composition results categorized members mostly as academicians, workers in the private or public sectors, and program managers. We assessed the team composition in the second part of this paper.

Development Methods

Table 2 summarizes the development steps of EEGs in LMICs. Each country carried out a specific process for developing its guidelines; many steps have been reported and identified. Some countries (n = 6) (Colombia, Cuba, Egypt, Malaysia, the Philippines, and Thailand) began by reviewing other national guidelines. In addition, nine countries (Bhutan, Brazil, China, Cuba, Egypt, Malaysia, Mexico, the Philippines, and Thailand) arranged workshops and consultation meetings for experts and stakeholders to propose recommendations and updates.

Table 2. Summary of the Development Steps of EEGs in LMICs

EEGs, economic evaluation guidelines; LMICs, low- and middle-income countries.

Five out of fourteen countries (Bhutan, Colombia, Cuba, India, and Indonesia) developed their EEGs with the support of international experts. Furthermore, in four of the fourteen countries (Brazil, Cuba, Indonesia, and the Philippines), International organizations such as WHO, the Pan American Health Organization (PAHO), the European Union, iDSI, the Health intervention and technology assessment program (HiTAP) of Thailand, and the National Institute for Health and Care Excellence (NICE), contributed in the development of EEGs.

Three countries (India, the Russian Federation, and South Africa) did not report the steps followed to develop their EEGs.

Quality Assessment Results

Table 3 summarizes the quality assessment results of the two domains: participants and methods. The Cuban guidelines had the highest final score for participants, while the Egyptian and Philippines guidelines had the highest final score for methods. In detail, Cuba clearly reported in its guidelines that their development team for HSG includes members who have an interest or stake in the recommendations. Moreover, the guidance development team was multidisciplinary and multisectoral. In the two documents describing their guidelines development process, Egypt reported that systematic and transparent methods were used to identify and review the evidence, and to reach a consensus regarding the final recommendations. In addition, Egypt is considered the best available and most contextually relevant evidence. Similarly, in their guidelines document, the Philippines addressed the use of systematic methods and transparency in reporting, as well as the use of the best available and up-to-date evidence.

Table 3. Quality Assessment Results

* Each item was assessed on whether its criteria have been met. A score of 1 was assigned for lowest quality, a score of 7 was assigned for the highest quality, and scores ranging between 2 and 6 were assigned when some of the criteria of the AGREE-HS item have been met, with higher scores indicating higher quality.

Discussion

A transparent approach to reporting the development process of EEGs is fundamental to assessing this process and generating information that can be beneficial for guidelines developers (13;38). This is especially true for LMICs, where experience in developing EEGs is limited and this process is uniquely challenging (7). In this paper, we systematically reviewed the development process of fourteen EEGs in LMICs, that is, the participants and the steps or the methods followed, and we assessed the quality of this process.

We identified fourteen EEGs in LMICs, compared with the thirteen reported by Daccache et al. in the systematic review conducted in June 2020 (7). This highlights that guidelines development is an ongoing process in LMICs, which are slowly but steadily adopting EEGs as a support tool for advice about the allocation of their scarce resources.

Our findings indicate that the responsible party or the development agency in the majority of included EEGs was the Ministry of Public Health, while in high-income countries (HICs) such as Canada, the United Kingdom, and Australia, independent bodies manage the development process (39–41). The responsibilities of Ministries of Health in system guidance in LMICs, where regulations and standards are de jure governance roles, are emphasized in the literature (42–44). Outside the government, this process can be improved by consultation with other major stakeholders, including the health professions, the private sector, civil society, philanthropic organizations, and academia, and by encouraging feedback by publishing discussion papers that set out draft proposals (44). Developers should bear in mind that during consultation, industry groups might seek to weaken regulation and to shift the goals that the government is seeking to achieve (44). The ministries’ involvement may ensure the implementability of these guidelines in developing countries.

Interestingly, we found reporting to be poor about participants involved in the development process: sector, interest, and discipline were not explicitly disclosed in the vast majority of included guidelines. Due to the heterogeneity of the extracted data, we failed to coherently present the team composition results. Inconsistencies regarding this issue are found even in HICs. For instance, Australia’s Pharmaceutical Benefits Advisory Committee (PBAC) did not report the disciplines of the guidelines developers, while Canada’s Canadian Agency for Drugs and Technologies in Health (CADTH) named the members of the guidelines working group, and the UK’s NICE briefly described the major roles of their guideline development team (39;41;45). A review of guideline organizations in Australia, Canada, Europe, New Zealand, and the U.S. found that guideline development groups (GDG) are composed of 10–20 members, and the number of disciplines is often three to five per group (46). In addition, guideline development was endorsed by the support of methodological experts (epidemiologists, information specialists, and others) and patient representatives (46). Adopting a multisectoral approach and engaging a multidisciplinary team is crucial for ensuring well-structured and high-quality guidelines (8;42). The GDG should include stakeholders such as consumers, health professionals who work within the relevant area, managers or policymakers, and individuals with the necessary technical skills, including information retrieval, systematic reviewing, group facilitation, project management, writing, editing, and expertise in health economics (46). The poor reporting on the development team prevented us from drawing conclusions about participants, and from identifying potential competing interests of development team members. This highlights the importance of good reporting and identification of the roles of members of the working group involved in the process, as well as clear conflict of interest statements.

A transparent and systematic process is fundamental for EEGs developers to accomplish their objective. We identified four main steps followed by EEGs developers in LMICs: conducting a review on national guidelines, organizing workshops, and getting support from either international experts or from organizations. Nevertheless, the steps followed were heterogeneous across guidelines. Only one country, Cuba, followed the four steps. According to the WHO, the optimal process for supporting the development of HSG is through finding evidence and translating evidence into guidance (8). That is, finding evidence using systematic reviews and translating evidence using a deliberative process, which may include informal or formal consensus involving several stakeholders with varied backgrounds, on development methods (8). Only six of the fourteen LMICs (Colombia, Cuba, Egypt, Malaysia, the Philippines, and Thailand) conducted reviews to generate evidence in developing their guidelines, and only five followed up this step by the deliberative process (Cuba, Egypt, Malaysia, the Philippines, and Thailand) (22;25;29;34;35;37). Other countries followed incomplete steps, that is, either a workshop, or international support, or a review (Bhutan, Brazil, China, Colombia, India, Indonesia, and Mexico) (18–21;31;33;34). Furthermore, while the AGREE enterprise provides a structured, systematic, and standardized methodological framework for developing and reporting HSG (11), we could not identify any country, except Cuba, following it. This is also the case in developed countries. For instance, Australia did not describe their guidelines development process, nor did the UK, which developed a manual for NICE guidelines, with the exception of some guidance including the appraisal of technology (47). In contrast, Canada, in their guidelines for the EE of pharmaceuticals, described the process for setting their guidelines (a workshop with Canadian stakeholders and international experts); this process was explicitly detailed by Towse in 1997 and Torrance in 1996 (48–50). Moreover, in 2018, Canada briefly described this process in the fourth edition of the guidelines (39). Standardizing the guideline development process would facilitate the production and adaptation of guidelines to different contexts and reduce spending resources on guideline development; this is of utmost importance for LMICs (51).

Finally, quality assessment scores were highest (nine out of fourteen) for the “methods” domain (means above 4), while lower scores were observed for the “participants” domain (seven out of fourteen with means below 4). Specifically, we noted that one-third of the identified EEGs failed to report or to engage multisectoral or multidisciplinary developers, and approximately 14 percent did not follow or did not report any recommended step as part of the development process. These results are in line with those reported by the systematic review of EEGs in LMICs, whereby countries with the lowest quality scores for the two domains had the most methodological weaknesses and gaps observed (7). Unfortunately, to our knowledge, the quality of the development process of EEGs in developed countries has not been previously assessed. Hence, a comparison with our results cannot be performed.

To our knowledge, this systematic review was the first to explore the development process of EEGs. Providing such information is crucial for guidelines developers, especially in the context of LMICs, where EEGs are increasingly being developed, and international organizations are supporting these countries in the development of guidelines. One major strength is that we followed both a predefined protocol and the recommended methods for reporting search strategies and systematic reviews (the PRISMA 2020 Statement). The search was broad and sensitive, without restrictions for language or publication date. Full-text screening and data extraction were done following best practices (in duplicate and using predefined forms). Two reviewers assessed the quality independently, following an in-depth overview of the AGREE-HS user manual and related documents.

One potential limitation to this work is that it could not include the EEGs published by Iran. A citation of these guidelines was identified on the ISPOR Web site; however, the full guidelines were inaccessible, even after contacting the ISPOR Iranian Chapter. Moreover, as for all systematic reviews, it is always possible that our search could have missed some guidelines. In addition, our quality assessment concerned what was presented in the guidelines and their related documents, and the development may be slightly different from what was reported. Finally, one limitation pertains to the absence of a threshold for differentiating between the high, moderate, and low quality of included studies. This limitation is inherent in the AGREE-HS tool, given the lack of an empirical basis for defining thresholds (11).

Conclusion

The current work fills a gap in the research pertaining to the development of guidelines in LMICs (7;46;51). We highlight the scarcity of published information related to this process and the suboptimal quality of included studies. This paper pinpoints the importance of including a multisectoral capacity and the relevance of a transparent, explicit, and systematic approach in developing and reporting on the development process of EEGs in LMICs. This work provides relevant material that assists international organizations—for example, WHO, GIN, AGREE enterprise—and supports guidelines developers in LMICs to identify the best practice for developing EEGs that fit their national context, and recommends a transparent approach in designing and reporting methods for the development of guidelines. Future studies should also investigate the barriers as well as facilitators that developers of guidelines faced while developing and setting their guidelines.