Mapping horizon scanning systems for medical devices: similarities, differences, and lessons learned

Abstract

Objectives

This article presents the mapping of horizons scanning systems (HSS) for medical devices, conducted by the Medical Devices Working Group of the International Horizon Scanning Initiative (IHSI MDWG). It provides an overview of the identified HSS, highlights similarities and differences between the systems, and lessons learned.

Methods

Potentially relevant HSS were identified through literature searches, scan of an overview of EuroScan members, and input from the IHSI MDWG members. Structured information was collected from organizations that confirmed having an HSS for medical devices.

Results

Sixteen initiatives could be identified, of which 11 are currently ongoing. The purposes of the HSS range from raising awareness of trends and new developments to managing informed decisions on innovative health services in hospitals. The time-horizon is most often 3 years up to a few months before market entry. Three models of identification of new technologies crystallized: a reactive (stakeholders outside HSS inform), a pro-active (actively searching multifold sources), and a hybrid model. Prioritization is often conducted by separate committees via scoring or debate. The outputs focus either on in-depth information of single technologies or on a class of technologies or on technologies in specific disease areas.

Conclusions

The identified HSS share the common experience that horizon scanning (HS) for medical devices is a resource-intensive exercise that requires a dedicated and skilled team. Insights into the identified HSS and their experiences will be used in the continued work of the IHSI MDWG on its proposal for an IHSI HSS for medical devices.

Background

The International Horizon Scanning Initiative (IHSI), launched in 2019, is a non-profit organization with members from nine different countries. Its mission is to establish a joint horizon scanning system (HSS) for pharmaceuticals. Such joint effort will help to avoid duplication of work and allow countries to prepare their health-care systems for any potentially disruptive, high-impact innovations, as well as aid procurement planning and price negotiations (1).

In 2021, during the development of the horizon scanning (HS) database for pharmaceuticals, IHSI set up a working group to explore the possibility of expanding its system to medical devices. Reasons for setting up the group included, among others, the fact that HS for medical devices has proven to be resource intensive and challenging to do on a country level. A large number of medical devices are under development, and many of the companies that develop devices are small or medium sized (2). It is difficult to have an overview of what is under development. The time of the initiation of a regulatory approval process for a device is unpredictable, and devices identified in an early development stage may never enter the marketing authorization process. International collaboration on HS within this field was considered necessary and desirable.

The IHSI Medical Devices Working Group (IHSI MDWG) comprises representatives from 16 organizations in 11 countries, including observers from four non-IHSI member countries. As a first step (Phase 1), the group decided to map HSS on medical devices, to explore how other organizations have set up their systems and learn from their experiences. National needs for HS for medical devices in the different member countries were also explored. Based on these findings, the group outlined the requirements and wishes for a future HSS and how this could be achieved (Phase 2). The final goal of the group is to prepare a concrete proposal for the development of an IHSI HSS for medical devices (Phase 3).

This article presents the first step of the IHSI MDWG’s work, the mapping exercise to identify HSS for medical devices. This step was included as the working group was not aware of any existing overviews that specifically focused on and described HSS for medical devices. The aim of the exercise was not to achieve a comprehensive list of HSS but to get an indication of the possible number of HSS and gain an understanding of the different types of HSS. The article provides an overview of the identified HSS, points out similarities and differences between the systems, and highlights lessons learned.

Methods

In November 2021, we conducted literature searches in PubMed and Google to identify HSS for medical devices. Searches were carried out using search terms characterizing HS and medical devices (see Supplementary file 1 for details). All publications that seemed to deal with HS for medical devices, based on the title and/or abstract fields of the record, were reviewed in full text. In addition, we were aware of an article from 2015 describing the EuroScan International Network member agency systems (3) that we reviewed to see whether further potentially relevant HSS could be identified. The compiled list of HSS was subsequently complemented by information from the IHSI MDWG members. All initiatives and organizations that seemed to be involved in HS for medical devices were included.

The identified, potentially relevant HSS were contacted via email or through an enquiry form on their website. Reminders were sent to those organizations that did not reply within 2–3 weeks. Provided that the contacted organization itself defined its HS activities as an HSS and confirmed medical devices being in the scope (or part of the scope) of its HSS, the initiative was included in our mapping exercise. Any planned, ongoing, or closed HSS for medical devices was eligible for inclusion. Organizations that were contacted but did not reply (n = 3) were excluded as it could not be confirmed that they had an HSS.

Structured information about the HSS was collected using a template, which was developed by the authors, in collaboration with the other IHSI MDWG members (see Supplementary file 2 for details). The template included open-ended questions about the purpose and scope of the HSS, time horizon, stakeholders to be informed, methods for identification, selection and prioritization, outputs, methods for monitoring identified technologies and updating reports, structure, and financing of the HSS, and lessons learned. To obtain as accurate and extensive information as possible, all organizations were asked to fill in this template themselves.

For the conduct of this article, we have harmonized the terms used in the completed templates, to allow a better data synthesis and to enable the presentation of the results in tables and graphics. However, to avoid any mistakes in doing so, before the publication of this article, the included HSS were provided the opportunity to confirm that the information included about their HSS was accurately represented.

Results

Sixteen HS initiatives for medical devices were identified through the mapping exercise (Table 1), of which 11 HSS are currently ongoing and active. Five programs are located in Europe (National Institute for Health and Care Research Innovation Observatory (NIHR IO)/ United Kingdom (UK), Norwegian Institute of Public Health (NIPH)/ Norway, the Spanish Network of Agencies for Health Technology Assessment (RedETS)/ Spain, the Dental and Pharmaceutical Benefits Agency (TLV) and the Swedish Association of Local Authorities and Regions (SALAR)/ Sweden, Zorginstituut Nederland (ZIN)/ the Netherlands), three in the Americas (Canadian Agency for Drugs and Technologies in Health (CADTH)/ Canada, National Commission for the Incorporation of Technologies (CONITEC)/ Brazil, Patient-Centered Outcomes Research Institute (PCORI)/ USA) and three in Asia (Agency for Care Effectiveness (ACE)/ Singapore, Israeli Center for Emerging Technologies (ICET)/ Israel, Malaysian Health Technology Assessment Section (MaHTAS)/ Malaysia). Another three HSS were identified that discontinued their services in recent years (Agency for Healthcare Research and Quality (AHRQ) HSS/ USA, Australian Safety and Efficacy Register of New Interventional Procedures-Surgical (ASERNIP-S)/ Australia, Managed Uptake of Medical Methods (MUMM)/ Finland) and two initiatives are currently inactive (Agenzia Nazionale per i Servizi Sanitari Regionali (AGENAS)/ Italy, National Institute for Health Research Community Healthcare MedTech and In-vitro Diagnostics Co-operative (NIHR MIC)/ UK), but not closed. No HSS in the planning stages were identified. At the time of conducting this mapping review, five of the identified ongoing HSS were also represented in the IHSI MDWG.

Table 1. Overview of the identified HSS for medical devices

Abbreviations: HS, horizon scanning; HSS, horizon scanning system; UK, United Kingdom.

Many of the identified HS initiatives are part of the mandate and/or a component of an organization or a national health care system. Only a few initiatives comprise ad-hoc HS activities or are project-based. However, in this article, for consistency, we use the term “HSS” for all the identified HS initiatives. As the number of both inactive and closed HSS is low and there were no specific deviating characteristics between them and the ongoing HSS, we report the results jointly for all identified HSS, unless otherwise specified in the text.

Purpose and stakeholders to be informed

The purposes of the HSS are manifold (Table 2). Some HSS have broad and general intentions such as to better prepare the healthcare system for new and emerging technologies, to aid in planning for resource allocation; or to support the uptake of innovative and effective technologies while safeguarding patients from potentially unsafe technologies before their widespread adoption. In contrast, depending on the stakeholders to be informed, the purposes can also be very specific and concrete, such as to identify medical devices and procedures that may need to be assessed within the national HTA-program (AGENAS, NIPH, and ZIN) or to manage informed decisions on innovative health services in the hospital setting (ASERNIP-S, ICET, and MUMM). In common is the purpose to inform strategic, financial, or operational planning (be it on the institutional or on the health system level) and to identify true – eventually even disruptive – innovations with the highest potential for clinical, financial, or health system impact.

Table 2. Purpose and stakeholders to be informed

Abbreviations: ACE, Agency for Care Effectiveness (Singapore); AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali (Italy); AHRQ, Agency for Healthcare Research and Quality (USA); ASERNIP-S, Australian Safety and Efficacy Register of New Interventional Procedures-Surgical (Australia); CADTH, Canadian Agency for Drugs and Technologies in Health (Canada); CONITEC, National Commission for the Incorporation of Technologies (Brazil); ICET, Israeli Center for Emerging Technologies (Israel); MaHTAS, Malaysian Health Technology Assessment Section (Malaysia); MUMM, Managed Uptake of Medical Methods (Finland); NIHR MIC, National Institute for Health Research Community Healthcare MedTech and In-vitro Diagnostics Co-operative (UK); NIHR IO, National Institute for Health and Care Research Innovation Observatory (UK); NIPH, Norwegian Institute of Public Health (Norway); PCORI, Patient-Centered Outcomes Research Institute (USA); RedETS, Spanish HTA Agencies Network (Spain); TLV, The Dental and Pharmaceutical Benefits Agency (Sweden); SALAR, The Swedish Association of Local Authorities and Regions (Sweden); ZIN, Zorginstituut Nederland (Netherlands); HS, horizon scanning; HT, health technology; HTA, health technology assessment; HCS, health care system; PM, policy-maker; DM, decision-maker; HCP, health care professional.

The primary stakeholders that are informed by the active HSS can be roughly categorized into two groups: hospital managers and regional or national health insurance procurement commissions; or ministerial health policy decision-makers and regulators. Some of the identified HSS serve both stakeholder groups. Other stakeholder groups, such as research funders and investigators to inform research priorities (MaHTAS and PCORI) are targeted less often. A few HSS consider citizens, patients, caregivers, and the general public as part of their stakeholder group (CADTH, CONITEC, NIHR IO, PCORI, and ZIN) and even less also the medical technology industry (NIHR IO and NIHR MIC).

Technology scope and time horizon

Most of the identified HSS have a broader scope, including both pharmaceuticals and medical devices (also often called non-pharmaceutical interventions and procedures). Few HSS have programs that specifically focus on non-pharmaceutical technologies (inactive: AGENAS, ASERNIP-S, and MUMM; active: ACE, CADTH, NIPH, RedETS, TLV&SALAR, and ZIN). They scan for new (and new uses of existing) medical devices, screening and diagnostic tests, procedures, services, and programs for care delivery, in recent years complemented by digital health technologies and medical informatics such as artificial intelligence and machine learning technologies. Some HSS have pre-defined priority areas (ACE, AHRQ, CADTH, NIHR IO, and PCORI), which in some cases can be only one specific indication (ZIN: diabetes) or one specific application (NIHR MIC: point-of-care diagnostics). Others have defined their priority areas by the characteristics of the respective technologies (MaHTAS: local innovations, expensive technologies).

The time horizon is most often 3 years up to a few (e.g., six) months before market entry and commercialization, depending on the stakeholders to be informed (Figure 1). In the majority of HSS, identification starts when the technology is at the late stage of development (in the regulatory pipeline) or entered the regulatory assessment process. Some HSS are also considering technologies in the post-marketing phase (MUMM, NIPH, and RedETS). In rare cases, also the established, late-stage technologies are within the scope of the HSS for disinvestment decisions on low-value care (NIPH).

Figure 1. Stage(s) of technology development the HSS focus on. Abbreviations: ACE, Agency for Care Effectiveness (Singapore); AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali (Italy); AHRQ, Agency for Healthcare Research and Quality (USA); ASERNIP-S, Australian Safety and Efficacy Register of New Interventional Procedures-Surgical (Australia); CADTH, Canadian Agency for Drugs and Technologies in Health (Canada); CONITEC, National Commission for the Incorporation of Technologies (Brazil); ICET, Israeli Center for Emerging Technologies (Israel); MaHTAS, Malaysian Health Technology Assessment Section (Malaysia); MUMM, Managed Uptake of Medical Methods (Finland); NIHR IO, National Institute for Health and Care Research Innovation Observatory (UK); NIHR MIC, National Institute for Health Research Community Healthcare MedTech and In-vitro Diagnostics Co-operative (UK); NIPH, Norwegian Institute of Public Health (Norway); PCORI, Patient-Centered Outcomes Research Institute (USA); RedETS, Spanish HTA Agencies Network (Spain); TLV, The Dental and Pharmaceutical Benefits Agency (Sweden); SALAR, The Swedish Association of Local Authorities and Regions (Sweden); ZIN, Zorginstituut Nederland (Netherlands); HSS, horizon scanning system; R&D, research and development; PMA, Premarket Approval; HDE, Humanitarian Device Exemptions; FDA, US Food and Drug Administration; CE, Conformité Européenne; HTA, health technology assessment; RWE, real-world evidence.

Methods of identification, selection, prioritization, monitoring and updating

Concerning methodologies of identification, three models in the HS of medical devices crystallized, based on the way new medical devices are identified: a reactive (stakeholders outside HSS inform HS), a pro-active (searching multifold sources) and a hybrid model (combining the two approaches) (Figure 2).

1. 1. In the reactive model, new medical devices are identified through signals from hospital districts, clinical experts, learned societies, research groups, consumers, or even technology developers (AGENAS, ICET, MUMM, NIHR MIC, and TLV/SALAR). Looking at the identified systems that use a reactive identification model, the scope and timing of these systems seems to be limited to devices already approved or shortly before placing them on the market, and limited to the initiatives of clinical specialties, and are therefore less systematic.

2. 2. In the pro-active HS initiatives, multiple sources are actively searched to identify candidate devices. In the identified systems, the aim of the searches is to identify devices before they are approved or marketed. Targeted searches are carried out in scientific journals, conference abstracts, clinical trial registries and study protocols, corporate websites, press releases, email news services, funding databases as well as technology transfer offices and patent information databases. Additional information from regulators (e.g., FDA medical devices database) is sought and the registration status in other countries is closely monitored. Some larger HSS apply semi-automated and automated data retrieval methods in addition to the manual (targeted) searching (NIHR IO, PCORI). For identification pre-defined criteria (such as clinical need and/or burden of disease) are used.

3. 3. In a hybrid model, the pro-active identification is complemented by input from clinical experts, health-care providers, and industry (ACE, ASERNIP-S, CADTH, MaHTAS, RedETS, ZIN).

Figure 2. Models for identification of technologies. Abbreviations: ACE, Agency for Care Effectiveness (Singapore); AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali (Italy); AHRQ, Agency for Healthcare Research and Quality (USA); ASERNIP-S, Australian Safety and Efficacy Register of New Interventional Procedures-Surgical (Australia); CADTH, Canadian Agency for Drugs and Technologies in Health (Canada); CONITEC, National Commission for the Incorporation of Technologies (Brazil); ICET, Israeli Center for Emerging Technologies (Israel); MaHTAS, Malaysian Health Technology Assessment Section (Malaysia); MUMM, Managed Uptake of Medical Methods (Finland); NIHR IO, National Institute for Health and Care Research Innovation Observatory (UK); NIHR MIC, National Institute for Health Research Community Healthcare MedTech and In-vitro Diagnostics Co-operative (UK); NIPH, Norwegian Institute of Public Health (Norway); PCORI, Patient-Centered Outcomes Research Institute (USA); RedETS, Spanish HTA Agencies Network (Spain); TLV, The Dental and Pharmaceutical Benefits Agency (Sweden); SALAR, The Swedish Association of Local Authorities and Regions (Sweden); ZIN, Zorginstituut Nederland (Netherlands).

For selecting technologies, explicit criteria – such as burden of disease and volume of patients, expected clinical benefit (novel or even disruptive technologies), existing treatments or response to unmet need, cost of the technology, organizational impact (related for instance to training and infrastructure), societal or ethical issues, safety/adverse events, and available evidence - are applied by nearly all HSS. However, compared to selection process, prioritization is less often standardized. The prioritization is often conducted by separate committees or councils with clinical experts, for instance using a score (1–5), a scale (“high”, “moderate” or “low”) or other rating systems.

Most HSS monitor and update information on the identified technologies by keeping track of ongoing trials and the regulatory status until an innovation becomes available in clinical practice outside the research environment. Decisions on re-introducing the technologies into the prioritization process or to eliminate them from the watch list are made accordingly. In contrast to the permanent monitoring of technologies, formal processes and explicit methods in updating earlier reports are less common.

Outputs

The final outputs are manifold; nevertheless, they can be classified into four different types of products:

1. 1. On single technologies: A targeted and in-depth report, focusing on a single or a few clinical applications of a health technology. It serves to provide an early assessment on the potential impact of a technology in the application(s) of interest (ACE: Horizon Scanning Brief; CADTH: Health Technology Update; MaHTAS: TechScan, TechBrief, Horizon Scanning Report; NIPH: Horizon Scanning Report; RedETS: Short Technology Briefings).

2. 2. On a class of technologies: Overviews of either a class of technologies or new types of interventions that provide background knowledge to guide early planning and may describe the development pipeline and current initiatives of relevance (ACE: Horizon Scanning Overview; CADTH: Issues in Emerging Technologies).

3. 3. Different technologies for a specific disease area: a report on emerging or new medical devices, digital technologies in diagnostics for a pre-defined priority area (ZIN: pilot on diabetes; AHRQ/PCORI: High Impact Reports/High Potential Disruption Reports).

4. 4. Only a watch list: listing multiple technologies without in-depth information (CADTH: Health Technology Trends to Watch List; RedETS: list of new and emerging technologies).

The outputs are published regularly, ranging from monthly to bi-annual publication.

Structure and financing

HS activities are almost exclusively funded publicly (by ministries of health/MoH), are often part of a larger institution (such as MoH, HTA agency, research institute or medical center), and funded within the budget of the respective organizations. The largest institution (NIHR IO) comprises 60 staff members. ECRI, which performs HS as a contractor to PCORI, employs a team of about 15 full-time individuals to perform this work, but can seek input from many more experts. The majority of HSS are small, having two to six staff members, often working part-time on HS, but are also advised by many more experts from their institutions. Dedicated exclusive budgets are rare.

Experiences and lessons learned

All HSS share the common experience that HS is a time-consuming and complex process, and requires skilled personnel, preferably a multidisciplinary team. In addition, the evidence base tends to be sparse or weak for medical technologies that are in earlier stages of development. An evaluation of the CADTH HSS products in 2018 looked back at 20 years of HS for devices and clinical interventions. The evaluation showed that from the 200 medical device publications from 1998–2018, 90 (45 percent) were followed by at least one subsequent CADTH publication on the topic. Of these, just under 50 percent were subsequently reviewed in two or more later CADTH publications. Several technologies were reviewed multiple (>5) times over many years. The timespan was sometimes 10 years or more, indicating that some technologies diffuse over a long period. Many factors influence the impact of technologies identified in HS - many will not realize potential impact despite strong signals indicating they will. Reasons for no further work included assessments by other HTA agencies, failure of product commercialization, lack of effectiveness of a technology, and technologies that are still in the lag period between HS identification and decision-making/diffusion. ACE and ZIN share the experience that it is more fruitful to prioritize technologies that are relatively more mature, such as those that have gained regulatory approvals (e.g., FDA approved) and/or those that have completed the post-marketing studies. These technologies would provide sufficient evidence for a meaningful assessment of its safety and effectiveness, allowing a better-informed prediction of its potential impact on patients and the healthcare system. For giving guidance to experts on applying the rating scales for addressing and assessing the potential impact of interventions on finances, on organizations, on staffing, and on health disparities a certain level of evidence must already be in place.

Other experiences are that the scope should be clear and the scanning should be limited to that scope. Deviations from initial scope should be documented and justified. Aligning to this will add to the robustness of the HS output (MUMM, NIHR IO, RedETS). In addition, clear methods for recording and reporting are necessary to contribute to transparency and reproducibility (PCORI, NIHR IO). Stakeholder participation throughout is essential and should be incorporated into the process and project management (NIHR IO). HS reports must be fit-for-purpose and targeted to the audience in order to guarantee that the information reaches them (ASERNIP-S, CONITEC). Agreement on publication and dissemination policy of the final output from the start of the project will contribute to more outputs being published and greater visibility of results (NIHR IO). A user-accessible database/website provides searchable, near real-time access to content previously available only through static, quarterly reports (PCORI). Evaluation of the system and its outputs (formal or informal) will help improve methods and processes, ascertain quality, and assess the use and impact of the HS products (MaHTAS, NIPH, PCORI). Furthermore, NIHR IO recommends that policy impact assessment, even though very difficult to undertake, should be embedded in the HS process.

Discussion

Sixteen HSS for medical devices were identified through the mapping exercise, of which 11 are active, two currently inactive and three closed. Even though the identified HSS serve a wide range of purposes and inform a variety of stakeholders, similarities among several HSS could be observed with regards to the time-horizon of interest and use of explicit methods for selection and prioritization. The final outputs are manifold, ranging from high-level lists of technologies to targeted, in-depth reports. A few HSS have their scope solely on medical devices, most HSS have a broader scope, including pharmaceuticals. It must be noted however that some of the organizations that stated that their scope is on medical devices may also conduct HS or have HSS for drugs, but this was not explicitly asked in the template.

On the one hand, the fact that the identified HSS were allowed to answer the questions in the template themselves, assured that the received information was precise and accurate. On the other hand, definitions and terms used by the different HSS varied and having exclusively free-text questions in the template created challenges in synthesizing and analyzing the received information. Even though all HSS answered all questions in our template, the extensiveness of answers and the level of provided details varied considerably. The use of another type of template that combined both closed- and open-ended questions and/or included more detailed questions, would likely have facilitated the analysis, and enabled better comparison of the identified HSS. Our attempt to group the information and harmonize the used terminology may have resulted in omission of some information. However, the fact that six HSS did review our article draft when provided the opportunity to check the content for accuracy before publication, helped us assure the accuracy of information.

As the time horizon of interest for different HSS may vary considerably (ranging from systems focusing on devices in an early stage of development to those focusing on devices that have already entered the market), we decided not to use any explicit pre-determined definition of “HSS” when selecting systems for inclusion. Provided that the contacted organization itself defined its system as an HSS, the system was included in our mapping exercise. This helped ensure that no HSS was overlooked due to the inclusion criteria being too strict. A limitation of our mapping is related to the information retrieval methods, which were pragmatic, driven by time constraints. Therefore, it is possible that we have missed some HSS.

Since the scope, selection, and prioritization processes of the identified HSS are connected to the national health-care priorities, the utilization of the systems in an international context seems to be limited. There are, however, several learning points that the IHSI MDWG will include in its continued work on the proposal for an IHSI HSS for medical devices. These include sources used to identify devices, the importance of having a clear purpose, scope, and time horizon(s) when developing an HSS, and the variety of outputs that can be produced. Our mapping also shows that an HSS requires a multidisciplinary team, as well as stakeholder involvement throughout the process. Some of these learning points overlap with the main issues identified by the HTAi Global Policy Forum in 2018 in their discussions on how current HSS could be further optimized to better inform future healthcare (4), and therefore seem to be crucial. Furthermore, our mapping indicates that continuous evaluation of the HSS is needed to improve and fine-tune methods and processes, and to assess the use and usefulness of its outputs. It is also recommended to embed impact assessment in the HS process. Many factors influence the impact of technologies identified in HS, which makes the prediction of technologies’ potential impact on patients and healthcare systems imperfect and challenging.

Packer et al (3) who investigated potentials and challenges for increased collaboration among EuroScan members, highlighted the lack of staff and finance as being some of the barriers to collaboration and identification process being the most suitable collaboration phase. For medical devices, there is a lack of good information sources, and a wide range of sources need to be considered (5; 6). Oortwijn et al. (4) who summarized the discussions of the HTAi Global Policy Forum in 2018, pointed out that even though filtration and prioritization of health technologies may need to be done on the national level, collecting and sharing information on new and emerging technologies in a cross-national database could be beneficial and increase efficiency. This supports the idea of creating a joint international HSS that can be used as a source of information and a conduit for collaboration.

Conclusions

Despite of differences between the identified HSS, they all share the common experience that HS for medical devices is a time-consuming and resource-intensive exercise that requires a dedicated and skilled team. Clearly defined purpose, scope, time horizon(s), and dissemination strategy seem to be key. Furthermore, stakeholder participation throughout the process is essential, as evidence is scarce for medical devices before their approval and the timing of market entry is difficult to estimate. Insights into the identified HSS and their experiences will be used in the continued work of the IHSI MDWG on its proposal for an IHSI HSS for medical devices.