Lifecycle evaluation of medical devices: supporting or jeopardizing patient outcomes? A comparative analysis of evaluation models

Abstract

Objectives

Lack of evidence regarding safety and effectiveness at market entry is driving the need to consider adopting a lifecycle approach to evaluating medical devices, but it is unclear what lifecycle evaluation means. This research sought to explore the tacit meanings of “lifecycle” and “lifecycle evaluation” as embodied within evaluation models/frameworks used for medical devices.

Methods

Drawing on qualitative evidence synthesis methods and using an inductive approach, novel methods were developed to identify, appraise, analyze, and synthesize lifecycle evaluation models used for medical devices. Data was extracted (including purpose; audience; characterization; outputs; timing; and type of model) from key texts for coding, categorization, and comparison, exploring embodied meaning across four broad perspectives.

Results

Fifty-two models were included in the synthesis. They demonstrated significant heterogeneity of meaning, form, scope, timing, and purpose. The “lifecycle” may represent a single stage, a series of stages, a cycle of innovation, or a system. “Lifecycle evaluation” focuses on the overarching goal of the stakeholder group, and may use a single or repeated evaluation to inform decision-making regarding the adoption of health technologies (Healthcare), resource allocation (Policymaking), investment in new product development or marketing (Trade and Industry), or market regulation (Regulation). The adoption of a lifecycle approach by regulators has resulted in the deferral of evidence generation to the post-market phase.

Conclusions

Using a “lifecycle evaluation” approach to inform reimbursement decision-making must not be allowed to further jeopardize evidence generation and patient safety by accepting inadequate evidence of safety and effectiveness for reimbursement decisions.

Introduction

Patients and healthcare payers value medical devices that are safe and effective. However, there is very little direct clinical evidence regarding the safety and efficacy of most medical devices when they first enter the market or even at the point of reimbursement (1–7). This has prompted many stakeholders to consider applying a lifecycle approach to evaluation with the aim of improving the evidence base over time.

However, the “medical device lifecycle” is a poorly defined concept and there is no consensus regarding what it encompasses nor regarding what is meant by lifecycle evaluation (8–11). Instead, their meaning is assumed, yet implicitly embodied within the evaluation frameworks being used.

Without clear definitions, different people assume different meanings and use these concepts for different purposes. Indeed, lifecycle evaluation may mean generating better evidence over time, but equally, it may mean accepting limited or poor-quality evidence early in the lifecycle based on the premise that more evidence will become available later (12). Unfortunately, such differences may impact patient safety if they are not made explicit. For example, assuming that a device has been proven to be safe and effective before being marketed means patient outcomes may not be analyzed to the extent that they would be if it were known that this is frequently untrue, thereby, delaying the detection of harmful devices (5;13–15). Therefore, it is necessary to understand what different people mean and understand by these concepts and how they use them.

Objectives

Aim

To explore the different meanings of “medical device lifecycle” and “lifecycle evaluation” as embodied within the evaluation models used by different actors involved with medical devices, and the potential impact that differences may have on patient safety.

Methods

The unit of analysis for this literature review is the model rather than the articles. Similar to Michie, Stralen, and West, we use the term “model” to mean ““a hypothetical description of a complex entity or process.”” (16), which in this study denotes any approach, framework, model or theory utilized to evaluate medical devices across their lifespan (Supplementary Material 1). For the sake of brevity, throughout this article the terms “model” or “conceptual model” will be used to encompass the entire range of theoretical or conceptual approaches, models, and frameworks.

The included models were drawn from diverse disciplines and, consequently, demonstrated extensive heterogeneity, requiring novel methods for analysis and synthesis. These are briefly described here using the ENTREQ guidelines (17), with more detail provided where indicated in the Supplementary Materials.

Synthesis Methodology

Using an inductive qualitative approach, models were synthesized using analytic techniques drawn from grounded theory, qualitative content analysis, thematic analysis, and meta-ethnography (18–24).

Approach to Searching

Due to the unexpectedly large number of lifecycle evaluation approaches found in the literature, and drawing on methods described previously, a pragmatic approach to searching was adopted (25–27). Indeed, complementary and snowball techniques have previously been found to be more effective and efficient at identifying relevant material when the topic is complex and multidisciplinary (26;27). Seminal texts and models were identified through scoping searches conducted in biomedical and multidisciplinary bibliographic databases, the Internet, and specific Web sites using keywords and phrases covering the concepts “medical device,” “lifecycle,” and “model.” Snowball searching (i.e., searching references, references of references, and citations) of seminal texts was employed to identify additional models. Electronic Search strategy: (Supplementary Material 2).

Data Sources

Systematic scoping searches: PubMed, EMBASE, and the Cochrane Library. Pragmatic searches: the internet, Google Scholar, and JSTOR.

Inclusion Criteria

Any lifecycle model that may be applied to a medical device (however, that may be conceptualized, e.g., as a product, industry, innovation, therapy, technology, intervention, software or hardware, or as a specific type of device) (Supplementary Material 1) covering several aspects or stages of a device’s lifetime was eligible for inclusion (Supplementary Material 3). Models devised solely for evaluating medicines were excluded, because their lifecycle stages differ considerably from medical devices, posing different problems and being subject to different requirements.

Study Screening Methods

Articles were systematically screened in phases against the inclusion and exclusion criteria (Supplementary Material 4). Initial screening was conducted concurrently with searching. Potentially relevant texts appearing to take a lifecycle approach were identified and saved in a document archive.

Model Selection

Documents in the archive were systematically screened against the inclusion criteria for data extraction into an Excel database, where they were then screened against a selection algorithm to select one representative text per model, with the remainder serving as reference materials. Searching, screening, selection, and data extraction iterated with data analysis. Selection ceased when data saturation had been reached (i.e., no significantly different models or new themes were emerging). Potentially eligible models identified subsequently were saved for triangulation (Supplementary Materials 3 and 4).

Model and Study Characteristics

The characteristics of the included models and their key reference texts are summarized in Table 2 (Supplementary Materials 8 and 9).

Appraisal Rationale

Quality appraisal (QA), though its use in qualitative research is contested (28), serves to deepen researchers’ understanding of the data being examined, making the QA process valuable for increasing insight (29). Appraisal items: No QA criteria have been universally accepted for appraising conceptual models. We adopted those used by D’Amour et al. (30), assessing whether a model was underpinned by theory, empirical data, and/or an explicit literature review strategy, assigning a score of one for each criterion met. Appraisal process: Models were appraised rather than articles, therefore, we drew on additional texts for the appraisal process since criteria may be met in different sources. Appraisal results are summarized in Table 2 (Supplementary Material 7). No model was excluded based on its score.

Data Extraction

Extracted data included study characteristics of reference texts; the model’s purpose, perspective, audience, characterization, scope, stages covered, factors included, focus of interest, representation (e.g., diagrams, graphs, mathematical equations, or textual descriptions), level of application, timepoints for analysis, process definition; and the quality criteria. Data were extracted primarily from the reference texts, but supplemented from others. Software: Word and Excel 2016 for Mac (Version 16.70).

Number of Reviewers

This study was conducted as part of a PhD, therefore, one reviewer (KH) conducted all aspects of searching, screening, selection, data extraction, quality appraisal, analysis, and synthesis, with supervision, oversight, discussion, reflection, and review by two supervisors (ST and JS).

Coding

The unit of analysis was the model, consequently, coding focused on the data extracted from the documents. Each model was categorized into one of four broad perspectives (Supplementary Material 5). As analysis progressed extracted data was coded into increasingly broad and more abstract categories, enabling comparison within and across categories. Summaries of each model were prepared, and visual tools created (Analysis cards) to enable a visual comparison of the models.

Derivation of Themes

Using the constant comparative method, together with reflective writing, which surfaced higher-order patterns in the data, themes relevant to patient safety were derived. Quotations are used to support the findings.

Positionality and Reflexivity

The researchers come from a medical, Health Technology Assessment (HTA), and/or policy background, mostly, from a positivist/postpositivist paradigm, but take a pragmatic worldview. We are interested in patient outcomes and healthcare efficiency. Consequently, personal professional perspectives and pre-conceptions were bracketed during the interpretive process and an inductive approach was taken to ground the analysis in the data (31, p. 27, 193–195).

Trustworthiness

To ensure researcher consistency, a system of double-entry, comparison, and validation of analytic data was developed. The document archive and Excel database provide a research audit trail, with Supplementary Materials ensuring transparency.

Synthesis Output

Several syntheses were created, including a comparative analysis of the models, the development of a model typology, and a thematic analysis. This article reports on the comparative analysis.

Reporting

The findings are presented in three sections, with a brief overview first, followed by a tabular and narrative description of the meanings attributed to “medical device lifecycle” and “lifecycle evaluation.” The discussion explores implications, whilst the conclusion sets out the key issues and potential solutions.

Findings

Fifty-two models were included in the synthesis, with 51 key texts, drawn from multiple different perspectives, and organized into four broad categories for comparison. Nine models were derived from Healthcare (HC), 16 from Policymaking (Policy), eight from Regulation (Reg), and 19 from Trade and Industry (T&I). Table 1 lists the models according to their perspectives, providing abbreviated names and the reference numbers for key texts, with their references provided in Supplementary Material 6.

Table 1. List of models included in the synthesis

Similar abbreviations indicate a similar name or focus: DA, decision algorithm; DOI, diffusion of innovations; EAES, European Association for Endoscopic Surgery; EIM, endoscopic innovation management; EUnetHTA, European network for Health Technology Assessment; FDA, Food and Drug Administration; HCanada, Health Canada; IC, innovation cycle; IDEAL, idea, development, exploration, assessment, and long-term study; 4S-IEE, 4-stage iterative economic evaluation; IRM, integrated risk management; IRP, innovation routinization process; MD, medical device; MDDP, medical device development process; MDRegLC, medical device regulatory lifecycle; NASA, National Aeronautics and Space Administration; nHTLC4I, new health technology life cycle for innovation; NPD, new product development; OIM, orthopedic innovation management; PLC, product life cycle; RE-AIM, reach, efficacy, adoption, implementation, maintenance; SaMDDP, software as a medical device development process; SG, stage-gate; TALC, technology adoption lifecycle; TGA, Therapeutic Goods Administration; TLC, Technology lifecycle; USVP, University Spin-off Venturing Process; VA, Veteran Administration.

Symbols key: ™, Trade and Industry; ^Π , Policymaking; ^Ψ , Healthcare; ^® , Regulation.

The models were characterized across seven dimensions related to the types of device, model, lifecycle, evaluation, and data; date of first appearance; and QA scores, which are summarized in Table 2, whilst Table 3 summarizes and compares their key features, including purpose, primary intended audience, lifecycle characterization, outputs, timing, and scope (Supplementary Material 10).

Table 2. Summary of model and study (reference text) characteristics

COI, conflict of interest; HC, healthcare; NPD, new product development; Policy, policymaking; QA, quality appraisal; Reg, regulation; T&I, trade and industry; WHO, World Health Organization.

Table 3. Summary of findings table

HC, healthcare; NPD, new product development; Policy, policymaking; Reg, regulation; T&I, trade and Industry.

What is Meant by “Medical Device Lifecycle”?

In this section, we describe the different forms, scopes, and types of process used to represent the lifecycle of a medical device. The most common lifecycle form used is a linear series of stages (though iteration may occur) moving from the idea, through its development, growth, maturity, decline, to end-of-life (or a subset of these). A second form is a cycle of innovation, with each original invention being incrementally refined, which means that, for these models, the lifecycle encompasses several generations of a device. A third form is a pattern describing how a single lifecycle phase changes over time (e.g., sales or adoption), therefore, these models focus on how the parameter is affected rather than the effects the product has on patients/end-users. In a sense, these are not actually device lifecycles, but they are, nevertheless, often what is meant by a device’s lifecycle. Another set of models describe the lifecycle as movement through various levels in a system, and are, therefore, multi-level evaluations. For example, where a novel idea is developed into a technology, which becomes a component in a device, that is itself incorporated into a system (e.g., software developed for a switch, used in a mobile arm, which becomes a component in a robotic assistant).

To compare models’ scope, we utilized the Health Care Technology Lifecycle (HCTLC) model (the green part in Figure 1) from the World Health Organization (WHO) (SR-20). It illustrates the phases of the medical device lifecycle in terms of the activities/processes required across its lifespan. This includes 17 specific stages of activity grouped into three phases – Provision, Acquisition, and Utilization. Regulation may be considered as a separate process, but in this model it is considered to be an integral part, where regulatory compliance is necessary for the device to transition to the next stage. The focus of interest during the provision stage is primarily on the new product development process, whilst the acquisition phase focuses primarily on adoption or sales (i.e., diffusion). During utilization, the main focus of the models is on evaluating outcomes. However, whilst certain models cover all of these phases and stages, generally, most cover only some. In fact, Table 3 shows that the scope of the lifecycle models varied considerably across perspectives. For example, the T&I models cover only the provision stages, and approximately half of the Policy models do not include the utilization stage, which means that these evaluation models do not examine (to any great extent) what happens after a device enters the healthcare system.

Figure 1. Model archetypes associated with the Healthcare Technology Life Cycle (adapted from WHO, SR-20).

To compare the types of process embodied in the models, we utilized Van de Ven’s process definitions – “(i) a logic used to explain a causal relationship in a variance theory, (ii) a category of concepts that refer to actions of individuals or organizations, and (iii) a sequence of events that describe how things change over time” (32, p. 169). We found that frequently the models encompassed both the second and third of these definitions. Nevertheless, approximately half of the models predominantly used a process definition that referred to actors’ actions, a quarter referred to a sequence of events, whilst five used both. The majority of these were stage models, that is they described the lifecycle as distinct stages rather than as a continuous distribution of a particular variable. This contrasts with the seven applying the first definition, exploring causal relationships, all of which characterized the lifecycle quantitatively, and none of which focused on evaluating patient/end-user outcomes.

What is Meant by “Lifecycle Evaluation”?

As shown in Tables 2 and 3, lifecycle evaluation differs in terms of what gets evaluated, when, how, and why. Most commonly, what is evaluated are the factors that are used to characterize the lifecycle, indicators of lifecycle progression, factors influencing the lifecycle trajectory or that are affected by it, and/or the outcomes from deploying a technology. However, relatively few of the models, and none of those from the T&I perspective, focus primarily on evaluating patient/end-user outcomes.

An evaluation may be a once-off event encompassing the entire lifecycle in a single evaluation (e.g., using empirical data to describe the sales/adoption pattern over time or by evaluating different lifecycle stages at a single point in time, either retrospectively using existing data or prospectively using modeling techniques) or it may be a lifecycle evaluation by virtue of the fact that the evaluation is repeated at several time points across the life of a technology. It is this latter type that is generally considered to be a “lifecycle approach” by the HTA community (33). However, 22 of the models included in this synthesis did not take this approach.

How the lifecycle is evaluated depends on the particular purpose of a given model. It may be performed using quantitative, qualitative, or mixed types of data, to conduct a descriptive, explanatory, predictive, or prescriptive evaluation. Table 3 shows that the most frequent purpose of evaluation is to define particular actions or activities that need to be undertaken at the different lifecycle stages, only 15 of which describe approaches to evaluating end-user or patient outcomes.

The main reason for adopting a lifecycle evaluation approach is to address a need for information. The models are used for gathering, generating, or conveying information/evidence required for making the best decisions to achieve the overarching goal. The purposes listed in Table 3 are simply intermediate purposes, whilst the overarching goal (and the corresponding decision) varies across perspectives. The primary goal for HC is to provide safe, effective, and efficient healthcare, therefore, its decisions relate to the adoption of clinically effective technologies. Whilst for T&I the goal is to achieve commercial success and generate profit. To do this it must decide on the appropriate resources to invest and the timing of investment, usually in new product development and/or marketing. Policymakers often have dual policy goals – to promote trade and industry and to ensure the provision of high-quality and cost-effective healthcare. Thus, their decisions depend on the policy goal, but generally relate to market regulation or resource allocation. Sometimes, however, these goals are conflicting, because easing access to healthcare markets for trade and industry usually means reducing the quality and quantity of evidence required for market access, which reduces the evidence available for evaluating health technologies for their cost-effectiveness. “Tension can arise between these objectives. Expediency must be balanced against adequate rigor, affordability with access. Aligning regulatory objectives with broader economic and industrial policy (e.g., to promote innovation, employment, growth, export and trade) may result in tension with goals of managing costs. In addition, each objective will be prioritised differently by stakeholder groups, adding a political dimension to the process.” (SR-47, p. 118).

The explicit goal for those from the regulatory perspective is not, as is frequently assumed, to ensure that medical devices entering the market are safe and effective (SR-35), but rather that the regulatory requirements of their particular jurisdiction are met. “Pre-market control is performed on the device to ensure that the product to be placed on-market complies with regulatory requirements.” (SR-20, p. 20, bold in original) Therefore, the safety and effectiveness of medical devices at market entry is dependent on the specific legal requirements in a given jurisdiction. Regulatory models differ, but for the most part regulators endorse the guidelines of the Global Harmonization Task Force and follow the approaches illustrated in Figure 2 suggested by the WHO (SR-20).

Figure 2. Regulation across the medical device lifecycle (adapted from WHO, SR-20).

Medical device regulators are responsible for deciding whether a health technology may be legally marketed in their jurisdiction. In some jurisdictions, this sometimes involves a market authorization process, where the products can only be placed on the market if authorized to do so by the regulators (e.g., in the US and Canada) usually on the basis of evidence of safety and effectiveness gleaned from the literature or clinical investigations. “… the premarket notification, or 510(k), process is a classification process, whereas PMA [Pre-market Approval] is a determination of safety and effectiveness that leads to approval. … For the 510(k) process, devices are cleared for marketing, not approved, and devices may not be marketed as ‘approved by FDA’” (SR-23, p.12, emphasis in original). In other jurisdictions (e.g., Australia, European Union (EU), Switzerland) market authorization is not required, instead manufacturers of high-risk devices must apply for an assessment of their conformity with the legal requirements by an external body, a conformity assessment body (CAS) (SR-51). If the CAS deems them to be compliant it authorizes them to affix a marking (e.g., the CE marking in the EU) on their device, a legal requirement for marketing the device that indicates it is legally compliant. This is not the same as market authorization, “Unlike medicinal products, medical devices do not undergo an official authorisation procedure. For these devices, Switzerland follows what is specified for the European Union (EU) system of compliance assessment and certification, based on bilateral agreements. Compliance with internationally valid norms is evaluated by private entities” (SR-51).

All regulatory models include a requirement for an evaluation of clinical data for all but the lowest-risk devices, but not necessarily on the same device, nor must the clinical data include a randomized controlled trial (RCT). “What consumer protections are required to bring new products to market? … FDA is required to provide the least burdensome path to market … This means that the trials should be parsimonious in size, focused in objectives, and not cumbersome in execution. If the questions can be answered in a post-marketing period then they should not be required before marketing” (9, slides 5,14). Consequently, this regulatory lifecycle approach means that evidence generation is frequently deferred until the post-market phase (34).

Discussion

The lack of evidence available at market entry is driving the push to adopt a lifecycle approach to evaluation for healthcare reimbursement decisions (SR-49). However, the differences noted in Tables 2 and 3 illustrate that, despite using the same words, people often mean very different things and have different reasons for using a lifecycle evaluation approach. Nevertheless, the concepts, as embodied in the model archetypes, have changed little over time. What has changed, as evidenced by the relatively recent emergence of the healthcare models, is the increasing pressure for earlier access to novel health technologies, resulting in increased pressure on healthcare providers and regulators.

However, the lack of evidence available at market entry/reimbursement is related to the regulators’ adoption of a lifecycle approach. “The concept of lifecycle regulation … also refers to an approach to regulation that factors the prospect of evidence generation postapproval into preapproval decision making” (34, p. 824). Therefore, adopting a lifecycle approach has meant that the regulators have accepted limited and poor-quality evidence, allowing devices to enter the market based on the expectation that more evidence will become available once the device is in use (34). However, this is frequently not the case (35–37).

Once devices are placed on the market regulators rely on post-market studies to address uncertainty regarding effectiveness and vigilance/surveillance activities to detect adverse outcomes. Yet, Rathi et al. have shown that many post-market studies are slow to be completed or not conducted at all (38). Whilst post-market surveillance is designed to detect only the most severe adverse outcomes, which are just a fraction of adverse outcomes that matter to patients (3;39).

Furthermore, clinicians and patients are not readily made aware of evidence regarding a device’s poor safety record, as demonstrated by Peters, Pellerin, and Janney, whose study showed that recalls of Class III devices in the US took more than 8 months to complete (40). Several authors have also demonstrated that although a device has been recalled it may still be used as a predicate (i.e., a substantially equivalent, legally marketed) device to support the market authorization of many subsequent devices through equivalence claims (38;41–44). Furthermore, because this evidence either never becomes available, or only becomes available years later, many people may be affected before it becomes clear that it is clinically inferior to alternative interventions (13–15;45).

Therefore, if HTA is to follow the example of the regulators’ lifecycle approach, it would mean accepting limited and inadequate evidence of safety and efficacy for reimbursement decisions with the promise that the required evidence would later become available. However, this assumes that the device is likely to be beneficial, unlikely to cause harm, and evidence will be made available (in a timely manner). Unfortunately, if manufacturers have already secured public funding for their devices, then there is very little incentive for them to generate additional evidence, which is costly and might show that their device is less safe or effective than alternatives, resulting in its market failure. However, without evidence the assumption that a device is safe and effective cannot be challenged or confirmed.

Industry may argue that high-quality evidence is difficult to develop due to inherent difficulties in conducting RCTs for medical devices (46), and that the costs are too great a burden for the small-to-medium enterprises (SMEs) that apparently predominate in the industry (47). However, why should patients bear the burden of having unproven devices used on them so that SMEs can be spared the cost of properly evaluating them?

Instead, governments could create healthcare infrastructures to support rigorous evidence generation in as efficient a manner as possible. There are schemes for “coverage with evidence development”, however, few devices are investigated in this way. Therefore, a more systematic approach is needed that enrolls all patients and all interventions into an ongoing and adaptive clinical trial comparing clinical outcomes for all treatments. This requires robust data collection and evaluation systems within healthcare, with secure individual patient and device identifiers, data linkage, and strong mechanisms of data protection. Such a system will require multi-stakeholder engagement and huge investment to develop, and is likely to be complex to operationalize. Nevertheless, its potential for improving clinical outcomes and patient safety is immense. Additionally, it would provide a means for industry to develop a robust evidence base to support applications for reimbursement and improve payers’ ability to fund the most cost-effective innovations. Therefore, future research should investigate how best to do this.

Limitations

This is not an aggregative systematic review, so models exist that have not been included. Nevertheless, by including a large sample from a broad variety of sources and perspectives this review provides a reasonable picture of the general form and content of lifecycle models being used. Due to the broad inclusion criteria many of the included models are not intended specifically for medical devices, however, this heterogeneity facilitated a rich exploration of each discipline’s perspectives and motivations. Additionally, the methods we used, particularly the QA criteria, are not validated, therefore, future methodological research should establish best practices for conducting reviews of multi-disciplinary conceptual models.

Conclusions

The medical device lifecycle and lifecycle evaluation have different meanings, and different purposes for different stakeholders. The lifecycle approach adopted by regulators has resulted in the deferral of evidence generation to the post-market period, resulting in a lack of evidence for reimbursement and clinical decisions. Therefore, research is needed to evaluate patient outcomes arising from early access to medical devices based on the promise of future evidence compared with evidence-based interventions. Furthermore, the HTA community could become involved in setting up rigorous systems within healthcare to systematically collect and analyze data on patient outcomes from the highest-risk devices (SR-44). Meanwhile, HTA bodies should not allow lifecycle evaluation to be used as an excuse for accepting inadequate evidence and should insist on appropriate evidence being made available for reimbursement decisions, as this may be the last defense some patients have against sub-optimal devices (1).