Can we use existing guidance to support the development of robust real-world evidence for health technology assessment/payer decision-making?

Abstract

Advances in the digitization of health systems and expedited regulatory approvals of innovative treatments have led to increased potential for the use of real-world data (RWD) to generate real-world evidence (RWE) to complement evidence from clinical trials. However, health technology assessment (HTA) bodies and payers have concerns about the ability to generate RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness. Consequently, there is a growing need for HTA bodies and payers to develop guidance for the industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing. We therefore sought to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of published guidance; and (iii) review findings with selected HTA/payer bodies. Four themes considered key to shaping the generation of robust RWE for HTA bodies and payers were identified as: (i) data (availability, governance, and quality); (ii) methodology (design and analytics); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. A range of guidance documents were found from trusted sources that could address these themes. These were discussed with HTA experts. This commentary summarizes the potential guidance solutions available to help resolve issues faced by HTA decision-makers in the adoption of RWD/RWE. It shows that there is alignment among stakeholders about the areas that need improvement in the development of RWE and that the key priority to move forward is better collaboration to make data usable for multiple purposes.

Introduction

Real-world data (RWD) has been used for decades by regulators for pharmacovigilance purposes and by HTA for contextualization of evidence to a specific health system setting, to extrapolate outcomes and input to economic modeling. With the advancement of digitization in health systems and expedited regulatory approvals of innovative treatments, there is greater potential for the use of RWD to generate real-world evidence (RWE) to complement evidence from clinical trials. However, many health technology assessment (HTA) bodies and payers have voiced concerns about the ability to develop RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness, and argue that randomized controlled trials (RCTs) should remain the key evidence base (1).

HTA is based on an evidence-based medicine paradigm, with a foundation of critical review of meta-analyses and RCTs. Experience with appraisal of RWD is often more limited, and different expertise is required for the generation of evidence based on RWD. For example, the generation of RWE may involve linking of data from a range of data sources and consider the impacts of creating retrospective definitions of basic aspects of a study, including patient eligibility, baseline characteristics, outcomes, and time windows. Whilst regulators are extending their interest in, and guidance for, the use of RWE to include the consideration of issues related to treatment effectiveness in a clinical practice setting, and appreciate the opportunity to decrease uncertainty in their decision-making, many national HTA bodies have not yet established clear guidance for industry on what RWE they would accept and how it will be appraised (2;3). The Registry Evaluation and Quality Standards Tool (REQueST) from European Network for Health Technology Assessment (EUnetHTA) supports the evaluation of clinical registries for use in HTA (4), but there is no other joint HTA guidance relating to RWD or RWE, nor are registries the sole source of RWD. As the availability of RWD from healthcare systems, patients, and other sources increases, so will the need for HTA bodies and payers to develop clear guidance for industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing decision-making for highly innovative technologies.

This commentary presents work undertaken to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of existing guidance published by individual HTA bodies, payers, regulators, multi-stakeholder collaborations, and academic groups; and (iii) share the initial opinion on the topic from selected HTA/payer bodies.

Known/published barriers to RWD/RWE uptake by HTA bodies and payers

To better understand the concerns surrounding the use of RWD/RWE raised by HTA bodies and payers, a scoping literature review of PubMed was conducted in April 2021 (see Supplementary Materials) to identify articles published in English, including interviews or group work involving several HTA/payer experts concerning the use of RWD/RWE. This initially focused on articles that included HTA bodies in Europe and then those outside Europe. Five key articles from multi-stakeholder groups involving a range of HTA bodies in Europe were identified (1;5–8) from the HTAi Policy Forum, IMI-GetReal, and RWE4Decisions. In addition, one article solicited German and other European stakeholder views on the value and challenges of RWE post-approval, and relevant RWD collection requirements (9).

Evidence from this scoping review suggested that the value of RWD, and the RWE it generates, is already accepted and used in support of HTA. RWE provides information about the incidence, prevalence, and natural history of the disease, compliance/adherence to treatment, quality of life, health system resource use, drug utilization, costs, and can support the development of transition probabilities for economic models. However, the evidence also highlighted concerns related to the use of RWE to demonstrate treatment effects. Key issues were extracted from each paper, as shown in Table 1. These included a range of topics about the quality of RWD, data infrastructure and access issues, transparency in curation and analysis, use of appropriate statistical methodology, transferability/generalizability of RWD/RWE, and the mistrust of conclusions made based on RWD. In addition, issues related to the lack of stakeholder collaboration were raised in terms of alignment of RWE requirements pre- and post-licensing (e.g., differences between HTA, payer, and regulators), and clarity about when RWE may be acceptable.

Table 1. Issues regarding the use of RWD/RWE reported by HTA/Payers in the EU

CEA, comparative effectiveness assessment; HTA, health technology assessment; REA, relative effectiveness assessment; RWD, real-world data; RWE, real-world evidence.

These issues were grouped into themes that were considered key to shaping the generation of robust RWE for HTA and payers: (i) data (availability, governance, and quality); (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. Papers involving views of HTA bodies and payers outside Europe (in Canada and USA) were then reviewed and issues extracted (data available on file) (10–18). There was a remarkable similarity in the issues raised, that confirmed themes identified in Europe which are presented as the pillars in Figure 1.

Figure 1. Known/published barriers to RWD/RWE uptake by HTA bodies and payers. EU, European Union; HTA, health technology assessment; RCT, randomized controlled trial; RWD, real-world data; RWE, real-world evidence.

Potential solutions to address HTA bodies/payer challenges with use of RWE

Recognizing that other fields, such as pharmacoepidemiology, have developed guidance about the generation of RWE that might be applicable to HTA, a further targeted literature review was undertaken to identify potential solutions to overcome the issues raised by HTA bodies and payers. This included a search of key RWD/RWE initiatives, HTA and regulatory websites, EUnetHTA, Google, and PubMed. In total, ninety-three publications from forty-one organizations or collaborations (regulators, academics, professional societies, expert collaboratives, and individual HTA bodies) were identified that presented guidance on the use of RWD or development of RWE in particular settings. These aligned well with the four pillars showing the commonality of issues across stakeholders.

Table 2 presents each of the four pillars and potential guidance that may be available for use or adaptation to help resolve issues faced in HTA decision-making. The key aspects of these publications are presented in the following sections.

Table 2. Initiatives providing solutions for the challenges and barriers to RWD/RWE adoption for HTA bodies and payers, stratified by the four identified pillars; (i) data (availability, governance, and quality), (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships

Note: Tools leveraged such as GRADE (Grading of Recommendations Assessment, Development and Evaluation) that address quality generally rank all non-randomized studies as ‘low quality’, regardless of the study quality.

AHRQ, body for healthcare research and quality; AMCP, academy of managed care pharmacy; CanREValue, Canadian real-world evidence for value of cancer drugs; CER, comparative effectiveness research; EHDEN, European Health Data Evidence Network; EMA, European Medicines Body; ENCePP, European Network of Centres for Pharmacoepidemiology and Pharmacovigilance; EU, European Union; EUnetHTA, European Network for Health Technology Assessment; FDA, food and drug administration; GRACE, Good ReseArch for Comparative Effectiveness; HMA, heads of medicines bodies; HTA, health technology assessment; i ~ HD, European Institute for Innovation through Health Data; IMI, innovative medicines initiative; IQWiQ, institute for quality and efficiency in health care; ISPE, international society for pharmaceutical engineering; ISPOR, professional society for health economics and outcomes research; NICE, National Institute for Health and Care Excellence; NPC, NATIONAL PHARMACEUTICAL COUNCIL; OHDSI, observational health data sciences and informatics; OMOP, observational medical outcomes partnership; OPERAND, observational patient evidence for regulatory approval and understanding disease; PE, pharmacoepidemiological; RCT DUPLICATE, randomized controlled trials duplicated using prospective longitudinal insurance claims: applying techniques of epidemiology; RECORD, REporting of studies conducted using observational routinely-collected data; REPEAT, reproducible evidence: practices to enhance and achieve transparency; RWE, real-world evidence; SPACE, structured preapproval and postapproval comparative study; STRATOS, STRengthening analytical thinking for observational studies; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology; WHO, World Health Organization.

Data (availability, governance, and quality)

The availability, governance, and quality of data were addressed in detail by many publications, particularly by collaborative groups. Initiatives led by HTA bodies and payers included EUnetHTA’s REQueEST tool for evaluating registries for HTA use (4), and the German national Institute for Quality and Efficiency in Health Care (IQWiG) guidance on the analysis of routine practice data for benefit assessment (19). ISPOR’s Task Force on Retrospective Databases checklist was created to assess issues unique to database studies, such as data reliability and validity (20). European collaboratives such as European Health Data Evidence Network (EHDEN) have identified data sources and developed approaches to support data quality and harmonization (e.g., via a common data model). Furthermore, regulators, research collaboratives and academics have published several checklists, guides, and reporting standards relating to various data aspects including relevance, reliability, fitness for use, quality, and privacy (see Table 2). All groups encourage pre-planning and transparency of approaches.

Methodology (design and analytic)

Methodological issues relating to design and analytics of real-world studies were considered in a number of the publications identified. Best practice guidance was published on the design and analysis of observational/non-randomized studies for comparative effectiveness research (CER) that is, GRACE, ISPE, STROBE, ISPOR, ENCePP, and PCORI (Table 2). Statistical methodologies for informing CER are widely reported (NICE 2015 (21), ISPOR 2009 (22), ISPOR 2012 (23), GRACE (24), AHRQ (25), and IMPACT HTA W6 (26)) and often include approaches to identifying and mitigating bias and confounding (EUnetHTA 2015 (27), ISPOR 2009 part I (22), AHRQ (25), ENCePP Methods (28), STRATOS (29), and NICE 2015 (21)). A comprehensive overview of existing guidance, frameworks, and checklists is provided by Jaksa et al. (30), as well as ISPOR, NICE, and EUnetHTA. The latter two checklists are HTA-specific and are summarized below.

As part of its robust statistical methodological guidance document, NICE described how it evaluates the quality of an RWD analysis, on treatment effect generally, and in the context of cost-effectiveness, using existing tools and checklists. These include ISPOR 2003; ISPOR-AMCP-NBC 2013 (31). The underlying assumptions of statistical methods are often overlooked by HTA reviewers and helpfully described by NICE DSU (2015), with a supportive algorithm to aid the appropriate method selection (21).

EUnetHTA’s guidance on the internal validity of non-randomized studies on interventions similarly includes a critical review of tools and checklist assessing risk of bias, recommending ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool) and RoBANS (Risk of Bias Assessment Tool for Non-randomized Studies) (27). In 2019, EUnetHTA developed the REQueST tool to support the evaluation of methodological information, essential registry standards, and additional requirements. Accompanying the tool is a ‘vision paper’ which explores the options for the long-term delivery, use and sustainability of REQueST beyond EUnetHTA Joint Action 3 (4). ISPOR and AHRQ also provide a checklist and guidance on registries, respectively. An extensive list of available checklists on design and analyses are available in the appendix of both documents (21;27).

Other key multi-stakeholder initiatives concerning methodology include IMPACT HTA (work package 2 and 6 specifically) (26) and IMI-GetReal. Please see Table 2 for further details.

Trust (transparency and reproducibility)

Several groups have developed publications focused on trust, transparency, and reproducibility. The multi-stakeholder RWE Transparency Initiative (ISPOR/ISPE/NPC/Duke-Margolis) reported practical recommendations for establishing a culture of transparency for the analysis and reporting of RWD/RWE studies, including the creation of an RWE registry that partnered with the Open Science Foundation to promote a more widespread culture of registering RWE study (32). For the reproducibility of RWE studies, a structured template for planning and reporting on RWE studies (STaRT-RWE) has been developed (33). The majority of organizations providing solutions related to trust in the generation and use of RWD/RWE were non-HTA body/payer specific and developed recommendations without HTA involvement.

Policy and partnerships

Identified policy and partnership-related barriers to the adoption of RWE by HTA bodies and payers included the lack of harmonization on policies, evidence requirements, and the lack of coordination at international level between HTA bodies and payers for RWD collection, acceptance, context of acceptance, and relevance. Collaborative initiatives including ISPOR/ISPE, ISPOR-AMCP-NPC; OHDSI, EHDEN, GetReal, CanREValue, and RWE4Decisions have all worked to develop solutions related to policy and partnerships. The Observational Health Data Science and Informatics (OHDSI) program was established in 2014 as an interdisciplinary partnership to bring out the value of health data through large-scale analytics, producing open-source solutions. OHDSI collaborates with the EHDEN (European Health Data and Evidence Network) Academy to support work related to data quality and provide education for all those working on RWD (34). RWE4Decisions, works at the policy level and is payer-led, but seeks to foster partnership among stakeholders to explore what RWD can be collected for innovative technologies that meet the needs of patients and healthcare systems, and to ensure efficient use of RWD/RWE to inform HTA body and payer decisions. A recent US Food and Drug Administration (FDA) and European Medicines Agency (EMA) collaboration developed a roadmap for international collaboration on RWE using COVID-19 as the model case (35;36) and FDA-Reagan-Udall-Friends of Cancer-Aetion developed the COVID-19 Evidence Accelerator, which involved the collaboration of leading experts in health data aggregation and analytics with the aim of sharing insights, comparing results, and answering key questions to inform the collective COVID-19 response (37).

Discussion

The four pillars

The findings of the literature reviews were presented to a panel of HTA experts at an RWE4Decisions webinar in October 2021 (38). The panelists agreed with the four pillar themes, noting that they addressed both policy-related issues and issues relating to processes for individual HTAs. The pillar relating to data (availability, governance, and quality) was considered as paramount by all panel members and methodology was also considered key. The improvement of transparency relating to RWE study conduct, including registration of RWE study designs and analysis plans, for example via the ISPOR portal, was also supported by the panel members. In terms of policy, the development of the European Health Data Space (39) was agreed as an important step. The panel considered the development of partnerships among stakeholders to be of high importance, particularly initiatives that bring together data sources that may be relevant to HTA, within a trusted research environment, such as EMA’s work on the Data Analysis and Real World Interrogation Network (DARWIN EU) (40).

In addition to the four main pillars, education to upskill HTA bodies and the need for senior-level HTA/payer commitment to provide resources for work in the field of RWE were seen as essential to underpinning infrastructure. Education and expertise links closely to the themes of methodology, trust, and partnership. There is a clear need for the upskilling of all stakeholders involved in the collection, curation, analysis, and appraisal of RWD/RWE. This includes education of and engagement with clinicians who collect RWD in the real-world healthcare setting, particularly those who manage disease registries (such as the European Reference Networks for rare diseases), or those who contribute to the assessment and documentation of outcomes as part of Outcomes Based Managed Entry Agreements (OBMEA). Education is often required to explain the objectives and information needs of HTA, and better engagement facilitates the identification of opportunities for collaboration to generate or provide access to RWD. Multi-stakeholder dialogues focused on planning for RWE were seen as an important aspect of education for all stakeholders to discuss the advantages and disadvantages of different RWD sources that could complement planned clinical studies or be used for OBMEA.

In terms of senior-level commitment and resourcing, the recent strategic initiative taken by the National Institute for Health and Care Excellence (NICE) was noted (41). The NICE Strategy 2021–2026 consists of four pillars, one of which is ‘leadership in data, research and science’, including using RWD to resolve gaps in knowledge and drive forward access to innovations for patients. This commitment has been demonstrated through the NICE leadership of Work Package 6 in the IMPACT HTA project “methodological guidance on the analysis and interpretation of non-randomized studies to inform health economic evaluation” (26). The resulting guidance has been used by NICE to inform their RWE Framework that was launched in June 2022 (42).

Can existing guidance be used by HTA bodies and payers?

The extent of existing available guidance from trusted resources was not known to all panel members, and several publications seemed highly relevant to address challenges faced in HTA, either in its present form, or with adaptation. Without detailed review of all available documents, it was unclear whether gaps remain, but the following areas were considered key for HTA and may be a focus for the development of HTA-specific guidance and policy inputs in the future.

All panel members agreed that RCTs should be undertaken to demonstrate relative efficacy, whenever possible, but they cannot provide all the information needed for HTA body and payer decision-making. Moreover, they agreed that RWE can provide important complementary evidence that can help resolve HTA/payer uncertainties. To provide robust RWE, systematized and transparent methods of data curation are needed that take account of data provenance (e.g., clinical registries, health claims data, patient-reported outcomes) and processes as well as the original purpose of data collection and the associated limitations that imposes. Interoperability and linking between different data sources are critical issues within the context of confidentiality governance legislation, which may be interpreted differently across EU member states or different data owners. Data discoverability is another important consideration; the signposting of high-quality data sources and how they can be accessed is essential. These issues have been addressed by many of the research collaboratives identified, and more recently by regulators, but HTA bodies and payers also have an important role to play in driving the development, access, and use of RWD.

Existing guidance to support protocol-driven high-integrity RWE studies may be sufficient for HTA purposes in terms of how to (i) clearly document data sources, (ii) how relevant patients will be identified, processes for curation of data according to definitions of exposure (treatment), outcomes and covariates, methods for analysis and appropriate sensitivity analyses, but this needs more detailed review. The publication of RWE protocols underpins the methodological pillar but is also an essential element in transparency and building trust.

It is widely acknowledged that greater collaboration is needed among HTA bodies and payers across jurisdictions to anticipate when RWD/RWE may be needed in an OBMEA post the initial HTA/payer decision and to align those RWD requirements, at least to a core data set (6;43). This alignment needs to extend to national and transnational regulatory agencies to consider their post-authorization data collection requirements. Collaborative approaches should acknowledge the needs of different decision-makers, data availability for specific diseases, member state data infrastructures, and in HTA body, clinical and patient capacities. Alignment on the quality of RWD generation for the pre-submission phase, and agreement on the core data set and the use of the same data systems for OBMEAs would help avoid duplication of efforts and be respectful to patients. The costs associated with gathering and analyzing RWD/RWE are also substantial and the role of each stakeholder, particularly industry must be agreed.

For all aspects, sharing of cases where RWE had been critically assessed in HTA or small pilot or demonstration projects were seen as valuable by the panel members.

Conclusions

In order to create robust RWE to inform HTA/payer decision-making it is helpful to consider four key pillars of (i) data (availability, governance, and quality); (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships, underpinned by education and senior level HTA/payer commitment. These themes are in alignment with those raised by other stakeholders. Furthermore, the guidance and tools developed to address issues related to these four pillars provide a strong foundation for HTA bodies and payers to enhance the use of RWE to inform decision-making. Data quality is considered as the highest priority, with a cultural shift needed so that HTA bodies and payers take a more leading role in discussions about RWD to ensure their needs are considered in data constructs (such as disease registries), and data enquiry systems (such as DARWIN EU), alongside input to larger policy initiatives, such as creation of the European Health Data Space (39). A range of methodological guides exist that could contribute to transparency and trust. However, to generate decision-grade RWE, investment is needed, not only in data infrastructure but also in human resources, as the generation and use of RWE requires knowledge and capacity development in all stakeholder groups. A collaborative approach is needed to ensure we do not ‘reinvent the wheel’ and make best use of existing expert guidance regarding the development of RWE that will ultimately improve patient outcomes and optimize treatment use. Existing and emerging collaborations such as RWE4Decisions, EUnetHTA, and the GetReal Institute play a crucial role in both signposting to existing guidance and in the development of bespoke guidance that can support the use of RWD/RWE in HTA and payer decision-making. Such guidance could be used by individual bodies to create their own policies and processes.

There is also a need to publicly document examples that show how the quality of RWD has been evaluated for HTA purposes (e.g., using EUnetHTA REQueEST), appraisals where RWE has been critically assessed, conditional data collection agreements (both national and across border – for example in BENELUXAI, FINOSE or across the nations of the UK or Canadian provinces) and reports of how additional data collection has informed decision-making and could have been improved. Such examples and existing guidance should be leveraged by HTA bodies to co-create, with multi-stakeholder input, seminal HTA guidance on RWD/RWE that is translatable and adaptable to the local context. Finally, we encourage HTA/payer experts to join DARWIN EU, European Health Data Space, and other health data infrastructure discussions to ensure that data, which are of particular interest to payers, such as healthcare utilization and cost data, are collected.