Beyond clinical and cost-effectiveness: The contribution of qualitative research to health technology assessment

Abstract

Recent developments in health technology assessment (HTA), including the promotion of a new and internationally accepted definition of HTA, have highlighted the need to go beyond clinical and cost-effectiveness to fully understand the potential value of health technologies. Multidisciplinary efforts to generate patient-focused evidence relevant to HTA, using both quantitative and qualitative approaches, are needed. Although it has been more than 20 years since opportunities for qualitative methods to inform HTA were first discussed, their use remains infrequent. The goal of this article is to resurrect the debate about the value of qualitative research in HTA. Drawing on examples from published literature, we propose five key areas where qualitative methods can contribute to HTA, complementary to studies of clinical and cost-effectiveness: (i) assessing acceptability and subjective value; (ii) understanding perspectives and providing context; (iii) reaching the groups other methods cannot reach; (iv) laying the groundwork for subsequent quantitative exercises; and (v) contributing to economic model development.

Introduction

Over the last few years, the health technology assessment (HTA) community witnessed the publication of two documents that could prove very influential in the way that the science, methods, and practice of HTA are perceived. First, in 2019, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) HTA Council Working Group released an up-to-date literature review on existing and emerging good practices in HTA. One of their conclusions was that “moving systematic review and synthesis beyond clinical, epidemiological, and economic research into qualitative and quantitative research in patient-, caregiver-, and citizen-generated information is an immediate need in HTA” (1). Second, in 2020, a new definition of HTA was produced through an international collaboration, co-led by the International Network of Agencies for Health Technology Assessment (INAHTA) and Health Technology Assessment International (HTAi) (2). The new definition emphasized again the multidisciplinary nature of the field, while incorporating a quite extensive note on the different dimensions of value that should be assessed when it comes to health technologies, including not only clinical and cost-effectiveness, but also societal and ethical issues, as well as wider implications for the patient, relatives, and the general population.

It is true that, to date, HTA has heavily relied on context-free, quantitative evidence (e.g., on investigating whether a technology works and is safe, patients for whom it is intended, and so on), restricting the notion of “value” to numerical considerations of benefits, risks, and costs (3). The results of the World Health Organization (WHO) 2015 Global Survey on HTA are a good example of this (4); for all types of health technologies surveyed, national authorities reported that emphasis was mainly placed on safety, clinical effectiveness, and economic/budgetary considerations. Limited (or, in some cases, no) attention was paid to more context-sensitive aspects, such as ethical and equity issues, feasibility considerations, or acceptability to patients and healthcare providers. Indeed, a central assumption running through the HTA field has been that rational decision-making needs to be based on objective evidence, derived from value-free observation, comparison, and experimentation, and aiming to establish “universal truths” that can be applicable regardless of time, place, or environment. As such, contextual issues related to the implementation (or de-implementation) of health technologies, and questions of acceptability and subjective value (i.e., what matters most to key stakeholders?), have been largely ignored. The growing recognition, however, of health care as a complex system, whose individual agents may not only have conflicting values, but also the freedom to behave in ways that are not always predictable (5), calls for a reconsideration of established theoretical and methodological approaches in HTA.

Qualitative research – focusing on the meanings people attach to experiences, the relationship between knowledge, experience, and action, and the social factors that shape these processes (6) – has a long tradition in the social sciences; yet, its role in HTA is still only marginally understood. The first in-depth discussion about the potential uses of qualitative methods in HTA came in 1998, when Murphy et al. (7) published the monograph “Qualitative research methods in health technology assessment: A review of the literature.” The report, commissioned by the UK National Health Service Research and Development (NHS R&D) HTA Program after having been identified as a priority by the Methodology Panel, concluded that qualitative research “can provide valuable information on the implementation and impact of health technologies on both health professionals and patients” (6). More than 20 years later, however, efforts to incorporate qualitative data in HTA processes remain scarce, although concerns about the capacity of HTA to evaluate complex and context-dependent technologies are increasingly being raised (8).

In this article, we seek to resurrect the debate about the value of qualitative research in HTA, by discussing some of the gaps that can be filled using qualitative methods. Drawing on our ongoing work in this area and our complementary skills and experience, ¹ we propose five key areas that we believe are broad enough to cover a range of topics and applications. These are: (i) assessing acceptability and subjective value; (ii) understanding perspectives and providing context; (iii) reaching the groups other methods cannot reach; (iv) laying the groundwork for subsequent quantitative exercises; and (v) contributing to economic model development. To illustrate our points, we use examples from published qualitative and mixed methods literature; these have been selected on the basis of our prior knowledge and to reflect diversity in approaches and objectives, and do not constitute by any means an exhaustive or representative list of research in the area.

Five gaps that qualitative research can fill

Assessing acceptability and subjective value

A substantial amount of qualitative research is currently undertaken alongside randomized controlled trials (RCTs), addressing not only issues related to trial design and conduct but also aspects of the intervention being trialed (9). Funded by the UK National Institute for Health and Care Research, the Cancer and Venous Access (CAVA) trial sought to determine the clinical and cost-effectiveness of three different central venous access devices, commonly used for the delivery of long-term chemotherapy: peripherally inserted central catheters (PICCs), centrally inserted tunneled catheters (Hickmans), and centrally inserted totally implantable venous access devices (ports). A total of 1,061 individuals from eighteen UK oncology units took part in CAVA, making it the largest RCT ever conducted comparing these devices. The trial also incorporated a qualitative component, with a view of better understanding patient and staff views on the acceptability of the three devices. Drawing on twenty-six interviews with clinical staff and eight focus groups with forty-two patients, Ryan et al. (10) showed that, although all three devices were well accepted by patients, ports were perceived to offer unique psychological benefits, including a greater sense of freedom and less intrusion in the context of personal relationships. Despite staff viewing ports more challenging from a clinical management perspective, they cited the same practical conveniences, as well as the emotional and psychological benefits of a less conspicuous or obtrusive device that patients themselves raised. Combined with effectiveness and cost-effectiveness data, the qualitative results provided the necessary evidence to support the development of recommendations for the wider adoption of ports in the UK (11).

Understanding perspectives and providing context

Geneau et al. (12) carried out a qualitative study to aid with the interpretation of their survey findings, suggesting that the average willingness-to-pay (WTP) for cataract surgery in Tanzania was 2,547 Tsh or about 2.30 USD, which was far below the actual cost of providing the service at that time (e.g., 70 USD in Kilimanjaro). Based on semi-structured interviews with forty-seven cataract patients, they found that an individual’s WTP for surgery concerned not only patients themselves but also their relatives. Specifically, participants who reported they were unwilling to pay anything for cataract surgery tended to prioritize the needs of younger family members, expressing concerns that they might be perceived as a “burden” by their children. In other words, they seemed to accept restrictions in their functioning, to improve the opportunities of younger family members. Among the respondents willing to place a monetary value on cataract surgery, WTP often related to how much financial support they felt comfortable asking for from their relatives. The figure of 20–30 USD, for instance, was perceived by most as the highest amount of money that their relatives would be prepared to contribute. Based on these findings, the authors concluded that individual WTP for health services might have little significance in certain age groups and cultures.

Reaching the groups other methods cannot reach

Reframing the title of Pope and Mays’ highly influential 1995 paper, “Reaching the parts other methods cannot reach” (13), qualitative methods are ideally placed to explore the experiences of the so-called “hard-to-reach” groups, such as people with rare diseases. Contrary to quantitative research that requires large sample sizes to produce accurate results, qualitative research can generate meaningful findings based on a much smaller number of “information-rich” cases. In a grounded theory study of Gitelman disease, Caiata-Zufferey et al. (14) used in-depth interviews with twelve Italian patients to develop a typology of their experiences. The authors were able to demonstrate that, in addition to ‘traditional’ influencers of health-related quality-of-life (HRQoL) and disease experience (e.g., demographics and symptom severity), there are other personal factors that also impact on how a disease affects an individual patient. They characterized four distinct patterns of disease experience: patients’ considering Gitelman disease a disabling illness, a normalized illness, a different normality, and an episodic disability. Each pattern of experience was associated with particular ways patients interpreted their symptoms, strategies for managing their condition, lifestyles, and risks to the patient’s psychosocial life. The authors posited that healthcare providers could benefit from considering patients’ own perception of disease to adjust the type of care and advice offered. Such typologies might be particularly useful in healthcare decision making, as these factors are likely to influence aspects directly relevant to HTA, such as treatment-taking behavior, adherence, and healthcare resource use.

Laying the groundwork for subsequent quantitative exercises

Informing the development of quantitative exercises is another important application of qualitative research in HTA. Qualitative methods have value in understanding which constructs should be included in HRQoL instruments, how to meaningfully describe health states for utility elicitation, or which attributes and levels should be included in a discrete choice experiment (DCE; 15–17). Coast and Horrocks (17) initially outlined the value of qualitative methods to inform DCE development in 2007. They recognized that methods traditionally used to generate attributes – which included literature review, clinician feedback, and RCT findings – suffered from methodological limitations and challenges to validity. At that time, they noted that while the value of qualitative methods had been suggested, specific guidance on how these methods could inform attribute and level development was lacking. Using findings from semi-structured interviews with nineteen dermatology patients as a case study, Coast and Horrocks illustrated the repeated iterations that a rigorous qualitative approach would require to fully explore all relevant issues to inform the development of the attributes for a subsequent DCE.

Contributing to economic model development

Qualitative methods can contribute to economic model development by informing overall modeling strategy and best practices (18) or design and parameter estimation for individual models (19). Chilcott et al. (20) used qualitative interviews to explore themes of errors in economic models for submission to HTA and potential solutions to mitigate the risk of such errors. Their findings highlighted the critical importance HTA modelers put on model credibility, and the importance of robust model validation in helping to establish that credibility. Kwon et al. (19) used data collected via focus groups and one-to-one interviews with older adults, to inform modeling approaches for an economic evaluation of a multidisciplinary falls prevention program. Through a framework analysis, they identified a series of methodological and evaluative challenges pertaining to model development. These included: how non-health outcomes and societal intervention costs may be captured; the need to understand and convey the dynamic and heterogeneous nature of aging within the model; the potential impact of key psychological and social factors on intervention update; and the importance of considering the broader concept of wellbeing rather than focusing strictly on physical health or even HRQoL. The analysis also informed model sensitivity and scenario analyses. The authors highlighted the value of qualitative methods specifically to inform how fall prevention programs can be administered to be feasible and effective, and more broadly to help improve the structural validity of economic models.

The way forward

HTA requires high-quality evidence to support informed decision making and, historically, this evidence has largely been derived using quantitative methods. Yet, the growing need for more meaningful stakeholder involvement in HTA has also sparked interest in what qualitative research can contribute to a field dominated by studies of clinical and cost-effectiveness (21). In addition, as shown in Figure 1, developments occurring over the last 20 years, such as the introduction of “qualitative research” as a MeSH term in 2003 or the creation of reporting guidelines for primary qualitative research and qualitative evidence synthesis (22–24), have been pivotal in improving the perceived rigor of these methods. In this article, we have sought to demonstrate that rigorous qualitative studies, providing access not just to a single patient’s voice but to numerous patient accounts systematically collected and analyzed, can meet the standards of high-quality evidence that HTA requires. Nevertheless, results from our recently completed review of HTA submissions to the National Institute for Health and Care Excellence (NICE) and the Canadian Agency for Drugs and Technologies in Health (CADTH) suggest that the quality of much of the qualitative evidence currently considered by decision makers could be improved (25). To make the best use of qualitative methods in HTA, these need to adhere to recognized quality and reporting standards and be focused on areas where they can truly illuminate issues in a unique way that quantitative methods cannot. Moving forward, the field will also benefit from more methodological innovation in the application of rapid qualitative approaches, as well as developing new ways to harness the synergies between qualitative and quantitative data in mixed methods studies.

Figure 1. Qualitative research in HTA: A timeline of key developments.