For approximately a decade, health technology assessment (HTA) organizations in Canada and around the world have been involving the public and patients in their work, a domain that has typically been dominated by scientific and clinical experts (Reference Hailey1–Reference Bridges and Jones4). Much of this activity falls under the banner of public and patient involvement (PPI), a term used to capture a broad range of efforts to involve members of the public and patients in processes that guide health system decision making. The increased attention being given to PPI in the HTA arena has been observed in several recent surveys and reviews of international practice (Reference Hailey, Werko and Bakri2;Reference Gagnon, Desmartis and Lepage-Savary5–Reference Whitty7), most of which have documented progress on this front as incremental and piecemeal.
Ontario has led some of the early PPI developments in the Canadian HTA field (Reference Levin, Goeree and Sikich8). In 2007, the Ontario Health Technology Advisory Committee (OHTAC)Footnote 1 , now a standing committee of Health Quality Ontario (HQO), which makes recommendations about health interventions to the health system and the Ontario Ministry of Health and Long-Term Care (9), formed a Public Engagement (PE) Subcommittee to guide its early efforts in this area (10). The PE Subcommittee's recommendations led to the introduction of several opportunities for public and patient involvement within its HTA process (Figure 1). Most notably, OHTAC established a 21-day public and professional consultation process for draft OHTAC recommendations. Additionally, enhanced Web-based communication mechanisms were established to post information about the agency's HTA process, draft HTA recommendations and reports, and to elicit wide-ranging input on draft and final recommendations (11).
Figure 1. HQO-OHTAC HTA process. HQO, Health Quality Ontario; OHTAC, Ontario Health Technology Advisory Committee. Note: This was the HQO Evidence Review Process at the time of the PE Subcommittee deliberations.
In addition to these institutionalized PPI mechanisms, between 2008 and 2010, OHTAC experimented with several ad-hoc initiatives for involving the public and patients at various stages in the HTA process (Reference Bombard12–Reference Abelson, Bombard, Gauvin, Simeonov and Boesveld14). While each experience yielded outputs that informed specific health technology assessments, addressing the more comprehensive task of sorting out which patients and publics to involve with what methods and at which juncture in the HTA process, has remained an elusive goal in the HTA literature and in practice. This study addresses this gap by reporting on the development and outputs of a comprehensive framework for involving the public and patients in a government agency's HTA process.
METHODS
The deliberations of OHTAC's PE Subcommittee (“Subcommittee”), together with an independently funded research project led by the first author, informed the framework's development. The Subcommittee met monthly between January 2013 and July 2014. Three evidentiary sources informed its deliberations: a Web site review of HTA organizations, a review of published literature about PPI in HTA, and a stakeholder dialogue (Reference Gauvin, Abelson and Lavis15;Reference Gauvin and Lavis16). Sources addressed the following questions: (i) Why are HTA organizations involving the public and patients? (goals and rationales). (ii) How are they doing this? (descriptive evidence about involvement approaches, including who is being involved and through which mechanisms). (iii) With what results? (evaluative evidence about the effectiveness of PPI approaches). (iv) What are the barriers, challenges, and facilitators of PPI in HTA organizations? (feasibility considerations)
Web Site Scan and Literature Review
We conducted a systematic Web site scan of international HTA agencies in August 2013. We identified eighty-three HTA agencies from forty-six countries from the member lists of EuNeHTA, INAHTA, Euroscan, and organizations reporting to the NIHR HTA Database. Member organizations included a mix of supportive and responsible HTA agencies. We included agencies with English-language Web sites (n = 17) or in-house translation capabilities (n = 36), which narrowed the sample to 53 HTA agencies across 34 countries. We scanned the Web sites for relevant links, followed by a keyword search using the terms “public participation”, “public engagement”, “public involvement”, “consumer involvement”, and “patient involvement”. We also gathered relevant documents from the Patient and Citizen Involvement section of the Health Technology Assessment international's (HTAi) Web site. We managed the extracted data and analysis using Excel®.
For literature sources, we included published surveys of international HTA practice related to PPI (Reference Hailey, Werko and Bakri2;Reference Menon and Stafinski6;Reference Whitty7;Reference Hailey and Nordwall17), a small but highly relevant set of reviews of empirical studies and conceptual analyses of PPI in HTA published between 2007 and 2011 (Reference Abelson, Giacomini, Lehoux and Gauvin3;Reference Gagnon, Desmartis and Lepage-Savary5;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18), and recent empirical and conceptual studies using the search strategy developed by Gagnon et al. (Reference Gagnon, Desmartis and Lepage-Savary5) for PubMed. The latter retrieved 360 potentially relevant papers published from February 1, 2009, to September 25, 2013. After reviewing titles and abstracts for relevance, we discarded papers that did not address PPI and retained 30 empirical papers and twelve conceptual papers for full-text review. A qualitative content analysis of these recent papers, together with previously published surveys and reviews (Reference Hailey, Werko and Bakri2;Reference Abelson, Giacomini, Lehoux and Gauvin3;Reference Gagnon, Desmartis and Lepage-Savary5–Reference Whitty7;Reference Hailey and Nordwall17;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18), captured evidence published from 1990–2013.
Stakeholder Dialogue
In May 2014, we convened a stakeholder dialogue with nineteen participants: two policymakers, six managers, four researchers, and four citizens from across Canada, and three international researchers. Five participants were either members of HQO personnel or the PE Subcommittee. The dialogue examined the problem of PPI in HTA, elements of a comprehensive approach for addressing it, and implementation considerations. We precirculated to participants an evidence brief (Reference Gauvin, Abelson and Lavis15) organized around each of these three dialogue topics. The event produced a dialogue summary (Reference Gauvin and Lavis16) that provided new information and critical input to the Subcommittee's deliberations and recommendations. The contents of these documents are publicly available (Reference Gauvin and Lavis16).
Framework Development Process
The framework development process followed an iterative approach comprised of three phases. The first phase involved critical reflection on the descriptive and evaluative results of the Web site scan and literature review. These sources provided the Subcommittee with essential guidance about current international practice, the available evidence to support different approaches, and feasibility and implementation considerations. A second phase established the conceptual foundation for the framework, drawing on material gathered during the first phase with a focus on seeking clarity about key terms such “patients”, “the public”, “involvement”, and “engagement”, and the establishment of guiding principles for determining when and how to involve patients and the public in Ontario's HTA process. The final phase involved the integration of the outputs of the first two phases with those of the evidence brief-informed stakeholder dialogue to produce the conceptually robust, evidence-informed framework for public and patient involvement that follows.
RESULTS
Web Site Scan and Literature Synthesis Results
Web site and literature content addressed four key themes: (i) the goals or rationales provided for PPI; (ii) the PPI approaches used at different stages in the HTA process; (iii) evidence of effectiveness or impact of involvement; and (iv) feasibility considerations, including barriers to and facilitators of PPI.
Goals and Rationales for Public and Patient Involvement
Our review of HTA Web sites identified only a small number (n = 5) of organizations that articulated a clear set of goals or rationales for involving the public or patients in their activities (19–23). They highlight 3 main goals for PPI: (i) improving the relevance of assessments (19;21;22); (ii) strengthening the research and complementing the expertise of healthcare professionals and researchers (21–23); or (iii) enhancing procedures, that is, the openness and inclusiveness of the decision process (20).
The published literature offers a more conceptually rich discussion of the goals for PPI. Earlier work, which first applied theories of public participation to HTA, is especially illuminating. From a democratic perspective, PPI may be a means for promoting accountable health systems that respond to public and patients’ values and expectations, and taxpayers’ financial stakes in publicly financed health systems (Reference Wynne24–Reference Bovens, Ferlie, Lynn and Pollitt26). Contexts of scarce resources and rapid technological change present policymakers with complex and contentious funding decisions (Reference Chinitz27;Reference Coulter and Ellins28). These contribute to the emergence of PPI mechanisms as a political imperative for more informed, transparent, accountable, and legitimate decisions about health technologies (Reference Abelson, Giacomini, Lehoux and Gauvin3;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18).
The value of PPI has also gained traction as a way of promoting a more comprehensive approach to HTA rather than relying solely on clinical and cost-effectiveness evidence. The political and ethical backdrops of the health technology policy landscape have precipitated a more comprehensive approach to determine which technologies a publicly funded health plan can justify morally, economically, and functionally. In recent years, researchers and HTA agencies have given more serious consideration to incorporating social values and ethics into HTAs (Reference Facey, Boivin and Gracia29;Reference DeJean, Giacomini, Schwartz and Miller30), to greater patient-focused HTAs that incorporate the patients’ values, needs, preferences, and lived experiences (Reference Bridges and Jones4;Reference Facey, Boivin and Gracia29;Reference Lehoux31), and to involving a broader range of stakeholders including patients and service users as well as the broader public in conducting HTAs (Reference Abelson, Giacomini, Lehoux and Gauvin3;Reference Bridges and Jones4;Reference DeJean, Giacomini, Schwartz and Miller30).
PPI in HTA may also achieve instrumental goals, to make better quality decisions across all the stages of an HTA (Reference Abelson, Giacomini, Lehoux and Gauvin3). For example, HTA agencies may seek the most meaningful ways to gather public and patient input to improve the prioritizing of requests, the scoping of the assessment topic, the development of recommendations, or the dissemination of findings (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18;Reference Facey, Boivin and Gracia29). Lastly, PPI may pursue developmental goals, such as increasing the public's understanding of health technologies and HTA, as well as strengthening the public's and patients’ competence and capacity to contribute to health technology-policy issues (Reference Abelson, Giacomini, Lehoux and Gauvin3;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18).
More recent published discussion echoes and builds on these goals. Generally, making the work of HTA agencies relevant to the public is cited as necessary to gain public support for its funding (Reference Menon and Stafinski6;Reference Drummond, Tarricone and Torbica32). Menon and Stafinski (Reference Menon and Stafinski6) advocate patient involvement in every step of the HTA process, to ensure that the assessment adopts a broader health condition perspective, rather than the narrower technology perspective characteristic of more traditional HTAs. Gagnon et al. (Reference Gagnon, Desmartis and Lepage-Savary5;Reference Gagnon, Desmartis and Gagnon33) cite three key rationales for PPI. First, PPI provides context for the research, which improves the usefulness of assessments for decision makers. In turn, this improves the appropriateness and applicability of the recommendations that arise from HTAs. Second, PPI might contribute to better acceptability, adoption, and implementation of recommendations. Finally, PPI focuses on the primary beneficiaries of the decisions.
Public and Patient Involvement Approaches
Our Web site scan identified thirteen of the fifty-three HTA agencies (25 percent) with documented approaches to PPI (20;22;23;34–44), roughly consistent with proportions documented by previous surveys (Reference Hailey, Werko and Bakri2;Reference Whitty7). It is worth noting that decisions to undertake PPI activities within agencies that support other agencies responsible for HTA decision making may be constrained by the scope and funding received for their work; however, we were unable to assess this objectively. The most commonly used PPI approach (n = 8 agencies) is to have patients and/or members of the public directly participate on committees of the agency (19;23;34;37;41;42;44); very few responding agencies indicated that they had engaged the public through more participatory approaches (Reference Whitty7;Reference Hailey and Nordwall17).
Conversely, the results of a self-reported survey of HTA agencies identified a much stronger emphasis on communication and consultation mechanisms (Reference Whitty7). The most commonly used mechanisms reported in the survey include communicating through public meetings, or consulting through the use of documents or focus groups (Reference Whitty7). Many of the agencies facilitated communication by preparing plain-language versions of their reports to increase accessibility of the assessment (Reference Hailey and Nordwall17). Patient organizations also increasingly participate in disseminating HTA findings (Reference Hailey and Nordwall17).
Those HTA agencies that use PPI approaches do so across all stages of the HTA process. Most (n = 7; 54 percent) formally solicit “public comment” toward the end of the HTA process (11;19;23;41;42;45–47). Public reviews of draft documents, surveys, or face-to-face discussions occur most typically at the appraisal and draft recommendation stage (Reference Whitty7). At the earliest topic selection stage, six agencies (46 percent) invite the general public to submit assessment requests (22;34;36;37;41;42). The HTA literature notes a trend among HTA agencies toward increased involvement of patients and the public in the early stages of HTA (e.g., topic selection, prioritization, and scoping stages), as well as the value of PPI in the evidence analysis process to capture patient values and experiences (Reference Hailey, Werko and Bakri2;Reference Abelson, Bombard, Gauvin, Simeonov and Boesveld14;Reference Oliver, Armes and Gyte48). However, only three agencies (23 percent) currently claim to involve the public or patients in articulating the research questions to guide the HTA process, or in the evidence collection and analysis process (23;34;44).
Evidence of PPI Effectiveness
We found minimal evaluation of the effectiveness of PPI or the impacts of involvement on the HTA process or decision making; both Web sites and literature evince this gap (Reference Menon and Stafinski6;Reference Abelson, Bombard, Gauvin, Simeonov and Boesveld14;Reference Jones and Einsiedel49;Reference Chambers, Wilson and Thompson50). The few efforts document potential instrumental benefits (e.g., including patient preferences and patient-relevant outcomes in HTAs) and developmental benefits (e.g., raising public awareness and understanding) (Reference Gagnon, Desmartis and Lepage-Savary5;Reference Moran and Davidson51). A small number of ethnographic evaluations document the impacts, and facilitators or barriers, of a particular type of public involvement mechanism, for example, face-to-face citizen deliberations (Reference Abelson, Bombard, Gauvin, Simeonov and Boesveld14;Reference Jones and Einsiedel49;Reference Davies, Wetherell and Barnett52).
PPI Feasibility Considerations
HTA agency Web sites do not address feasibility considerations related to PPI, but the published literature emphasizes organizational and recruitment challenges (Reference Menon and Stafinski6;Reference Drummond, Tarricone and Torbica32;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis53). Cultural challenges include tensions between the traditional focus within HTA organizations on clinical and economic evidence, and pressures to incorporate patient and social values input. Public involvement initiatives must combat beliefs that patient views are an anecdotal and biased source of evidence, and perceptions that patients and publics are unable to contribute meaningfully to the process (Reference Menon and Stafinski6;Reference Drummond, Tarricone and Torbica32;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis53). Some papers also noted perceptions that engaging patients and the public would politicize what should be a solely evidence-informed process (Reference Menon and Stafinski6).
Key organizational challenges include the time, financial resources, and expertise required to support high-quality PPI. Ensuring capacity requires organizational commitment. Recruitment challenges involve struggles to obtain “representative” input, and concerns that PPI will allow narrow interests to trump fairness considerations (Reference Oliver, Armes and Gyte48). Enablers to meaningful PPI include a supportive organizational culture, appropriate supports for patient/public committee members and those interacting with them, dedicated time devoted to “patient perspectives” on meeting agendas, and strategic use of new and existing networks of patient organizations for external consultations (Reference Menon and Stafinski6;Reference Facey, Boivin and Gracia29).
Framework for Public and Patient Involvement in Ontario's HTA Process
Informed by the committee deliberations and evidentiary sources discussed above, our PPI framework is organized around four actionable elements detailed in the following sections.
Element 1: Articulate the Principles and Goals for Public and Patient Involvement in HTA
The committee agreed to six principles that should guide HQO's approach to public and patient involvement: (i) purposeful; (ii) pragmatic; (iii) fair and equitable; (iv) proportional; (v) evidence-informed; and (vi) transparent (Supplementary Table 1). Many of these principles are common to the values identified in HTAi's Values and Quality Standards for Patient Involvement in HTA, which resulted from an 18-month Delphi consensus process with input from 150 respondents in thirty-nine countries worldwide (54).
Table 1. Building a common language for public and patient involvement in HTA
HQO, Health Quality Ontario; OHTAC, Ontario Health Technology Advisory Committee.
Linked to these principles is a set of operational goals specific to the HQO-OHTAC process, and which provide the foundation for the PPI framework:
-
1. PPI will be strengthened and supported: (i) to improve the quality of the outputs that arise from each stage of the HQO-OHTAC process; (ii) to create more informed, transparent, accountable and legitimate processes for deliberating about health technologies; (iii) to promote a more robust and comprehensive science of HTA that investigates social values and ethics, as well as patients’ problems, lived experiences, outcomes, and preferences; (iv) to increase public and patient understanding of health technologies and HTA, and to strengthen the public's and patients’ competence and capacity to contribute to various stages of the health technology assessment process.
-
2. PPI efforts will be informed by evidence (where possible), best practice in the absence of evidence, and sound principles.
-
3. Evaluation is to be embedded into all aspects of HQO's PPI activities given the lack of a strong evidence base in this area and given HQO-OHTAC's emphasis in this area.
-
4. PPI efforts will support and, in turn, be supported by HQO's decision-making framework, which includes a specific emphasis on the incorporation of social values and ethics into the evidence-review process.
Element 2: Establish a Common Language to Support Public and Patient Involvement Efforts
Conceptual and empirical studies have noted divergent views within the HTA community, about what “public and patient involvement” means (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18). This ambiguity can lead to conflicting goals and visions for PPI in HTA (Reference Hailey, Werko and Bakri2;Reference Whitty7;Reference Hailey and Nordwall17;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18), and more practically, conflicting views about who should be engaged, who they represent, what role they should play at different stages of the HTA process, and using what types of involvement mechanisms (Reference Menon and Stafinski6;Reference Whitty7;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis18). Our framework emphasizes the importance of establishing a common language to effectively support efforts to involve the public and patients in the HTA process. Informed by the literature synthesis, we defined the “who” of PPI with three categories of relevant perspectives: (i) public (or publics or citizens); (ii) patients, family members, and caregivers; and (iii) stakeholders (Supplementary Figure 1).
The “what” of PPI was established by adopting Rowe and Frewer's (Reference Rowe and Frewer55) typology of involvement, which includes three commonly cited levels of involvement: communication, consultation, and participation (Supplementary Figure 2). Each type reflects a different set of relationships between the initiator of the involvement activity and the participants. Taken together, these mechanisms are often portrayed as a spectrum of involvement with increasing opportunities for participants to be empowered through each stage of the PPI process. These conceptualizations are linked to illustrate the early but critical step in any PPI strategy where key decisions are made about which publics to involve and how, guided by explicit goals (Table 1).
Figure 2. Rationales and Goals for Patient and Public Involvement by HTA Stage. *Newly proposed stage to provide early opportunity for PPI. †This stage was added following the tabling of the subcommittee's report in July 2014. ‡Not addressed by the subcommittee. HQO, Health Quality Ontario; HTA, Health Technology Assessment; OHTAC, Ontario Health Technology Advisory Committee.
Element 3: Describe a Flexible Menu of Public and Patient Involvement Approaches
A key message from the Web site scan and literature synthesis is that there is no one-size-fits-all approach to PPI. Rather, the choice of method should be considered in the context of each HTA stage and matched to the motivation for incorporating societal and/or patient perspectives into the process, and the relevant societal and/or patient values at stake. We have undertaken this mapping process in two stages. In the first stage (Figure 2), we have listed the potential goals for PPI that might be pursued in each HTA stage. In the second stage (Figure 3), we have provided a menu of PPI mechanisms and tools that could be used in each of these stages depending on the goal and the societal or patient perspectives sought.
Figure 3. Who to Involve and How by HTA Stage. *Indicates link to Subcommittee report recommendation. HQO, Health Quality Ontario; HTA, Health Technology Assessment; OHTAC, Ontario Health Technology Advisory Committee.
The mapping process focuses on enhancing the transparency, legitimacy, and overall quality of at the front end of the HQO-OHTAC process, especially in the topic selection and scoping stages. This emphasis reflects the Subcommittee's priorities to facilitate increased awareness and understanding among interested patients, broader publics, and stakeholder groups, and about how topics are selected for review and appraisal, a stage of the HTA process that is opaque in most HTA organizations.
Seen as equally important was the need to increase the relevance and responsiveness of OHTAC's work by undertaking consultation and participation activities to prospectively identify potential review topics and emerging issues of concern to patients and priority populations, and to ensure that the framing of the questions guiding the evidence review process reflect the values of relevant health technology users. These areas of emphasis were reinforced by our synthesis of the evidence, which revealed increased attention to, and sophistication of, the methods used to involve patients and citizens in these early HTA stages (e.g., patient members of expert panels, on-line and face-to-face consultations with individual patients, and/or patient groups, and the synthesis of primary qualitative research studies about patients’ views and experiences).
The committee also sought to strengthen and diversify the communication and consultation activities already present in the later stages of the HQO-OHTAC process, again, to enhance the awareness of OHTAC's work among relevant populations, and to encourage broader input on its draft recommendations.
Element 4. Measurement and Evaluation
Given the weak evidence base for PPI in the HTA field, efforts in this regard should initially focus on basic formative evaluation metrics to determine whether the intended goals of the PPI activities are being achieved, including basic process and impact measures such as numbers and types of consultations, how different types of PPI input are being used in the various stages of the HTA process, and the resources required to carry out relevant activities. These will provide valuable information to inform mid-course adjustments to approaches. Over time, more robust evaluation metrics could be developed, allowing the possibility to compare different PPI approaches using trial and quasi-experimental designs consistent with several recently published high-quality public involvement evaluation studies in related fields (Reference Carman, Heeringa and Heil56;Reference Boivin, Lehoux, Lacombe, Burgers and Grol57).
DISCUSSION
Our framework makes novel and important contributions to the field of public and patient involvement in HTA. First, it explicitly identifies a range of goals that might guide HTA activities and links these to different constituencies of patients and citizens, and appropriate PPI mechanisms. Explicit mapping of PPI goals to participants, and approaches, has been absent from international HTA, and other health system, PPI practices. Second, we systematically map these PPI goals, participants, and approaches through each stage of a government agency's HTA process. Together these provide organizations with a comprehensive menu of options, indicating specific goals and stages that may warrant PPI and which approaches may add value beyond current efforts. Third, our framework closely aligns with the mission of the responsible government agency and its current efforts to refresh its decision-making framework, which features four attributes: (i) overall clinical benefit, (ii) value for money, (iii) social values and ethics, and (iv) feasibility. The framework suggests specific ways PPI may inform these decision criteria, bringing rigor and sophistication particularly to the task of integrating social and patient value judgments with clinical and economic evidence into the HTA process.
As a product of a committee process embedded within a large government agency, the framework development process faced several constraints. Emphasis on pragmatism and implementation feasibility may have come at the expense of comprehensiveness. The Subcommittee's timeline of 18 months prevented the piloting of key framework elements, which may have improved its robustness. The compressed timeframe also limited the committee's ability to comprehensively address the central question of which health technologies might be good candidates for a PPI strategy and what criteria should guide these decisions. Finally, the tailoring of the framework to the specific HTA context of a government agency in Ontario, Canada may limit its relevance to organizations with broader or more limited HTA mandates.
These limitations may also be viewed as strengths. In particular, the pragmatic approach taken by a committee membership with diverse backgrounds and experiences related to HTA, and other health system settings, likely contributed to a more grounded product that resonates with organizations facing similar challenges. Moreover, the alignment of the committee's work with an independently funded research team focused on a similar set of objectives, with access to multiple data sources, including the reflections from a stakeholder dialogue, were unique features that likely strengthened the quality of the final product and its applicability to a variety of settings.
Uptake of the Framework and Corresponding Recommendations
The PPI framework was accompanied by a series of recommendations designed to make measurable progress over a short period of time and with limited resource investments (Reference Ontario11). The Board of HQO received these recommendations in October 2014 and has subsequently moved toward implementing several of them (Reference Ontario58).
CONCLUSION
The framework presented combines a strong conceptual foundation with actionable elements informed by a synthesis of international practice and published research evidence, and a dialogue with Canadian and international stakeholders. As such, the framework as a whole, or selected core elements, should be broadly applicable to other HTA agency processes in Canada, and to other health system organizations pursuing similar health quality agendas. We encourage these organizations to consider this framework as they work toward strengthening their PPI practices.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462316000362
Supplementary Figure 1: http://dx.doi.org/10.1017/S0266462316000362
Supplementary Figure 2: http://dx.doi.org/10.1017/S0266462316000362
CONFLICTS OF INTEREST
The authors report no conflicts of interest.
