A review of reported stakeholder engagement in early-stage translational research

Abstract

Background:

We conducted a scoping review of peer-reviewed literature to describe the nature and extent of reporting on the involvement of stakeholders in early-stage translational research.

Methods and results:

We conducted two literature searches in six databases, screened records and full-text articles, and abstracted and analyzed data from included publications. The literature searches yielded unduplicated 2,894 records. After screening, 13 articles were included.

Findings:

Our review of the literature yielded rare reports of engagement in early-stage translational research. Half of included articles reported on engagement with patients, clinicians, and researchers while fewer that one in three reported on engagement with policymakers, industry, and insurers. One in four reported engagement in the publication’s acknowledgments but not in the main text. More than half drew unmeasured conclusions about the outcomes of engagement.

Interpretation:

Our definition of early-stage translation pointed to a specific set of peer-reviewed research; our findings indicate a reporting gap and not necessarily a gap in practice. By addressing four themes–developing a shared language, identifying frameworks and principles, creating a repository of resources, and establishing a research agenda, research leaders can develop new insights about how to engage communities in early-stage translational research.

Introduction

The engagement of stakeholders and communities in clinical and translational research is increasingly expected as investigators, journal editors, and funders in the United States recognize its potential value [1–5]. To address this expectation, a considerable amount of guidance has been published to define important terms [5–9], plan engagement activities, [10,11] develop evaluation measures [12–15] and report on approaches [16].

To date, however, most of the attention has been focused on engagement in the latter stages of translational research: in clinical outcomes research (T3), and in health services, health policy, and public health research (T4). Comparatively little is known about community and stakeholder engagement in the earlier stages of translational research: in pre-clinical studies focused on transformation of concepts and testable prototypes – devices, diagnostics, drugs, or data algorithms – into impact on clinical care and health (T.5); discovery of human health interventions (T1) and discovery of clinical applications (T2). Since 2014, the National Center for Advancing Translational Science (NCATS) at National Institutes of Health (NIH) has requested new insights into stakeholder engagement in early-stage translational research. In the most recent clinical and translational science awards (CTSA) Funding Opportunity Announcement (published in 2018), for instance, NCATS requested that each CTSA applicant should conduct “research in the structures, rationales, operations, purposes, outcomes, and metrics of engagement with stakeholders at each step of the translational process to elucidate principles and practices that make translation maximally efficient, focused, and relevant.” [emphasis added] [17]. We set out in this scoping review to explore what is reported about stakeholder engagement in T.5–T2 peer-reviewed publications since the CTSA program was launched in 2006.

Our primary audiences for this work include funders, research organizations, and researchers working in early-stage translational science. The paper will be helpful, especially to research leaders who wish to invest in and incentivize the engagement of stakeholders in early-stage translational research and to researchers who want to develop community engagement in their early-stage translational research portfolios.

In previous work, we defined stakeholder as an individual or group who is responsible for or affected by health- and healthcare-related decisions that can be informed by research evidence (see Supplementary Material 1) [6]. Meaningful engagement offers real opportunities for stakeholders to inform research during preparations for research such as in topic selection and question development, during the conduct of research such as in data collection and analysis, and after research is concluded to ensure uptake and adoption of the results [5].

Methods

We used a four-step process to conduct the scoping review: search; screening; abstraction; and analysis.

Search

Search #1. We searched MEDLINE for English-language studies that reported on stakeholder engagement in T.5 to T2 research beginning in 2006, the year that the CTSA program was launched, through 2019. In this stage, we sought to capture publications that: 1) reported on early-stage clinical and translational science; (2) included commonly-used terms for communities and stakeholders; and 3) included commonly-used terms for engagement. Followed by consultation with an expert in automated search strategies, this search strategy included more than 80 terms (see Supplementary Material 2).

Search #2. A second search was conducted after the first screening yielded a considerable number of hits but few relevant titles and abstracts. Continuing with expert consultation from an expert in automated search strategies, the second search used fewer than 40 terms and was run again through MEDLINE, yielding fewer and better-targeted results, and updating results through September, 2020 (see Supplementary Material 2). Both searches were retained, cross-referenced, and included in the title and abstract record screening.

Screening

Following searches, we conducted a preliminary screen of article titles and abstracts, using double review by trained readers and adjudication of disagreements by the principal investigator. Records deemed uncertain or relevant proceeded to full-text screening and review. During full-text screening, publications were screened in sequence against three a priori exclusion criteria. Titles and abstracts or full-text articles were excluded if they were (1) not in English, (2) not early-stage translational research (T.5–T2), or (3) did not involve or address stakeholder engagement. We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart to track search, screening, and inclusion data.

To train and support screeners, we developed screener training documents describing the early stages of translational science and defining sufficient evidence of stakeholder engagement (see Supplementary Material 3). To describe early-stage translational science, we excerpted online resources provided by NCATS [18] and Tufts CTSI [19] and we shared published definitions and frameworks of translational research [20–22]. To establish evidence of stakeholder engagement, we developed a table describing where to look in a research publication, what to look for, and how to judge whether there is sufficient evidence of engagement in research. To be defined as sufficient evidence, stakeholders had to belong plausibly to one of seven groups of individuals and organizations who may use or be affected by health research (patients and the public, providers, purchasers, payers, policymakers, product makers, and other principal investigators) [6]. Publications were considered to be stakeholder-engaged if they reported working with any of these stakeholders in the publication’s text or in its acknowledgments.

This review addresses involvement of stakeholders as partners in, not as subjects of, research. Thus, surveys of patients or clinicians as subjects of research, but not as fully-fledged partners, did not qualify on their own for inclusion, and we did not try to infer if a survey led to further engagement with its respondents as stakeholders after the report was published.

However, if a report described use of a survey of stakeholders to inform priorities or conduct further research, it qualified for full-text review.

Abstraction

Data were collected into a MS Excel spreadsheet. We abstracted both publication- and study-level data, including article-identifying information (authors’ names and affiliations, funding source(s), and date of publication), study information (objectives; population, intervention, comparator, outcomes, time, setting, and study type (PICOTSS); translational stage; findings, limitations), and engagement information (who initiated, who was engaged, where reported, engagement activities, stage or research in which engagement was studied, evidence of stakeholder engagement, and outcomes of engagement).

Analysis

Our unit of analysis was the publication. We classified publications using both study-level information and engagement information. We report on the types of stakeholders that were engaged in the included publications and the stages of research, the types of research activities, and the intensity by which they were engaged. We assessed the reported outcomes of engagement.

Results

Search #1 yielded 2,696 records. Search #2 yielded 264 records. After de-duplication, 2,894 records proceeded to record screening of titles and abstracts. Record screening yielded 131 records for full text screening. After full text screening, 13 publications were identified for inclusion in the review (See Fig. 1) [23–35]. No two included publications came from the same study.

Figure 1. Preferred reporting items for systematic reviews and meta-analyses flowchart. Notes: after de-duplication, 2,894 records were identified in a systematic search of the literature. Title, abstract, and full-text screening against formal inclusion and exclusion criteria yielded 13 publications 22–354 for inclusion in the review.

Study-Level Information

Of the thirteen included publications (see Supplementary Material 4), two reported two or more funders, ten reported one funder and one did not report any funding. Funders included NIH (N = 5, Dawson, Dickert, Galadanci, Kuhlman, Sankaranarayanan), other U.S. public sources (N = 4, Dickert, Galadanci, Seifer, Talebizadeh), and international public and private sources (N = 6, Anguzu, Ezran, Galadanci, Nagpal, Osterholm, Wamwenje). Nine of the 13 publications reported on T2 research. Two were examples of T.5, one of T1 research, and one reported on more than one stage of translation. Nine of the 13 included publications represented the generation of new evidence from a single study. Three were evidence syntheses from multiple studies or from expert opinion. One reported on dissemination and implementation of research evidence.

Included research publications were supported by both public and private research funders. The studies spanned research on a range of populations, interventions, outcomes, study designs, and settings and it included studies in both developed and developing countries. Few publications reported engagement across the lifecycle of research, from planning research, to conducting research, and using research evidence in decision-making. Engagement was rarely reported during preparations for research when research topics were selected and research teams were formed. Engagement after the completion of research was also rare when research evidence was prepared for dissemination and implementation in future practice or the next phase of translation. Engagement with stakeholders was most often reported during the conduct of research. Of interest, the methods by which stakeholders contributed to research activities increased in intensity as research progressed toward the dissemination and implementation phase of work.

Engagement content

Information on who was engaged, how, and in what research stages and activities was reported in the text in 11 of 13 publications. In two of the 13 studies, evidence of engagement with stakeholders was reported in the publication’s acknowledgments section (Dickert, Sahay). One publication summarized a conference discussion about involving patients as partners in basic science research (Kuhlman).

In all included publications, engagement of stakeholders was initiated by researchers, and none by existing research-community partnerships or by community members. Ten publications reported engagement with patients, defined broadly as patients, communities, patient advocates, and family members (see Fig. 2). Seven and eight publications, respectively, included reports of engaging with providers and principal investigators. Four reported engaging with product makers such as drug and device manufacturers, and four reported engaging with policymakers. One publication each reported on engagement with payers of healthcare services or with purchasers of health plans and insurance policies.

Figure 2. Stakeholder types reported in T0.5-T2 research. Notes: Similar to our findings in a previous scoping review on reported engagement in T3 and T4 patient-centered outcomes research, 40 reported engagement in T.5– T2 was more likely to involve patients and providers than other stakeholder type.

Engagement processes

The majority of included publications (N = 9) involved stakeholders during the conduct of research, but only a minority involved stakeholders while preparing for research (N = 4) or while disseminating and using its findings in practice (N = 3; see Table 1). Those publications reporting engagement during preparation for research described building research capacity of stakeholders prior to study launch or working with stakeholders to select the research topic. Engagement during the conduct of research was reported for a range of research activities, including defining the research question, choosing relevant outcomes, designing a research protocol, drafting or revising study materials & protocols, recruiting participants, monitoring patient data and safety, and collecting data. Engagement during dissemination was reported for implementing evidence in practice, evaluating research, and identifying topics for future research.

Table 1. Research stages, publications, activities, and stakeholders

Source: Authors’ analysis of full text articles.

Notes: Engagement intensity was calculated using a four-point system describing engagement used in the research stage. Levels of engagement could consist of unidirectional communication about a research activity (1 point), bi-directional consultation where stakeholder opinions were considered by researchers who carried out a research activity (2 points), two-way collaboration where stakeholder and researchers contributed to a research activity (3 points), or co-production where stakeholders and researchers shared control of a research activity (4 points). Multiple scores for one author indicate multiple types of engagement within a single publication. The highest level of intensity was used to estimate medians within research stage.

Publications included in this review indicated that stakeholders contributed to research as members of steering committees or expert panels, and as individuals representing a community. We used a recent adaptation of Arnstein’s “ladder of participation” [36] to categorize these engagement approaches into four levels of intensity: communication (a unidirectional approach in which researchers share study information with stakeholders), consultation (a bi-directional approach in which researchers invite stakeholders to offer advice on a research activity), collaboration (a bi-directional approach in which researchers invite stakeholders to actively contribute to a research activity), and co-production (a bi-directional approach in which researchers and stakeholders share control over a research activity) [37]. Most included studies reported involving stakeholders at the level of consultation, but we found that engagement was increasingly intensive as research progressed from preparation through conduct of research to dissemination and implementation of research findings; stakeholders were more likely to be considered full partners in the work when it came time to put evidence into practice.

Engagement outcomes

Engagement outcomes were measured in a minority of the included publications. In these instances, study authors sought to establish replicable findings that could guide the path forward for future engagement in early-stage translational research. These kinds of findings included observations about the feasibility of engaging stakeholders, study resources that may be consumed by engagement, or stakeholder views and perceptions about the value of engaging in research work. In one publication, impacts on the research process were measured.

Overall acceptance of personal EFIC [exception from informed consent] enrollment was 70.8% among community consultation participants, consistent with some other published reports. Interactive methods were associated with increased acceptance of EFIC and recall of relevant study details; however, acceptance within interactive methods was highly variable, and participants had lower recall of study risks. Choice of consultation method appears to impact both the nature of feedback and the extent to which participants understand study content [25].

More commonly, study authors reported unmeasured assertions about engagement. They tended to focus on engagement as a necessary ingredient to overcoming trial and study challenges. These included challenges such as slow enrollment, low participation, low uptake, and low trust, but in most instances, the conclusions they drew were not supported by evidence presented in the paper itself. Table 2 shows a wide range of these assertions.

Table 2. Unmeasured assertions about engagement in early-stage translational research

Source: Full texts of included articles.

Notes: NR = not reported.

Most of the included publications offered recommendations about engaging stakeholders in future work. They recommended engagement as a means to building support for other elements of their studies, such as recruitment. They recommended engagement as a path to informing future research priority setting or future evidence implementation in practice settings. They also recommended engagement as a way to build relationships in communities in which research work is ongoing.

Discussion

In this review, we identified previously published peer-reviewed literature reporting on the engagement of stakeholders in early-stage translational research. In focus groups described in an earlier publication, early-stage translational researchers describe some level of engagement in their work, but few described reporting on engagement in their peer-reviewed publications [38]. The work in this review appears to bear that finding out: of nearly 2,900 publications identified in two searches, only 13 qualified for inclusion in our review.

We can conclude that reporting on engagement in peer-reviewed publications is extremely rare in early-stage translational research. Further, the content and quality of reporting on who was engaged, how, and in what stages and activities of research were highly variable. A higher proportion of reported engagement with patients and clinicians compared to other stakeholders is consistent with our findings in a previous review of engagement in T3-4 translational research. However, the relatively high proportion of engagement with other researchers found in the current review was not seen in our previous review [39]. This may be because we did not consider multi-disciplinary author lists to count as engagement unless it was reported as such in the main text. The absence of reported engagement with other stakeholder groups – also consistent with our previous review – may reflect either a missed opportunity to record interactions with these groups or a lower frequency of engagement with those stakeholders.

Empirical evaluations of engagement processes and outcomes were infrequently reported.

Recommendations favoring more engaged research were nevertheless common, possibly reflecting substantial trust in the discipline among its practitioners. However, the routine publication of unmeasured assertions that engagement is “key,” “essential,” “critical,” and “imperative” presents a serious risk to trustworthiness of the discipline as a whole: to those in the translational science community who need to see evidence about engagement before adopting it as routine practice, unmeasured claims about its outcomes may strain credulity about its usefulness or importance.

However, these findings may be summed up as a reporting gap and not necessarily a gap in practice. While many early-stage translational scientists may wish to engage with stakeholders in their work and are willing to do so, academic incentives do not reward them for reporting on this aspect of their work. The absence of shared language, workable frameworks, and practical guidance for engagement in early-stage translational science may also impose a self-limiting horizon on how much can be accomplished and competently reported in this nascent field.

To close the gaps, we recommend that community engagement experts develop guidance to address questions that researchers and partners have about what engagement means, how to do it, and how to assess it. U.S.-focused guidance is needed in four areas: (1) how to establish a shared language for engagement so that all parties can articulate and understand what it means to engage stakeholders in early-stage translational research; (2) how to explore and develop frameworks and principles for stakeholder engagement across the translational spectrum and in particular in the earliest stages of translation; (3) the need for a repository of existing resources – including practical guidance, consulting, training, and technical assistance – for supporting a new era of engaged research; and (4) a proposed research agenda addressing priorities in the science of stakeholder engagement.

To date, there is sufficient preparation among research institutions, researchers, and private and public funders to prepare for new insights into these four themes. Private foundations and networks [40–44] and the NIH through its NCATS [17] have been exploring solutions to these challenges already, but their efforts lack broad coordination across the U.S. The work happening in these many places should be studied and synthesized. A coordinated program for advancing engaged research across the full translational spectrum is needed.

Our study faces important limitations. First, the definitions and search terms we used to describe stakeholder, engagement, and early-stage translational research may have pointed to an idiosyncratic set of publications. For instance, we sought out explicit reports of engagement in clinical discovery research. This focus led us to a de facto exclusion: we did not include reports of engagement in early-stage environmental health research, where many more publications can be found. This decision was intentional: in 2017, scientific leaders in the National Institute for Environmental Health Science published a new translational framework that significantly departs from key concepts in the T1–T4 framework [22]. They did so to address a common problem in their field: the first stage of translation in their new framework – Fundamental Questions – commonly encompasses population studies conducted in community settings, such as establishing new links between environmental exposures and health outcomes. Including this body of work would have blurred our focus on clinical discovery conducted in carefully controlled laboratory and clinical research settings, where engagement principles and practices may be distinct from those used to guide collaborations on research conducted in community settings.

Further, although we trained our reviewers on multiple versions of the T1–T4 framework, we relied on the Tufts CTSI’s operating definition to compose our search terms and conduct our screening, which could have affected our results. However, in our view, the Tufts definition is distinguished by adding a half stage to and not subtracting from other definitions. The Tufts definition through T2 is compatible with others, ending at the same place with Phase II clinical trials.

Finally, conducting a scoping review on a vast and variably described field of research is notoriously challenging. It becomes the reviewer’s job to give definition to a whole field with non-uniform reporting practices. For this reason, we developed extensive training documents and definitions to guide our work before it began. The potentially broad scope of a field and its variable reporting practices are arguments for and not against conducting a scoping review, however challenging such a review may be.

For all of these reasons, we recognize the possibility that other scoping reviews might come up with a higher count or different set of publications. However, if other reviews were to focus on engagement in early-stage clinical discovery, they would be unlikely to overwhelm our findings or overturn our recommendations. Our finding is that reporting on engagement is rare and of variable quality. Our recommendations are to work across four thematic areas: developing a shared language, identifying frameworks and principles, creating a repository of existing resources, and funding a research agenda for conducting and reporting on engagement in early-stage translational research.