In low- and middle-income countries (LMICs) striving to achieve universal health coverage, the involvement of different stakeholders in formal or informal ways in health technology assessment (HTA) must be culturally and socially relevant and acceptable. Challenges may be different from those seen in high-income countries. In this article, we aimed to pilot a questionnaire for uncovering the context-related aspects of patient and citizen involvement (PCI) in LMICs, collecting experiences encountered with PCI, and identifying opportunities for patients and citizens toward contributing to local decision- and policy-making processes related to health technologies.

Through a collaborative, international multi-stakeholder initiative, a questionnaire was developed for describing each LMIC's healthcare system context and the emergence of opportunities for PCI relating to HTA. The questionnaire was piloted in the first set of countries (Brazil, Indonesia, Nigeria, and South Africa).

The questionnaire was successfully applied across four diverse LMICs, which are at different stages of using HTA to inform decision making. Only in Brazil, formal ways of PCI have been defined. In the other countries, there is informal influence that is contingent upon the engagement level of patient and citizen advocacy groups (PCAGs), usually strongest in areas such as HIV/AIDS, TB, oncology, or rare diseases.

The questionnaire can be used to analyze the options for patients and citizens to participate in HTA or healthcare decision making. It will be rolled out to more LMICs to describe the requirements and opportunities for PCI in the context of LMICs and to identify possible routes and methodologies for devising a more systematic and formalized PCI in LMICs.

To classify, analyze, and compare published guidelines for economic evaluation within health technology assessment (HTA) in European countries and highlight differences and similarities.

We performed a literature review to identify published guidance for the conduct and assessment of economic evaluation studies that are undertaken within the context of HTA processes in European countries. Organizations and working groups were identified via the ISPOR, INAHTA, and EUnetHTA databases. Following the identification of official documents, we performed a qualitative content analysis to highlight discrepancies or common practices under the following categories: comparator, perspective on costs/benefits, time horizon, economic evaluation method, instrument used for utility measurement, outcome measure, source for efficacy, modeling, sensitivity analysis, discounting, and incremental cost-effectiveness ratio.

A total of nineteen guidance documents were identified (in English) providing data for the analysis in nineteen countries. The comparative content analysis identified common practices in most countries regarding the approaches to the choice of comparator, source of data, the preferred economic evaluation method, the option for a lifetime analytical horizon, discounting, and the choice of key outcome measure—for which, most countries recommend the use of the EQ-5D instrument. Differences were mainly found in the choice of perspective, dealing with uncertainty and sensitivity analysis, the use of end points, and the required use of modeling.

The use of economic evaluation constitutes one of the key pillars of the HTA process in Europe. Although a methodological convergence has occurred during the last few years, notable differences still remain.

This paper aims to describe the development of a flowchart to guide the decisions of researchers in the Spanish Network for Health Technology Assessment of the National Health System (RedETS) regarding patient involvement (PI) in Health Technology Assessment (HTA). By doing so, it reflects on current methodological challenges in PI in the HTA field: how best to combine PI methods and what is the role of patient-based evidence.

A decisional flowchart for PI in HTA was developed between March and April 2019 following an iterative process, reviewed by the members of the PI Interest Group and other RedETS members and validated during an online deliberative meeting. The development of the flowchart was based on a previous methodological framework assessed in a pilot study.

The guidelines on how to involve patients in HTA in the RedETS were graphically represented in a flowchart. PI must be included in all HTA reports, except those that assess technologies with no relevant impact on patients’ experiences, values, and preferences. Patient organizations or expert patients related to the topic of the HTA report must be identified and invited. These patients can participate in protocol development, outcomes' identification, assessment process, and report review. When the technology assessed affects in a relevant way patient experiences, values, and preferences, patient-based evidence should be included through a systematic literature review or a primary study.

The decisional flowchart for PI in HTA contributes to the current methodological challenges by proposing a combination of direct involvement and patient-based evidence.

Despite a large number of mobile apps in the field of mental health, it is difficult to find a useful and reliable one, mainly due to the fact that the effectiveness of many apps has not been assessed scientifically. The present study aimed to assess the effects of mental health apps on managing the symptoms of stress, anxiety, and depression.

A comprehensive literature search was conducted in PubMed, Scopus, EMBASE, Cochrane, and Web of Science databases for the papers published from 2000 to 2019. Studies were included if they reviewed articles or mobile apps for their effectiveness in stress, anxiety, and depression. The reviews that had considered mobile apps or web-based mobile applications as an intervention or part of intervention were included, as well.

A total of 4,999 peer-reviewed articles were identified, out of which nine systematic reviews met the inclusion criteria. Seven systematic reviews measured depression outcomes, three measured stress, and five systematic reviews measured anxiety symptoms. The applications that used behavior change strategies, such as Cognitive Behavioral Therapy, Acceptance and Commitment Therapy, and Behavioral Activation, reported significant effects on depression, anxiety, and stress.

It seems that mental health apps can be promising media for reducing depressive symptoms. This field is an emerging area of mobile health, and further research should be done in future in order to reach conclusive evidence.

Increasing emphasis is given on involving patients in health technology assessment (HTA). While this is mainly done at the level of regional and national HTA agencies, this tendency is also emerging in local HTA units. In this study, we provide the results of a survey conducted in local HTA units in the province of Quebec, Canada. The aim of the survey was to provide an overview of local HTA unit practices to involve patients, users, caregivers, and citizens in their process, their interest in doing so, and their information needs for this.

The survey was conducted in 2017 with a response rate of eleven units over a possibility of twelve.

Three units out of eleven (27.3 percent) never involved patients or members of the public in their processes and all indicated that they will involve them in the next few years. The three most important needs for support identified in the HTA units were in: recruiting and selecting patients; integrating experiential knowledge; and knowing and implementing the best methods and practices for partnership.

Patient involvement in local HTA units is quickly evolving and that is why they urgently need tools to involve more effectively patients and members of the public in their process.

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

Integration of ethics into technology assessment in healthcare (HTA) reports is directly linked to the need of decision makers to provide rational grounds justifying their social choices. In a decision-making paradigm, facts and values are intertwined and the social role of HTA reports is to provide relevant information to decision makers. Since 2003, numerous surveys and discussions have addressed different aspects of the integration of ethics into HTA. This study aims to clarify how HTA professionals consider the integration of ethics into HTA, so an international survey was conducted in 2018 and the results are reported here.

A survey comprising twenty-two questions was designed and carried out from April 2018 to July 2018. Three hundred and twenty-eight HTA agencies from seventy-five countries were invited to participate in this survey.

Eighty-nine participants completed the survey, representing a participation rate of twenty-seven percent. As to how HTA reports should fulfill their social role, over 84 percent of respondents agreed upon the necessity to address this role for decision makers, patients, and citizens. At a lower level, the same was found regarding the necessity to make value-judgments explicit in different report sections, including ethical analysis. This contrasts with the response-variability obtained on the status of ethical analysis with the exception of the expertise required. Variability in stakeholder-participation usefulness was also observed.

This study reveals the importance of a three-phase approach, including assessment, contextual data, and recommendations, and highlights the necessity to make explicit value-judgments and have a systematic ethical analysis in order to fulfill HTA's social role in guiding decision makers.

The Ministry of Health in Québec requested the National Institute of Excellence in Health and Social Services to produce clinical and implementation recommendations for the prophylaxis, diagnosis, and treatment of Lyme disease.

(i) Describe the process of trialing different modalities of patient engagement as a means to integrate a diversity of patient perspectives and (ii) Describe the learning process of INESSS regarding the integration of the patient perspective.

All documents were analyzed, and a survey with all advisory committee members and semi-structured interviews with stakeholders were conducted. Each interview was transcribed verbatim and imported into QDA miner software for the purposes of analysis. Data analysis was carried out concurrently with data collection to allow for an iterative approach between data collection and analysis.

Five methods to integrate the perspectives of patients were used: (i) interviews with patients, (ii) inclusion of patient partners within the advisory committee, (iii) literature review, (iv) focus groups with one patient association, and (v) feedback from patient associations on recommendations intended for decision makers and other targeted stakeholders. The patient partners influenced decisions by sharing their experiential knowledge. The patient interviews and the literature review added an in-depth perspective on the disease and experience with the healthcare system. The patient association members shared their perspectives and helped disseminate the recommendation to sustain a practice change.

The combination of methods to collect and integrate patients’ knowledge and patient associations’ perspectives helped develop a comprehensive understanding of a controversial object of evaluation.

The Patient-Centered Outcomes Research Institute (PCORI) horizon scanning system is an early warning system for healthcare interventions in development that could disrupt standard care. We report preliminary findings from the patient engagement process.

The system involves broadly scanning many resources to identify and monitor interventions up to 3 years before anticipated entry into U.S. health care. Topic profiles are written on included interventions with late-phase trial data and circulated with a structured review form for stakeholder comment to determine disruption potential. Stakeholders include patients and caregivers recruited from credible community sources. They view an orientation video, comment on topic profiles, and take a survey about their experience.

As of March 2020, 312 monitored topics (some of which were archived) were derived from 3,500 information leads; 121 met the criteria for topic profile development and stakeholder comment. We invited fifty-four patients and caregivers to participate; thirty-nine reviewed at least one report. Their perspectives informed analyst nominations for fourteen topics in two 2019 High Potential Disruption Reports. Thirty-four patient stakeholders completed the user-experience survey. Most agreed (68 percent) or somewhat agreed (26 percent) that they were confident they could provide useful comments. Ninety-four percent would recommend others to participate.

The system has successfully engaged patients and caregivers, who contributed unique and important perspectives that informed the selection of topics deemed to have high potential to disrupt clinical care. Most participants would recommend others to participate in this process. More research is needed to inform optimal patient and caregiver stakeholder recruitment and engagement methods and reduce barriers to participation.

Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid).

A selection of medical informatics journals was hand searched to identify a “gold standard” (GS) set of references about apps. The GS set was divided into a development and validation set. The filters’ search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters’ overall recall, specificity, and precision were calculated using all the references identified from the hand search.

Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.

The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

Health technology assessment (HTA) can impact health inequities by informing healthcare priority-setting decisions. This paper presents a novel checklist to guide HTA practitioners looking to include equity considerations in their work: the equity checklist for HTA (ECHTA). The list is pragmatically organized according to the generic HTA phases and can be consulted at each step.

A first set of items was based on the framework for equity in HTA developed by Culyer and Bombard. After rewording and reorganizing according to five HTA phases, they were complemented by elements emerging from a literature search. Consultations with method experts, decision makers, and stakeholders further refined the items. Further feedback was sought during a presentation of the tool at an international HTA conference. Lastly, the checklist was piloted through all five stages of an HTA.

ECHTA proposes elements to be considered at each one of the five HTA phases: Scoping, Evaluation, Recommendations and Conclusions, Knowledge Translation and Implementation, and Reassessment. More than a simple checklist, the tool provides details and examples that guide the evaluators through an analysis in each phase. A pilot test is also presented, which demonstrates the ECHTA's usability and added value.

ECHTA provides guidance for HTA evaluators wishing to ensure that their conclusions do not contribute to inequalities in health. Several points to build upon the current checklist will be addressed by a working group of experts, and further feedback is welcome from evaluators who have used the tool.

To evaluate the methodological and reporting characteristics of search methods of systematic reviews (SRs) using the Preferred Reporting of Systematic Reviews and Meta-Analyses (PRISMA) checklist and the Risk Of Bias In Systematic reviews (ROBIS) tool.

A sample of 505 SRs published in 2016 was taken from KSR Evidence, a database of SRs, and analyzed to assess compliance with Information sources and Search of the PRISMA checklist. Domain 2 (D2) (Identification and Selection of Studies) of the ROBIS tool was used to judge the risk of bias in search methods.

Regarding Information sources and Search of PRISMA, twenty percent of SRs which claimed to be PRISMA-compliant in their methods, were compliant; twenty-four percent of SRs published in journals that require PRISMA reporting were compliant; nineteen percent in total were found to be compliant. Twenty-eight percent of SRs were judged to be at a low risk of bias in D2 and so searched widely with an effective strategy and, finally, ten percent were both compliant with the reporting of Information sources and with Search of PRISMA and were judged to be at a low risk of bias in D2.

Ninety percent of SRs are failing to report search methods adequately and to conduct comprehensive searches using a wide range of resources. Editors of journals and peer reviewers need to ensure that they understand the requirements of PRISMA and that compliance is adhered to. Additionally, the comprehensiveness of search methods for SRs needs to be given more critical consideration.