Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results.

We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group.

The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model.

Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Published evidence on health service interventions should inform decision-making in local health services, but primary effectiveness studies and cost-effectiveness analyses are unlikely to reflect contexts other than those in which the evaluations were undertaken. A ten-step framework was developed and applied to use published evidence as the basis for local-level economic evaluations that estimate the expected costs and effects of new service intervention options in specific local contexts.

Working with a multidisciplinary group of local clinicians, the framework was applied to evaluate intervention options for preventing hospital-acquired hypoglycemia. The framework included: clinical audit and analyses of local health systems data to understand the local context and estimate baseline event rates; pragmatic literature review to identify evidence on relevant intervention options; expert elicitation to adjust published intervention effect estimates to reflect the local context; and modeling to synthesize and calibrate data derived from the disparate data sources.

From forty-seven studies identified in the literature review, the working group selected three interventions for evaluation. The local-level economic evaluation generated estimates of intervention costs and a range of cost, capacity and patient outcome-related consequences, which informed working group recommendations to implement two of the interventions.

The applied framework for modeled local-level economic evaluation was valued by local stakeholders, in particular the structured, formal approach to identifying and interpreting published evidence alongside local data. Key methodological issues included the handling of alternative reported outcomes and the elicitation of the expected intervention effects in the local context.

Telemedicine may improve healthcare access and efficiency if it demands less clinician time than usual care. We sought to describe the degree to which telemedicine trials assess the effect of telemedicine on clinicians’ time and to discuss how including the time needed to treat (TNT) in health technology assessment (HTA) could affect the design of telemedicine services and studies.

We conducted a scoping review by searching clinicaltrials.gov using the search term “telemedicine” and limiting results to randomized trials or observational studies registered between January 2012 and October 2023. We then reviewed trial registration data to determine if any of the outcomes assessed in the trials measured effect on clinicians’ time.

We found 113 studies and of these 78 studies of telemedicine met the inclusion criteria and were included. Nine (12 percent) of the 78 studies had some measure of clinician time as a primary outcome, and 11 (14 percent) as a secondary outcome. Four studies compared direct measures of TNT with telemedicine versus usual care, but no statistically significant difference was found. Of the sixteen studies including indirect measures of clinician time, thirteen found no significant effects, two found a statistically significant reduction, and one found a statistically significant increase.

This scoping review found that clinician time is not commonly measured in studies of telemedicine interventions. Attention to telemedicine’s TNT in clinical studies and HTAs of telemedicine in practice may bring attention to the organization of clinical workflows and increase the value of telemedicine.

The purpose of this study is to evaluate the validity of the standard approach in expert judgment for evaluating precision medicines, in which experts are required to estimate outcomes as if they did not have access to diagnostic information, whereas in fact, they do.

Fourteen clinicians participated in an expert judgment task to estimate the cost and medical outcomes of the use of exome sequencing in pediatric patients with intractable epilepsy in Thailand. Experts were randomly assigned to either an “unblind” or “blind” group; the former was provided with the exome sequencing results for each patient case prior to the judgment task, whereas the latter was not provided with the exome sequencing results. Both groups were asked to estimate the outcomes for the counterfactual scenario, in which patients had not been tested by exome sequencing.

Our study did not show significant results, possibly due to the small sample size of both participants and case studies.

A comparison of the unblind and blind approach did not show conclusive evidence that there is a difference in outcomes. However, until further evidence suggests otherwise, we recommend the blind approach as preferable when using expert judgment to evaluate precision medicines because this approach is more representative of the counterfactual scenario than the unblind approach.

The objective was to identify and describe the published guidance and current academic discourse of ethical issues and standards related to the use of Social Media Research for generating patient insights for the use by health technology assessment (HTA) or health policy decisions.

A scoping review of the literature was conducted in PubMed and Embase and identified 935 potential references published between January 2017 and June 2021. After title and abstract screening by three reviewers, 40 publications were included, the relevant information was extracted and data were collected in a mind map, which was then used to structure the output of the review.

Social Media Research may reveal new insights of relevance to HTA or health policies into patient needs, patient experiences, or patient behaviors. However, the research approaches, methods, data use, interpretation, and communication may expose those who post the data in social media channels to risks and potential harms relating to privacy, anonymity/confidentiality, authenticity, context, and rapidly changing technologies.

An actively engaged approach to ensuring ethical innocuousness is recommended that carefully follows best practices throughout planning, conduct, and communication of the research. Throughout the process and as a follow-up, there should be a discourse with the ethical experts to maximally protect the current and future users of social media, to support their trust in the research, and to advance the knowledge in parallel to the advancement of the media themselves, the technologies, and the research tools.

The study, conducted by the French National Authority for Health (HAS), aimed to identify available online health technology assessment (HTA) training tools for patients, specifically those used by HTA bodies (HTAbs) and major selected European and international patient and consumer groups (PCGs), to inform an HTA training approach for HAS.

A literature search, a review of selected websites, semi-structured interviews with HTAbs and patient groups, and discussion within a dedicated working group to help target the needs and preferences of patients and consumers were conducted. Online HTA training tools relating to HTA and patient and public involvement (PPI), published or translated into English or French, were included in the study results.

Eighty-two online HTA training tools for patients and consumers were selected according to the specified inclusion criteria coming from sixteen international HTAbs, nine European and international PCGs, and thirteen other organizations. Two main categories emerged: the first relating to HTA and the second relating to PPI in HTA. The formats of these tools ranged from interactive and non-interactive formats with varying accessibility and assessment methods. No journal articles mentioned explicitly the content and format of PCG training tools.

This research served as a basis for HAS to develop their own HTA training tools and materials for patients and consumers. Two training tools were subsequently developed, guided by the needs and preferences of a patient and consumer working group, and were published on the HAS website in November 2022.

Horizon scanning for health technology appraisal (HTA) in England involves topic notification to the National Institute for Health and Care Excellence (NICE) via technology briefings. This activity is undertaken by the Innovation Observatory with submission timelines designed to ensure that HTA decisions align with regulatory approval time. In this paper, we aimed to track and assess the progression and current status of the topics notified for HTA and provide a descriptive analysis of these topics.

Technology briefings were mapped from submission to NICE technology appraisal/highly specialized technologies recommendations from April 2017 until October 2021. This was done using a combination of searches on Google and NICE website, searching a downloadable spreadsheet containing NICE topic selection decisions, and querying NICE Topic Selection team. Analysis was undertaken regarding type of indications and interventions of submitted topics and published guidance.

Six-hundred and ninety-three topics entered the NICE scoping process, of which 94 percent were prioritized. As of November 2021, approximately 39 percent of prioritized topics were in scoping/in progress, 31 percent were proposed/completed, 20 percent were suspended/terminated, and 4 percent were referred back to Innovation Observatory (IO) for further monitoring.

Our work demonstrates that horizon scanning for HTA is a complex and time-intensive process. Timelines and progress through HTA is challenging due to the growing number of innovative medicines, significant uncertainties, and limited transparency in clinical development and regulatory pathways. A better understanding of clinical trials and regulatory requirements may help eliminate some of this uncertainty and improve timely HTA.

Though there have been longstanding discussions on the value of ethics in health technology assessment (HTA), less awareness exists on ethics information retrieval methods. This study aimed to scope available evidence and determine current practices for ethics information retrieval in HTA.

Literature searches were conducted in Ovid MEDLINE, LISTA, Scopus, and Google Scholar. Once a list of relevant articles was determined, citation tracking was conducted via Scopus. HTA agency websites were searched for published guidance on ethics searching, and for reports which included ethical analyses. Methods sections of each report were analyzed to determine the databases, subject headings, and keywords used in search strategies. The team also reached out to information specialists for insight into current search practices.

Findings from this study indicate that there is still little published guidance from HTA agencies, few HTAs that contain substantial ethical analysis, and even less information on the methodology for ethics information retrieval. The researchers identified twenty-five relevant HTAs. Ten of these reports did not utilize subject-specific databases outside health sciences. Eight reports published ethics searches, with significant overlap in subject headings and text words.

This scoping study of current practice in HTA ethics information retrieval highlights findings of previous studies—while ethics analysis plays a crucial role in HTA, methods for literature searching remain relatively unclear. These findings provide insight into the current state of ethics searching, and will inform continued work on filter development, database selection, and grey literature searching.

This study aimed to assess whether there have been changes in the quality of clinical evidence submitted for government subsidy decisions on cancer medicines over the past 15 years.

We reviewed public summary documents (PSDs) reporting on subsidy decisions made by the Pharmaceutical Benefits Advisory Committee (PBAC) from July 2005 to July 2020. Information was extracted on the study design, directness of comparison, sample size, and risk of bias (RoB). Changes in the quality of evidence were assessed using regression analysis.

Overall, 214 PSDs were included in the analysis. Thirty-seven percent lacked direct comparative evidence. Thirteen percent presented observational or single-arm studies as the basis for decisions. Among PSDs presenting indirect comparisons, 78 percent reported transitivity issues. Nearly half (41 percent) of PSDs reporting on medicines supported by head-to-head studies noted there was a moderate/high/unclear RoB. PSDs reporting concerns with RoB increased by a third over the past 7 years, even after adjusting for disease rarity and trial data maturity (OR 1.30, 95% CI: 0.99, 1.70). No time trends were observed regarding the directness of clinical evidence, study design, transitivity issues, or sample size during any of the analyzed periods.

Our findings indicate that the clinical evidence supplied to inform funding decisions for cancer medicines is often of poor quality and has been deteriorating over time. This is concerning as it introduces greater uncertainty in decision making. This is particularly important as the evidence supplied to the PBAC is often the same as that supplied to other global decision-making bodies.

The evolution of breast cancer (BC) treatments has resulted in tailored therapies for the different types and stages of BC. Each treatment has a profile of benefits and adverse effects which are taken into consideration when planning a treatment pathway. This study examines whether patients’ preferences are in line with what is considered important from decision makers viewpoint.

An online discrete choice experiment was conducted in six European countries (France, Germany, Ireland, Poland, Spain, UK) with BC patients. Six attributes were included: overall survival (OS), hyperglycemia, rash, pain, functional well-being (FWB), and out-of-pocket payment (OOP). Sixteen choice sets with two hypothetical treatments and a “No treatment” option were presented. Data were analyzed with the use of heteroscedastic conditional, mixed logistic, and latent class models. Marginal rate of substitution (MRS) were estimated for OOP versus the rest of attributes to establish the ranking of preferences for each attribute.

Two hundred and forty-seven patients with advanced or metastatic BC and 314 with early-stage BC responded. Forty-nine percent of patients were < 44 years old and 65 percent had completed university education. The MRS of the analysis demonstrated that “severe pain” is the highest dis-preferred attribute level, followed by “severe impairment in FWB” and OS. Four classes of patients as “decision makers” were identified.

This study suggests that there is heterogeneity in treatment preferences of BC patients depending on their sociodemographic and disease-related characteristics. In combination with clinical guidelines, patient preferences can support the selection and tailoring of treatment options.

The Scottish Medicines Consortium (SMC) conducts early health technology assessment (HTA) of new medicines on behalf of NHSScotland. Assessment of end-of-life (EoL), orphan, and ultra-orphan medicines includes a process to gather evidence from patients and carers during Patient and Clinician Engagement (PACE) meetings. The output of PACE meetings is a consensus statement describing the medicine’s added value from the perspective of patients/carers and clinicians. The PACE statement is used by SMC committee members in decision making. This study compared how PACE participants and SMC committee members rate the importance of information in PACE statements for these medicines.

A survey was undertaken of patient group (PG) representatives and clinicians who participated in PACE meetings, and SMC committee members.

PACE participants who responded (26 PG representatives and 14 clinicians) rated health benefits and ability to take part in normal life as important/very important. Convenience of administration and treatment choice received the lowest rating. Hope for the future received the most diverse response. PACE participants generally rated the importance of quality of life themes higher than committee members (n = 20) but the rank order was similar. Differences between the proportion of PACE participants and committee members who rated themes as important/very important were greatest for treatment choice and hope for the future.

In general, PACE themes and subthemes that were rated highly by PACE participants were also considered important by SMC committee members, indicating that information captured during PACE meetings is relevant when making decisions on EoL, orphan, and ultra-orphan medicines.

Health technology assessment (HTA) is the systematic evaluation of various properties and effects of a health technology. HTA can serve as a bridge between the world of knowledge and that of decision making, offering decision makers the best summary of scientific evidence. Scoping HTA reports in the context of dentistry can help researchers identify grey areas; help practitioners make evidence-based decisions and further initiate better policy making.

To provide an overview on HTAs pertaining to oral health and dentistry in the past decade, map the extension and scope of the methodological practices, key findings, and limitations.

A scoping review was conducted using the Joanna Briggs Institute framework. A comprehensive search for HTA reports was done through the International Network of Agencies for Health Technology Assessment Database from January 2010 to December 2020. Consecutively, electronic databases (PubMed and Google Scholar) were searched. Finally, thirty-six reports were included in this review and analyzed.

A total of 709 articles were initially identified, of which thirty-six met the inclusion criteria. Reviewed HTAs focused on various specialties of dentistry worldwide. Maximum number of reports (N = 5) were related to “prosthodontics and dental implants” and technologies related to preventive dentistry were most commonly assessed (N = 4).

Functional, appropriate, and evidence-based information provided through HTA pertaining to oral health on a regular basis will enable decision makers to have enough data to make decisions on the future use of new technology, modify existing policies, accelerate its translation into practice, and ensure provision of robust dental healthcare services.

The hazard ratio (HR) is a commonly used summary statistic when comparing time to event (TTE) data between trial arms, but assumes the presence of proportional hazards (PH). Non-proportional hazards (NPH) are increasingly common in NICE technology appraisals (TAs) due to an abundance of novel cancer treatments, which have differing mechanisms of action compared with traditional chemotherapies. The goal of this study is to understand how pharmaceutical companies, evidence review groups (ERGs) and appraisal committees (ACs) test for PH and report clinical effectiveness in the context of NPH.

A thematic analysis of NICE TAs concerning novel cancer treatments published between 1 January 2020 and 31 December 2021 was undertaken. Data on PH testing and clinical effectiveness reporting for overall survival (OS) and progression-free survival (PFS) were obtained from company submissions, ERG reports, and final appraisal determinations (FADs).

NPH were present for OS or PFS in 28/40 appraisals, with log-cumulative hazard plots the most common testing methodology (40/40), supplemented by Schoenfeld residuals (20/40) and/or other statistical methods (6/40). In the context of NPH, the HR was ubiquitously reported by companies, inconsistently critiqued by ERGs (10/28), and commonly reported in FADs (23/28).

There is inconsistency in PH testing methodology used in TAs. ERGs are inconsistent in critiquing use of the HR in the context of NPH, and even when critiqued it remains a commonly reported outcome measure in FADs. Other measures of clinical effectiveness should be considered, along with guidance on clinical effectiveness reporting when NPH are present.

To identify which international health technology assessment (HTA) agencies are undertaking evaluations of medical tests, summarize commonalities and differences in methodological approach, and highlight examples of good practice.

A methodological review incorporating: systematic identification of HTA guidance documents mentioning evaluation of tests; identification of key contributing organizations and abstraction of approaches to all essential HTA steps; summary of similarities and differences between organizations; and identification of important emergent themes which define the current state of the art and frontiers where further development is needed.

Seven key organizations were identified from 216 screened. The main themes were: elucidation of claims of test benefits; attitude to direct and indirect evidence of clinical effectiveness (including evidence linkage); searching; quality assessment; and health economic evaluation. With the exception of dealing with test accuracy data, approaches were largely based on general approaches to HTA with few test-specific modifications. Elucidation of test claims and attitude to direct and indirect evidence are where we identified the biggest dissimilarities in approach.

There is consensus on some aspects of HTA of tests, such as dealing with test accuracy, and examples of good practice which HTA organizations new to test evaluation can emulate. The focus on test accuracy contrasts with universal acknowledgment that it is not a sufficient evidence base for test evaluation. There are frontiers where methodological development is urgently required, notably integrating direct and indirect evidence and standardizing approaches to evidence linkage.

Treatment seeking for gender dysphoria (GD) has increased manifold in western countries. This has led to increased interest on evidence-base of treatments, but also discussions related to human rights, identity politics, gender-related structures, and medicalization. Combining these discourses into coherent health policy is difficult. Health technology assessment (HTA) is the golden standard for assessing whether a medical intervention should be included in a health system. A comprehensive HTA should include medical, safety, and cost-utility perspectives, but often also ethical, societal, organizational, and legal concerns. Still, ethics is often omitted in practice. This paper aims to demonstrate how integrated ethical analysis influenced a HTA of complex and controversial topics like GD.

A HTA of medical treatments of GD was conducted using integrated ethical analysis based on the EUnetHTA-model. This integrates ethical thinking into the whole HTA, explicitly analyses ethical topics, and balances arguments using several ethical theories.

Integrating ethics had a significant impact on the HTA process and recommendations. It influenced how the HTA was planned and executed, emphasized autonomy and justice when creating the recommendations, and helped the workgroup to understand the complexity of combining different stakeholders’ discourses. Tensions between scientific evidence, expectations, and values became explicit.

Comprehensive HTA provides an important, integrative approach to considering complex and controversial topics in health systems. HTA emphasizes multidisciplinary and multi-stakeholder approach but simultaneously forces a pragmatic, results-oriented, and evidence-based approach on all argumentation. Ethical analysis can facilitate interactions between stakeholders, bridge different discourses, and help formulate widely acceptable guidelines and policy decisions.

Artificial intelligence (AI) is seen as a major disrupting force in the future healthcare system. However, the assessment of the value of AI technologies is still unclear. Therefore, a multidisciplinary group of experts and patients developed a Model for ASsessing the value of AI (MAS-AI) in medical imaging. Medical imaging is chosen due to the maturity of AI in this area, ensuring a robust evidence-based model.

MAS-AI was developed in three phases. First, a literature review of existing guides, evaluations, and assessments of the value of AI in the field of medical imaging. Next, we interviewed leading researchers in AI in Denmark. The third phase consisted of two workshops where decision makers, patient organizations, and researchers discussed crucial topics for evaluating AI. The multidisciplinary team revised the model between workshops according to comments.

The MAS-AI guideline consists of two steps covering nine domains and five process factors supporting the assessment. Step 1 contains a description of patients, how the AI model was developed, and initial ethical and legal considerations. In step 2, a multidisciplinary assessment of outcomes of the AI application is done for the five remaining domains: safety, clinical aspects, economics, organizational aspects, and patient aspects.

We have developed an health technology assessment-based framework to support the introduction of AI technologies into healthcare in medical imaging. It is essential to ensure informed and valid decisions regarding the adoption of AI with a structured process and tool. MAS-AI can help support decision making and provide greater transparency for all parties.

There has been a lack of health technology assessment (HTA) methods for novel digital health technologies (DHTs) such as mHealth, artificial intelligence, and robotics in Finland. The Digi-HTA method has been developed for this purpose. The aim of this study is to determine whether it would be possible to use Digi-HTA recommendations to support healthcare decision-makers. Secondly, from the perspective of companies offering different types of DHT products, this study assesses the suitability of using the Digi-HTA framework to perform HTAs for their products.

Feedback about Digi-HTA recommendations was collected from healthcare professionals. DHT companies provided input about the Digi-HTA framework. Data were collected via a web-based survey and were analyzed using qualitative methods.

Of the twenty-four healthcare professional respondents, twenty said that the Digi-HTA recommendations contained all the necessary information, and twenty-one found them useful for their work. Respondents hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes and healthcare professionals would be more informed about this new HTA process. The questions of the Digi-HTA framework were applicable for different DHT products based on the responses from DHT companies (n = 8).

According to the study participants, although the Digi-HTA recommendations include clear and beneficial information, their integration into healthcare decision-making processes should be improved. Responses from DHT companies indicate that the Digi-HTA framework would be an appropriate tool for performing assessments for their products. To generalize the findings of this study, more comprehensive studies will be needed.

The reimbursement process for innovative health technologies in Hungary lacks any formalized assessment of clinical added benefit (CAB). The aim of this research is to present the development, retrospective testing, and implementation of a local assessment framework for determining the CAB of cancer treatments at the Department of Health Technology Assessment of the National Institute of Pharmacy and Nutrition in Hungary.

The assessment framework was drafted after screening existing methods and a retrospective comparison of local reimbursement dossiers to that of German and French methods. The Magnitude of Clinical Benefit Scale of the European Society for Medical Oncology was chosen to rate the extent of CAB in oncology, as part of a conclusion complemented by the assessment of endpoint relevance and the quality of evidence. Several rounds of retrospective assessments have been conducted involving all clinical assessors, iterated with semistructured discussions to consolidate divergence between assessors. External stakeholders were consulted to provide feedback on the framework.

Retrospective assessments resulted in average more than 75 percent concordance between assessors on each element of the conclusion. Input from ten stakeholders was also incorporated; stakeholders were generally supportive, and they mostly commented on the concept, the elements of the framework, and its implementation.

The procedure is suitable for routine use in the decision-making process to describe the CAB of antineoplastic technologies in Hungary. Further extension of the framework is required to cover more disease areas for structured and comparable conclusions on CAB of innovative health technologies.