OP115 Scoping Meta-Review On Methods Used To Assess Artificial Intelligence-Based Medical Devices For Heart Failure

Abstract

Introduction

The International Federation of Medical and Biological Engineering created a multidisciplinary working group to discuss assessments of artificial intelligence and machine learning (AI/ML) applications in health care. Engineers, clinicians, and economists identified evidence generation as a critical topic. Heart failure (HF) was selected to investigate the available evidence on the clinical effectiveness and safety of AI/ML applications. Attention was paid to transparency of AI/ML methods and their data sources.

Methods

A scoping review was conducted on AI/ML algorithms developed for the management of HF. A search for systematic reviews, scoping reviews, and meta-analyses published from 1976 to October 2022 was conducted in Embase, MEDLINE, and Scopus.

Results

Of 456 relevant publications, 21 papers were included in the final analysis. Most papers (10 systematic reviews, five meta-analyses, and six non-systematic or scoping reviews) included studies conducted in North America. No study was conducted in Africa. The healthcare setting was not clearly stated in approximately half of the studies. A lack of agreement was noticed regarding the quality assessment tools used among the reviews. The most common data source for AI/ML algorithms was electronic health records, but in some cases data sources were not reported. While deep learning emerged as the most common adopted methodology, covariates were not always included in the algorithm development. The review demonstrated that comparative assessment of algorithms requires further investigation, given the high variability in the comparator used (e.g., clinical gold standard, other AI/ML algorithms, or other statistical methods). The main investigated endpoints were the incidence of HF and the number of hospital admissions.

Conclusions

When assessing innovative health technologies such as AI/ML applications in health care, evidence is among the main challenges. Our scoping review, focusing on algorithms developed to manage HF, showed that the biggest challenges relate to the quality of the studies, the adoption of a comparative approach, and transparency of methods.