355 Validation of an artificial intelligence Algorithm for predicting diagnosis-related groups in a community health system

Abstract

Objectives/Goals: This study aims to evaluate the performance of a third-party artificial intelligence (AI) product in predicting diagnosis-related groups (DRGs) in a community healthcare system. We highlight a use case illustrating how clinical practice leverages AI-predicted information in unexpected yet advantageous ways and assess the AI predictions accuracy and practical application. Methods/Study Population: DRGs are crucial for hospital reimbursement under the prospective payment model. The Mayo Clinic Health System (MCHS), a network of clinics and hospitals serving a substantial rural population in Minnesota and Wisconsin, has recently adopted an AI algorithm developed by Xsolis (an AI-focused healthcare solution provider). This algorithm, a 1D convolutional neural network, predicts DRGs based on clinical documentation. To assess the accuracy of AI-generated DRG predictions for inpatient discharges, we analyzed data from 930 patients hospitalized at MCHS Mankato between March 2 and May 13, 2024. The Xsolis platform provided the top three DRG predictions for the first 48 hours of each inpatient stay. The accuracy of these predictions was then compared against the final billed DRG codes from the hospital’s records. Results/Anticipated Results: In our validation set, Xsolis achieved a top-3 DRG prediction accuracy of 71% at 24 hours and 81% at 48 hours, which is lower than the originally reported accuracy of 81.1% and 83.3%, respectively. Interestingly, discussions with clinical practice leaders revealed that the most valuable information derived from the AI predictions was the expected geometric mean length of stay (GMLOS), which Xsolis was perceived to predict accurately. In the Medicare system, each DRG is associated with an expected GMLOS, a critical factor for efficient hospital flow planning. A subsequent analysis comparing predicted GMLOS with the actual length of stay showed variances of -0.10 days on day 1 and 0.14 days on day 2, indicating a high degree of accuracy and aligning with clinical practice perceptions. Discussion/Significance of Impact: Our research underscores that clinical practice can leverage AI predictions in unexpected yet beneficial ways. While initially focused on DRG prediction, the associated GMLOS emerged as more significant. This suggests that AI algorithm validation should be tailored to specific clinical needs rather than relying solely on generalized benchmarks.