362 Evaluating the educational quality of ChatGPT as a health information resource for patients with acute myeloid leukemia (AML)

Abstract

Objectives/Goals: Upon diagnosis, patients with acute myeloid leukemia (AML) have significant information needs. Given its recent increase in popularity, patients may use ChatGPT to access information about AML. We will examine the quality, reliability, and readability of information that ChatGPT provides in response to frequently asked questions (FAQs) about AML. Methods/Study Population: From FAQs on the top 3 patient-facing websites about AML, we derived 26 questions, written in lay terms, about AML diagnosis, treatment, prognosis, and functional impact. We queried ChatGPT-4o on 10/14/2024 using a new Google account with no prior history. We asked each question in a separate chat window once, verbatim, and without prompt engineering. After calibration, 5 oncologists independently reviewed ChatGPT responses. We assessed quality via the Global Quality Scale (GQS), scored from 1 (poor) to 5 (excellent) based on flow, topic coverage, and usefulness. For reliability, we assessed whether each response addresses the query and is factually accurate, elaborating on specific inaccuracies. For readability, we assessed Flesch-Kincaid Grade Level, Gunning Fog Index, and Simple Measure of Gobbledygook. Results/Anticipated Results: This will be a descriptive analysis of ChatGPT responses. For quality and reliability assessments, we will report Fleiss’ kappa for inter-rater reliability and expect substantial agreement or greater (≥0.61). Per prior studies in other domains, we hypothesize that ChatGPT responses will have good quality on average (i.e., GQS score near 4). We hypothesize that nearly all responses will address their query and will mostly be accurate; a minority of responses may have partial inaccuracies. Finally, we hypothesize that readability metrics will suggest that a higher educational level (e.g., college-level education) is required for comprehension. Overall, these findings will help elucidate strengths and limitations of ChatGPT for AML and guide discussion of factors patients should be aware of when using ChatGPT. Discussion/Significance of Impact: No prior study has examined the educational quality of ChatGPT for AML. Our study will detail whether patients are receiving trustworthy and meaningful information, identify misinformation, and provide guidance to oncologists when recommending information resources to patients or fielding questions that patients may raise after using ChatGPT.