376 Using a large language model to create lay summaries of clinical study descriptions

Abstract

Objectives/Goals: We assessed the feasibility of using a large language model (LLM) to create lay language descriptions of study protocols for recruitment, which has the potential to improve accessibility and transparency of clinical studies and enable participants to make informed decisions. Methods/Study Population: All studies from a clinical research recruitment platform were included, which features human-written lay descriptions and titles for study recruitment. Corresponding protocol summaries in the IRB system were extracted and translated into lay language using a LLM (gpt-35-turbo-0613). A subset was used to develop prompt variations through an iterative process. Prompt strategies evaluated include chain-of-thought and few-shot prompting techniques. LLM-generated and human-written descriptions were compared for readability using Flesch–Kincaid and Simple Measure of Gobbledygook (SMOG) reading grade levels and information completeness using Word Movers’ Distance (WMD). Results/Anticipated Results: A total of 55 study descriptions were included – 10 were used to develop prompts and 45 were used for evaluation. The final LLM instructions included multistep prompts. The LLM was first instructed to produce a two- to three-sentence long description without using scientific jargon and included two pairs of examples. The LLM was then asked to shorten the description and finally to provide an engaging title. LLM-generated and human-written summaries were similar in length (median (IQR) 328 (278.5–360.5) vs. 342 (203–532.5) characters, respectively). LLM-generated summaries had lower Flesch-Kincaid grade level (5.15 vs. 8.28, p Discussion/Significance of Impact: An LLM can be used to generate lay language summaries that are readable at a lower grade level while maintaining semantic similarity. This approach can be used to improve the drafting of summaries for recruitment, thereby improving accessibility to potential participants. Future work includes human evaluation and implementation into practice.