P3037 - Use of ChatGPT4, an Online Artificial Intelligence Large Language Model, for Predicting Colonoscopy Screening Intervals

Patrick Chang, MD¹, Maziar Amini, MD², Denis Nguyen, MD³, Helen Lee, NP¹, Jennifer Phan, MD³, James Buxbaum, MD, MS³, Ara B. Sahakian, MD^3
1University of Southern California, Los Angeles, CA; ²Los Angeles General Hospital, University of Southern California, Los Angeles, CA; ³Keck School of Medicine of USC, Los Angeles, CA

Introduction: Artificial Intelligence (AI) Large Language Models (LLM) have potential for interpretation of clinical data. This study evaluated the performance of ChatGPT4, the leading LLM, for determining colon cancer screening intervals, as compared to real-life recommendations made by a clinical endoscopy team.

Methods: Colonoscopy data was prospectively captured from a major safety-net hospital and a large tertiary referral center. ChatGPT version 4 (OpenAI, San Francisco, CA) was used for this study. Text input started with a prompt to determine the screening interval based on AGA 2020 guidelines, followed by procedure, clinical data including family history and pathology reports entered verbatim into the chat model. ChatGPT recommendations were compared to real-life recommendations made by the endoscopist that performed the procedure. The endoscopists included sixteen gastroenterology fellows and three expert attending endoscopists. The ChatGPT and endoscopist recommendations were compared to guideline-directed recommendations made by a panel of two study authors (3^rd year GI fellow and GI attending). Exclusion criteria were IBD, genetic syndromes, advanced therapeutics, inadequate bowel prep, and procedures without screening recommendations. The primary outcome was alignment of endoscopist panel to ChatGPT4 compared to real-life recommendations and secondary outcome was inter-rater reliability between ChatGPT4 and endoscopy panel.

Results: From January to March 2023, 179 colonoscopy procedures were evaluated. Entry into ChatGPT4 was successful in 100% with AI recommendations made in all patients. Guideline-driven endoscopist panel recommendations aligned with ChatGPT4 recommendations in 97.8% of cases, compared to endoscopy team recommendations in 89.4% (McNemar's Test for proportions, p=0.001).

Inter-rater reliability test between endoscopist panel and ChatGPT4 showed strong agreement (Table 1, Overall K = 0.833; 95%CI: 0.748-0.917, p< 0.001). Recommendations for 3-year follow up had substantial agreement (0.637) (0.61-0.8) and recommendations for 5, 7-10, and 10 year intervals had near-perfect agreement (0.914-955).

Discussion: Initial real-world results suggest that pre-trained LLM, such as ChatGPT4, provide strong alignment with consensus guidelines for interval screening colonoscopy follow up when properly prompted. This presents among the first data evaluating the utility of LLM for this application. Further study in a larger cohort is needed.


Disclosures:


Patrick Chang, MD¹, Maziar Amini, MD², Denis Nguyen, MD³, Helen Lee, NP¹, Jennifer Phan, MD³, James Buxbaum, MD, MS³, Ara B. Sahakian, MD³. P3037 - Use of ChatGPT4, an Online Artificial Intelligence Large Language Model, for Predicting Colonoscopy Screening Intervals, ACG 2023 Annual Scientific Meeting Abstracts. Vancouver, BC, Canada: American College of Gastroenterology.