P1207 - Comparing Large Language Models Accuracy in Following Interval Surveillance Colonoscopy Guidelines

Sunday, October 22, 2023

3:30 PM – 7:00 PM PT

Location: Exhibit Hall

Has Audio

Presenting Author(s)

OO

Olufemi E. Osikoya, MD

Medical City Arlington
Arlington, Texas

Gregory Brennan, MD¹, Olufemi E. Osikoya, MD²
¹Texas Digestive Disease Consultants, Mansfield, TX; ²Medical City Arlington, Arlington, TX

Introduction: Providing pathology results and appropriate recommendations after resection of colon polyps is mandatory. Large language models (LLMs) such as ChatGPT and Google Bard, have shown promise in clinical workflows such as pathology results letters. We tested whether LLMs could provide appropriate surveillance recommendations based on current guidelines from the US multi-society task force for post-colonoscopy follow-up.

Methods: Our aim was to compare the accuracy of ChatGPT 3.5, ChatGPT 4, and Google Bard in providing appropriate interval surveillance recommendations. An example prompt being “Write a patient pathology result letter after a colonoscopy with one tubular adenoma polyp (< 10mm) resected. Include recommendations for when the next surveillance colonoscopy should be completed.” Seventeen different post polypectomy surveillance queries and responses were analyzed (correct, partially correct, incorrect) compared to USMSTF guidelines.

Results: When comparing the LLMs, Goggle Bard performed the best (TABLE 1). Google Bard provided the most correct responses and the least incorrect responses. Bard provided correct recommendations in 76% of queries (13/17), partially correct recommendations in 18% of queries (3/17) and incorrect recommendations in 6% of queries (1/17). ChatGPT 4 performed better than ChatGPT 3.5. ChatGPT 4 provided correct recommendations in 70% of queries (12/17), partially correct recommendations in 24% of queries (4/17) and incorrect recommendations in 6% of queries (1/17). ChatGPT 3.5 provided the most incorrect recommendations at 24% (4/17). Finally, all LLMs generated readable appropriate pathology results letters with complete information and recommendations. Bard letters were shorter than ChatGPT letters.

Discussion: The LLMs analyzed here provided different recommendations to the 17 post polypectomy surveillance scenarios. Google Bard provided the most correct recommendations. These differences can be explained by Google Bards connectivity to the internet, which ChatGPT lacks. ChatGPT 4 is an updated version and was superior to 3.5. Interestingly, Bard and ChatGPT directly referenced the USMSTF guidelines. ChatGPT 3.5 did not reference any sources. ChatGPT 4 also occasionally referenced the British Society of Gastroenterology (BSG) and European Society of Gastrointestinal Endoscopy (ESGE). Overall, partially correct recommendations were common in all LLMs. Using LLMs shows promise but, their current accuracy limits real world adoption.

Colonoscopy Finding	USMSTF Interval Surveillance Colonoscopy Recommendation	ChatGPT 3.5	ChatGPT 4	Google Bard
1 tubular adenoma < 10 mm	7- 10 years	3-5 years	5-10 years	7-10 years
3 tubular adenomas < 10 mm	3- 5 years	3 years	3 years	3-5 years
5 tubular adenomas < 10 mm	3 years	1-3 years	3 years	3 years
>10 tubular adenomas	1 year	1 year	1 year	3 years
One or more Adenoma > 10 mm	3 years	3 years	3 years	3 years
Adenoma with tubulovillous or villous histology	3 years	1-3 years	3 years	3 years
Adenoma with high-grade dysplasia	3 years	3-6 months	3 years	3 years
Piecemeal resection of adenoma > 20 mm	6 months	3-6 months	6 months	6 months
10 hyperplastic polyps < 10 mm (rectum or sigmoid)	10 years	5 years	3-5 years	10 years
1 SSP < 10 mm	5-10 years	5-10 years	5-10 years	7-10 years
3–4 SSPs < 10 mm	3-5 years	3 years	3 years	3-5 years
5–10 SSPs < 10 mm	3 years	1-3 years	3 years	3 years
SSP > 10 mm	3 years	3 years	3 years	3 years
SSP with dysplasia	3 years	6-12 months	3 years	3 years
Hyperplastic polyp > 10 mm	3-5 years	5 years	3-5 years	5 years
Traditional serrated adenoma	3 years	3-5 years	3 years	3-5 years
Piecemeal resection of SSP >20 mm	6 months	6-12 months	2-6 months	6 months

Table: Table 1. Comparison of accuracy of large language models in generating appropriate interval surveillance colonoscopy recommendations. Green= correct recommendation. Orange = partially correct recommendation. Red= incorrect recommendation.

Disclosures:

Gregory Brennan indicated no relevant financial relationships.

Olufemi Osikoya indicated no relevant financial relationships.

Gregory Brennan, MD¹, Olufemi E. Osikoya, MD². P1207 - Comparing Large Language Models Accuracy in Following Interval Surveillance Colonoscopy Guidelines, ACG 2023 Annual Scientific Meeting Abstracts. Vancouver, BC, Canada: American College of Gastroenterology.