P0751 - Accuracy of a Chat-Vased Artificial Intelligence Model in Answering Frequently Asked Questions and Providing References Related to Inflammatory Bowel Disease

Sunday, October 22, 2023

3:30 PM – 7:00 PM PT

Location: Exhibit Hall

Has Audio

Presenting Author(s)

Anthony Kerbage, MD

Cleveland Clinic
Cleveland, OH

Anthony Kerbage, MD¹, Jean-Paul Achkar, MD, FACG¹, Ruthvik Padival, MD¹, Joseph Kassab, MD², Joseph El Dahdah, MD¹, Carol Rouphael, MD¹
¹Cleveland Clinic, Cleveland, OH; ²Cleveland Clinic Foundation, Cleveland, OH

Introduction: With the rise of easily accessible Artificial Intelligence (AI)chatbots like ChatGPT, patients now have novel ways to search for health-related information. However, their accuracy in answering disease-specific questions is unexplored. We sought to assess the accuracy of ChatGPT-4 in providing appropriate responses and references to frequently asked IBD questions.

Methods: Using Google Trends, an online tool that tracks and analyzes the popularity of specific search queries, we formulated 20 questions based on frequently searched terms related to IBD over the past 5 years. The questions were entered into ChatGPT-4 on 5/5/2023, and the model was asked to provide 2 references for each response. Three gastroenterologists independently reviewed the answers and references for accuracy. They were asked to provide a detailed grade (100% accurate, 100% inaccurate, partly inaccurate, accurate with missing information) and overall grade (accurate, inaccurate). References were graded as suitable, unsuitable (existent but unrelated to answer) or nonexistent. When references for the same answer had different grading, the overall grade was given according to the least accurate reference (nonexistent >unsuitable >suitable). Answers with disagreement were discussed by the reviewers, permitting potential adjustments post-discussion. Agreement level was computed using free-marginal Kappa coefficients (κ).

Results: There was 100% agreement for the overall grade (accurate or inaccurate; κ=1) and almost perfect agreement for the detailed grade (86.7%, κ=0.82, 95%CI [0.66, 0.98]) following consensus discussion. On average, 55% of ChatGPT-4 responses were rated as 100% accurate, 40% partly inaccurate and 5% accurate with missing information on detailed assessment. Overall, the accuracy level of chatGPT-4 responses was 75% (Table 1). In terms of references, 15% of questions had at least one reference that was unsuitable and 50% of questions had at least one reference that was non-existent. An example is presented in Figure 1.

Discussion: This study demonstrates the potential role of ChatGPT-4 as a healthcare tool addressing online IBD-related patient queries. Although the model's accuracy was not perfect, an overall accuracy level of 75% is acceptable, indicating a promising potential. However, before contemplating the possibility of a comprehensive integration of ChatGPT-4 into healthcare, we must recognize the necessity for improvement, particularly in providing suitable references.

Figure: Figure 1. Example of a question entered into ChatGPT-4 and assessment of the answer’s accuracy and references’ suitability by one gastroenterologist reviewer

	Question	Overall accuracy	References
1	How is IBD diagnosed? Provide two references.	Accurate	Unsuitable
2	What is the difference between Crohn’s disease and ulcerative colitis? Provide two references.	Inaccurate	Unsuitable
3	What are the treatment options for IBD? Provide two references.	Accurate	Nonexistent
4	How can I manage fatigue associated with IBD? Provide two references.	Accurate	Nonexistent
5	What is the impact of IBD on mental health? Provide two references.	Accurate	Suitable
6	Can IBD affect fertility or pregnancy? Provide two references.	Inaccurate	Unsuitable
7	How can I explain IBD to friends and family? Provide two references.	Accurate	Suitable
8	What is the link between IBD and arthritis? Provide two references.	Accurate	Suitable
9	Can probiotics help manage IBD symptoms? Provide two references.	Accurate	Suitable
10	What are the surgical options for treating IBD? Provide two references.	Accurate	Suitable
11	How can I find support and resources for living with IBD? Provide two references.	Accurate	Suitable
12	What is the difference between IBD and Irritable Bowel Syndrome (IBS)? Provide two references.	Inaccurate	Suitable
13	What is the best diet for IBD? Provide two references.	Inaccurate	Nonexistent
14	What are the causes of IBD? Provide two references.	Accurate	Nonexistent
15	What is calprotectin? Provide two references.	Accurate	Nonexistent
16	How to recognize an IBD flare-up? Provide two references.	Accurate	Nonexistent
17	What is ileitis? Provide two references.	Accurate	Nonexistent
18	Do cigarettes help with IBD? Provide two references.	Inaccurate	Nonexistent
19	I was recommended a biologic for IBD. Will this lead to infections and cancer? Provide two references.	Accurate	Nonexistent
20	Which biologic is the best to treat ulcerative colitis? Provide two references.	Accurate	Nonexistent

Table: Table 1. List of the 20 IBD-related questions entered into ChatGPT-4 and grading of the responses’ overall accuracy and references suitability.

Disclosures:

Anthony Kerbage indicated no relevant financial relationships.

Jean-Paul Achkar indicated no relevant financial relationships.

Ruthvik Padival indicated no relevant financial relationships.

Joseph Kassab indicated no relevant financial relationships.

Joseph El Dahdah indicated no relevant financial relationships.

Carol Rouphael indicated no relevant financial relationships.

Anthony Kerbage, MD¹, Jean-Paul Achkar, MD, FACG¹, Ruthvik Padival, MD¹, Joseph Kassab, MD², Joseph El Dahdah, MD¹, Carol Rouphael, MD¹. P0751 - Accuracy of a Chat-Vased Artificial Intelligence Model in Answering Frequently Asked Questions and Providing References Related to Inflammatory Bowel Disease, ACG 2023 Annual Scientific Meeting Abstracts. Vancouver, BC, Canada: American College of Gastroenterology.