P1224 - Reliability Concerns: Can AI Interpret Nuanced Medical and Ethical Scenarios in the Field of Gastroenterology

Rajmohan Rammohan, MD, Dilman Natt, MD, Sai Greeshma Magam, MD, Melvin Joy, MD, Tulika Saggar, MD, Achal Patel, MD, Abhishek Tadikonda, MD, Jiten Desai, MD, Paul Mustacchia, MD, MBA
Nassau University Medical Center, East Meadow, NY

Introduction: AI tools like ChatGPT and Google Bard are gaining traction in healthcare, notably gastroenterology, offering benefits such as vast data knowledge, swift responses, and easy access. However, their reliability in medical and ethical decision-making is uncertain. They provide information effectively but cannot fully emulate the nuanced understanding and empathy of human medical professionals. Especially in ethical decisions, which demand comprehension of personal and contextual factors, these AI tools should serve as adjuncts, not replacements, to expert human judgment.

Methods: The study evaluated the medical and ethical dependability of two widely-used chatbots, ChatGPT, and Google BARD, within the gastroenterology sphere. A questionnaire was administered to both bots, with their responses being rated using a 1-10 Likert scale where 1 indicated exceptional accuracy. To ensure unbiased evaluation, two independent assessors analyzed each bot's answers. The goal was to systematically evaluate the chatbots' competencies and trustworthiness using this performance review. The involvement of dual evaluators and the application of the Likert scale aimed to mitigate any potential bias, therefore strengthening the validity of the findings.

Results: Our study compared the dependability of ChatGPT and Google BARD in medical management scenarios. ChatGPT scored 21% (p < 0.01), and Google BARD scored 19% (p=0.022) in terms of reliability when juxtaposed with standardized practices. Among the chatbots, ChatGPT had a higher score relative to Google BARD (67% vs. 41%, p=0.034). However, both chatbots' reliability scores were inferior compared to standard practice. This underscores the importance of reliability in developing gastroenterology-focused chatbots and the need for ongoing research and improvements in this field.

Discussion: Despite potential benefits, AI tools like ChatGPT and Google Bard currently fall short in assisting medical and ethical decisions in gastroenterology, as shown by lower reliability scores against standardized guidelines. Although ChatGPT marginally outperformed Google Bard, both fail to match the nuanced understanding and empathy of human healthcare professionals. This underlines the crucial need for AI dependability and the importance of ongoing research to enhance these technologies, ensuring they support, not supplant, human judgment in decision-making.




Disclosures:


Rajmohan Rammohan, MD, Dilman Natt, MD, Sai Greeshma Magam, MD, Melvin Joy, MD, Tulika Saggar, MD, Achal Patel, MD, Abhishek Tadikonda, MD, Jiten Desai, MD, Paul Mustacchia, MD, MBA. P1224 - Reliability Concerns: Can AI Interpret Nuanced Medical and Ethical Scenarios in the Field of Gastroenterology, ACG 2023 Annual Scientific Meeting Abstracts. Vancouver, BC, Canada: American College of Gastroenterology.