โก Quick Summary
A recent study found that physicians significantly outperform ChatGPT in providing accurate and high-quality responses to real-world pharmacotherapy queries. The findings suggest a strong caution against relying on AI chatbots for pharmacotherapy counselling.
๐ Key Details
- ๐ Study Participants: Three independent evaluators with varying levels of medical expertise (beginner, advanced, expert).
- ๐งฉ Queries Analyzed: 70 real-world pharmacotherapeutic queries.
- โ๏ธ AI Technology: ChatGPT version 3.5.
- ๐ Evaluation Metrics: Quality of information, answer preference, correctness, and language quality.
๐ Key Takeaways
- ๐จโโ๏ธ Physicians’ responses were rated higher in quality compared to those generated by ChatGPT.
- โ ChatGPT produced factually incorrect information more frequently than physicians.
- ๐ฃ๏ธ Language quality was rated higher for physician responses by beginner and expert evaluators.
- โ๏ธ Evaluators’ consensus favored physician-generated responses over AI-generated ones.
- ๐ซ Caution advised against using ChatGPT for pharmacotherapy counselling.
- ๐ Study conducted at Hannover Medical School from June to October 2023.
- ๐ Inter-rater reliability assessed using Krippendorff’s alpha.
๐ Background
The integration of artificial intelligence in healthcare has sparked interest and debate, particularly regarding its role in clinical decision-making. While AI technologies like ChatGPT offer potential benefits in terms of accessibility and speed, their reliability in critical areas such as pharmacotherapy remains questionable. This study aims to clarify the current capabilities of AI in this domain.
๐๏ธ Study
Conducted at Hannover Medical School, the study involved three evaluators with different levels of medical expertise who compared responses from ChatGPT and physicians to 70 pharmacotherapeutic queries. The evaluators assessed the responses based on quality, correctness, and language, providing a comprehensive evaluation of AI’s performance in real-world scenarios.
๐ Results
The results indicated that all evaluators rated the quality of information from physician-generated responses as superior to that of ChatGPT. Furthermore, factually incorrect information was identified more frequently in the AI’s responses. While the beginner and expert evaluators noted a higher quality of language in physician responses, the advanced evaluator did not find a significant difference.
๐ Impact and Implications
The findings of this study highlight the limitations of AI in pharmacotherapy counselling. As healthcare increasingly embraces technology, it is crucial to ensure that AI tools are reliable and accurate, particularly in areas that directly impact patient care. This study serves as a reminder of the irreplaceable value of human expertise in clinical settings.
๐ฎ Conclusion
This study underscores the current inadequacies of AI, specifically ChatGPT, in providing reliable pharmacotherapy counselling. While AI can assist in various healthcare applications, it is essential to approach its use with caution, particularly in critical decision-making areas. Continued research and development are necessary to enhance AI’s capabilities and ensure its safe integration into healthcare practices.
๐ฌ Your comments
What are your thoughts on the role of AI in healthcare? Do you believe it can ever match the expertise of human professionals? Let’s discuss! ๐ฌ Leave your thoughts in the comments below or connect with us on social media:
Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling.
Abstract
AIMS: To assess the utility of the artificial intelligence (AI) chatbot ChatGPT (openly available version 3.5) in responding to real-world pharmacotherapeutic queries from healthcare professionals.
METHODS: Three independent and blinded evaluators with different levels of medical expertise and professional experience (beginner, advanced, and expert) compared AI chatbot- and physician-generated responses to 70 real-world pharmacotherapeutic queries submitted to the clinical-pharmacological drug information centre of Hannover Medical School between June and October 2023 with regard to quality of information, answer preference, answer correctness and quality of language. Inter-rater reliability was assessed with Krippendorff’s alpha. Two separate investigators not otherwise involved in the conduct or analysis of the study selected the top three clinically relevant errors in chatbot- and physician-generated responses.
RESULTS: All three evaluators rated the quality of information of physician-generated responses higher than the quality of information of AI chatbot-generated responses and, accordingly, thought that the physician-generated responses were better than the chatbot-generated responses (answer preference). All evaluators detected factually wrong information more frequently in chatbot-generated responses than in physician-generated responses. Although the beginner and expert evaluators rated the quality of language of physician-generated responses higher than the quality of language of chatbot-generated responses, there was no significant difference according to the advanced evaluator.
CONCLUSIONS: ChatGPT’s responses to real-world pharmacotherapeutic queries were substantially inferior compared to conventional physician-generated responses with regard to quality of information and factual correctness. Our study suggests that to date it must be strongly cautioned against the use of ChatGPT in pharmacotherapy counselling.
Author: [‘Krichevsky B’, ‘Engeli S’, ‘Bode-Bรถger SM’, ‘Koop F’, ‘Schulze Westhoff M’, ‘Schrรถder S’, ‘Schumacher C’, ‘Pape T’, ‘Stichtenoth DO’, ‘Heck J’]
Journal: Br J Clin Pharmacol
Citation: Krichevsky B, et al. Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling. Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling. 2025; (unknown volume):e70321. doi: 10.1002/bcp.70321