โก Quick Summary
This study evaluated the performance of retrieval-augmented generation (RAG) large language models (LLMs) for infective endocarditis (IE) prophylaxis in dental procedures, revealing that RAG models significantly improved accuracy compared to non-RAG models. The findings highlight the potential of LLMs as clinical decision support tools, although caution is advised in their real-world application.
๐ Key Details
- ๐ Question Set: Established IE prophylaxis questions from previous research
- ๐งฉ Models Tested: Ten LLMs integrated with RAG using MiniLM L6 v2 embeddings and FAISS
- โ๏ธ Evaluation Method: Five independent runs with and without preprompting
- ๐ Best Performing Model: Grok 3 beta with 90.0% accuracy with preprompting
๐ Key Takeaways
- ๐ RAG models showed improved accuracy in IE prophylaxis queries.
- ๐ก Preprompting enhanced performance across all tested LLMs.
- ๐ DeepSeek Reasoner achieved the highest mean accuracy of 83.6% without preprompting.
- โ ๏ธ Claude 3.7 Sonnet had the lowest accuracy, highlighting variability among models.
- โฑ๏ธ Task time increased for both undergraduate and postgraduate students when using LLM support.
- ๐ Clinical relevance emphasizes the need for digital literacy among clinicians and students.
- ๐ Study published in the International Dental Journal, 2025.
๐ Background
The integration of large language models (LLMs) in healthcare is a growing trend, particularly in providing rapid responses to clinical queries. However, traditional LLMs often struggle with accuracy and relevance due to their reliance on static datasets. The introduction of retrieval-augmented generation (RAG) aims to address these limitations by grounding responses in up-to-date, domain-specific information, making them more applicable in clinical settings.
๐๏ธ Study
This study utilized an established question set focused on IE prophylaxis to evaluate the effectiveness of RAG-augmented LLMs. A total of ten models were tested, with a specific focus on their performance with and without preprompting techniques. Additionally, a pilot study involving ten dental students assessed the utility of the best-performing LLM as a clinical decision support tool.
๐ Results
The results indicated that DeepSeek Reasoner achieved the highest mean accuracy of 83.6% without preprompting, while Grok 3 beta reached an impressive 90.0% accuracy with preprompting. In contrast, Claude 3.7 Sonnet demonstrated the lowest accuracy, with 42.1% without preprompts and 47.1% with them. Overall, preprompting was found to enhance the performance of all models tested.
๐ Impact and Implications
The findings from this study suggest that RAG-augmented LLMs can provide rapid and accessible support for clinical decision-making in dental procedures related to IE prophylaxis. However, the variability in accuracy among different models underscores the importance of maintaining professional judgment and digital literacy when utilizing these tools in practice. As LLM technology continues to evolve, its integration into clinical workflows could significantly enhance patient care.
๐ฎ Conclusion
This study highlights the potential of RAG-augmented LLMs in improving clinical accuracy for IE prophylaxis queries. While the results are promising, it is essential for healthcare professionals to approach these tools with caution, ensuring that their outputs are critically evaluated against current medical knowledge. Continued research and development in this area could lead to more effective applications of AI in healthcare, ultimately benefiting patient outcomes.
๐ฌ Your comments
What are your thoughts on the use of LLMs in clinical decision-making? Do you see potential challenges or benefits in their application? ๐ฌ Join the conversation in the comments below or connect with us on social media:
Evaluating Retrieval-Augmented Generation-Large Language Models for Infective Endocarditis Prophylaxis: Clinical Accuracy and Efficiency.
Abstract
INTRODUCTION AND AIMS: The use of large language models (LLMs) in healthcare is expanding. Retrieval-augmented generation (RAG) addresses key LLM limitations by grounding responses in domain-specific, up-to-date information. This study evaluated RAG-augmented LLMs for infective endocarditis (IE) prophylaxis in dental procedures, comparing their performance with non-RAG models assessed in our previous publication using the same question set. A pilot study also explored the utility of an LLM as a clinical decision support tool.
METHODS: An established IE prophylaxis question set from previous research was used to ensure comparability. Ten LLMs integrated with RAG were tested using MiniLM L6 v2 embeddings and FAISS to retrieve relevant content from the 2021 American Heart Association IE guideline. Models were evaluated across five independent runs, with and without a preprompt (‘You are an experienced dentist’), a prompt-engineering technique used in previous research to improve LLMs accuracy. Three RAG-LLMs were compared to their native (non-RAG) counterparts benchmarked in the previous study. In the pilot study, 10 dental students (5 undergraduate, 5 postgraduate in oral and maxillofacial surgery) completed the questionnaire unaided, then again with assistance from the best performing LLM. Accuracy and task time were measured.
RESULTS: DeepSeek Reasoner achieved the highest mean accuracy (83.6%) without preprompting, while Grok 3 beta reached 90.0% with preprompting. The lowest accuracy was observed for Claude 3.7 Sonnet, at 42.1% without preprompts and 47.1% with preprompts. Preprompting improved performance across all LLMs. RAG’s impact on accuracy varied by model. Claude 3.7 Sonnet showed the highest response consistency without preprompting; with preprompting, Claude 3.5 Sonnet and DeepSeek Reasoner matched its performance. DeepSeek Reasoner also had the slowest response time. In the pilot study, LLM support slightly improved postgraduate accuracy, slightly reduced undergraduate accuracy, and significantly increased task time for both.
CONCLUSION: While RAG and prompting enhance LLM performance, real-world utility in education remains limited.
CLINICAL RELEVANCE: LLMs with RAG provide rapid and accessible support for clinical decision-making. Nonetheless, their outputs are not always accurate and may not fully reflect evolving medical and dental knowledge. It is crucial that clinicians and students approach these tools with digital literacy and caution, ensuring that professional judgment remains central.
Author: [‘Rewthamrongsris P’, ‘Thongchotchat V’, ‘Burapacheep J’, ‘Trachoo V’, ‘Khurshid Z’, ‘Porntaveetus T’]
Journal: Int Dent J
Citation: Rewthamrongsris P, et al. Evaluating Retrieval-Augmented Generation-Large Language Models for Infective Endocarditis Prophylaxis: Clinical Accuracy and Efficiency. Evaluating Retrieval-Augmented Generation-Large Language Models for Infective Endocarditis Prophylaxis: Clinical Accuracy and Efficiency. 2025; 76:109344. doi: 10.1016/j.identj.2025.109344