⚡ Quick Summary
This study evaluated the performance of chatbots in conducting systematic reviews (SR) in pediatric dentistry, comparing their effectiveness to that of non-expert reviewers and experts. The findings suggest that while chatbots can enhance efficiency, human oversight remains essential for accuracy and completeness.
🔍 Key Details
- 🤖 Chatbots tested: ChatGPT4 and Gemini
- 👥 Reviewers: Two non-expert reviewers and two expert reviewers
- 📋 Tasks assessed: PICO question formulation, search query development, study screening, data extraction, and risk of bias assessment
- 📊 Performance metrics: Accuracy, precision, sensitivity, specificity, F1-score, Cohen’s Kappa, and Global Quality Score
🔑 Key Takeaways
- 📈 ChatGPT slightly outperformed Gemini in PICO formulation.
- 📚 Experts identified 1,261 records, while ChatGPT identified 569, Gemini 285, and non-experts 722.
- 🔍 Screening results: Chatbots had 90% sensitivity but less than 25% precision.
- 📊 F1-scores: Chatbots scored <40%, while non-experts achieved 39%.
- 📊 Data extraction scores: ChatGPT (31.6), Gemini (29.2), non-experts (30.4).
- 🤝 Risk of bias agreement: ChatGPT (49.4%), Gemini (51.2%), non-experts (48.8%).
- ⚠️ Human oversight is crucial for ensuring accuracy in systematic reviews.
- 🌟 Chatbots could significantly enhance the efficiency of systematic reviews.
📚 Background
Conducting systematic reviews is a vital process in evidence-based medicine, particularly in fields like pediatric dentistry. However, the traditional methods can be time-consuming and labor-intensive. The integration of artificial intelligence (AI) and chatbots into this process presents an exciting opportunity to streamline workflows and improve efficiency.
🗒️ Study
This study aimed to evaluate the performance of two chatbots, ChatGPT4 and Gemini, in performing discrete steps of a systematic review on AI in pediatric dentistry. The researchers compared the chatbots’ performance to that of non-expert reviewers and expert reviewers across five critical tasks, providing a comprehensive analysis of their capabilities.
📈 Results
The results indicated that ChatGPT slightly outperformed Gemini in PICO question formulation. In terms of study identification, experts identified the most records, while the chatbots lagged behind. Notably, the chatbots demonstrated high sensitivity in screening but struggled with precision, achieving less than 25%. The data extraction scores were relatively close among all groups, but human reviewers still maintained a slight edge in overall performance.
🌍 Impact and Implications
The findings of this study highlight the potential for chatbots to enhance the efficiency of systematic reviews, particularly in the screening and data extraction phases. However, the necessity for human oversight cannot be overstated, as it ensures the accuracy and completeness of the review process. This balance between AI and human expertise could pave the way for more efficient research methodologies in pediatric dentistry and beyond.
🔮 Conclusion
This study underscores the promising role of chatbots in conducting systematic reviews, particularly in pediatric dentistry. While they can significantly enhance efficiency, the importance of human oversight remains critical for maintaining the quality and reliability of the review process. Future research should continue to explore the integration of AI technologies in systematic reviews to further improve healthcare outcomes.
💬 Your comments
What are your thoughts on the use of chatbots in systematic reviews? Do you believe they can effectively complement human reviewers? Let’s discuss! 💬 Leave your thoughts in the comments below or connect with us on social media:
Chatbots for Conducting Systematic Reviews in Pediatric Dentistry.
Abstract
OBJECTIVES: The performance of chatbots for discrete steps of a systematic review (SR) on artificial intelligence (AI) in pediatric dentistry was evaluated.
METHODS: Two chatbots (ChatGPT4/Gemini) and two non-expert reviewers were compared against two experts in a SR on AI in pediatric dentistry. Five tasks: (1) formulating a PICO question, (2) developing search queries for eight databases, (3) screening studies, (4) extracting data, and (5) assessing the risk of bias (RoB) were assessed. Chatbots and non-experts received identical prompts, with experts providing the reference standard. Performance was measured using accuracy, precision, sensitivity, specificity, and F1-score for search and screening tasks, Cohen’s Kappa for risk of bias assessment, and a modified Global Quality Score (1-5) for PICO question formulation and data extraction quality. Statistical comparisons were performed using Kruskal-Wallis and Dunn’s post-hoc tests.
RESULTS: In PICO formulation, ChatGPT outperformed Gemini slightly, while non-experts scored the lowest. Experts identified 1,261 records, compared to 569 (ChatGPT), 285 (Gemini), and 722 (non-experts). Screening showed chatbots having 90% sensitivity, >60% specificity, <25% precision, and F1-scores <40%, versus non-experts' 84% sensitivity, 91% specificity, and 39% F1-score, respectively. For data extraction, ChatGPT yielded a (mean±standard deviation) score of 31.6±12.3 (max. was 45), Gemini 29.2 ±12.3, and non-experts 30.4 ±11.3, respectively. For RoB, the agreement with experts was 49.4% for ChatGPT, 51.2% for Gemini 48.8% for non-experts (p>0.05).
CONCLUSION: Chatbots could enhance SR efficiency, particularly for the study screening and data extraction steps. Human oversight remains critical for ensuring accuracy and completeness.
Author: [‘Rokhshad R’, ‘Mohammad FD’, ‘Nomani M’, ‘Mohammad-Rahimi H’, ‘Schwendicke F’]
Journal: J Dent
Citation: Rokhshad R, et al. Chatbots for Conducting Systematic Reviews in Pediatric Dentistry. Chatbots for Conducting Systematic Reviews in Pediatric Dentistry. 2025; (unknown volume):105733. doi: 10.1016/j.jdent.2025.105733