Overview
A clinical artificial intelligence tool developed by researchers at the University at Buffalo has shown exceptional performance on the United States Medical Licensing Exam (USMLE), outperforming both other AI tools and many physicians. This finding was published in JAMA Network Open.
Key Findings
- The tool, named Semantic Clinical Artificial Intelligence (SCAI), achieved a score of 95.2% on Step 3 of the USMLE.
- In comparison, the GPT-4 Omni tool scored 90.5% on the same exam.
- SCAI is designed to enhance clinical decision-making by providing reasoned responses based on extensive medical knowledge.
Research Insights
Lead author Peter L. Elkin, MD, chair of the Department of Biomedical Informatics at UB, emphasized that SCAI is not just a tool but a partner in clinical reasoning:
“As physicians, we are used to using computers as tools, but SCAI is different; it can add to your decision-making and thinking based on its own reasoning.”
Methodology
The researchers evaluated SCAI against the USMLE, which tests a physician’s ability to apply medical knowledge and demonstrate patient-centered skills. The model was trained using:
- Natural language processing software.
- Authoritative clinical information from diverse sources, excluding biased data.
- Semantic triples to create networks that allow logical inferences.
Technological Advancements
Elkin noted that SCAI employs advanced techniques such as:
- Knowledge graphs to identify new connections in medical data.
- Retrieval-augmented generation to access external knowledge databases, reducing inaccuracies.
- Formal semantics to provide context for more accurate responses.
Future Implications
Elkin believes that SCAI has the potential to:
- Enhance patient safety.
- Improve access to care.
- Democratize specialty care by making information accessible to primary care providers and patients.
However, he stresses that AI is meant to augment, not replace, the role of physicians:
“Artificial intelligence isn’t going to replace doctors, but a doctor who uses AI may replace a doctor who does not.”
Reference
Elkin PL, Mehta G, LeHouillier F, Resnick M, Mullin S, Tomlin C, Resendez S, Liu J, Nebeker JR, Brown SH. Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE. JAMA Netw Open. 2025 Apr 1;8(4):e256359. doi: 10.1001/jamanetworkopen.2025.6359