Yusuf Yesil, MD, MIS 
Hospitals are eager to adopt locally hosted AI models to avoid sharing patient data with commercial cloud vendors. A new multicenter trial shows that open-source software can match top commercial systems on medical analysis. Yet the same open systems break down almost completely when pushed to generate medical misinformation.
That safety gap poses a severe operational risk for health systems considering on-premise deployment. While open models lower cloud expenses, they may expose hospitals to clinical errors and malicious prompt manipulation.
Diagnostic Power Meets Guardrail Failure
Researchers tested 7 model variants using 50 real-world patient records from 10 Chinese hospitals across 14 clinical specialties. The evaluation compared open-source models like DeepSeek-R1 and DeepSeek-V3 against commercial options including GPT-4o and o3-mini.
For diagnostic accuracy across 47 valid admission notes, reasoning-focused models took the lead. Both DeepSeek-R1 and Microsoft Copilot’s GPT-4o-reasoning correctly diagnosed 64% (30/47) of complex cases. That performance mirrors broader findings on Large Language Model Influence on Diagnostic Reasoning, where reasoning architectures consistently outperform basic foundation models.
The models also parsed discharge summaries with high precision across extraction, summarization, and inference tasks:
- Extraction accuracy: DeepSeek-R1 and GPT-4o-reasoning tied at 97.7% (293/300).
- Summarization accuracy: GPT-4o-reasoning reached 87.5% (175/200), while DeepSeek-R1 scored 83.5% (167/200).
- Inference accuracy: GPT-4o-reasoning hit 87%, slightly ahead of DeepSeek-R1 at 85%.
- Hallucination rates: Flawed outputs ranged from 38.6% (27/70) in GPT-4o-reasoning to 73% (73/100) in DeepSeek-V3.
Adversarial Attacks Expose Safety Gaps
The clear divide emerged during security testing. Researchers subjected each model to 4 rounds of simulated adversarial prompt attacks designed to extract medical false information.
DeepSeek-R1 failed to defend itself, generating medical disinformation in 98% (49/50) of test scenarios and offering a refusal rate of just 2%. GPT-4o-reasoning posted the strongest defense, but still surrendered to malicious prompts in 70% (35/50) of cases with a 30% refusal rate. These findings reinforce work on the evaluation and mitigation of the limitations of large language models in clinical decision-making, proving that benchmark accuracy does not equal safety in production.
Retrospective accuracy on real records shows that open-source models possess genuine clinical utility. However, a model that yields to adversarial prompts in 98% of tests cannot run unsupervised in a clinic. Hospital IT teams must build external validation layers and guardrails before letting locally deployed models interact with patient records or clinical staff.
Read the full peer-reviewed study in the Journal of Medical Internet Research.
