A new double-blind trial reveals that specialized medical AI still cannot match human doctors when prescribing antibiotics for complex hospital infections.

Can we trust an AI to pick the right antibiotic when a patient’s life is on the line? Many tech advocates argue that general-purpose models can handle specialized medical tasks if they are prompted correctly. This new study complicates that narrative. It shows that while reasoning techniques help, domain-specific training remains mandatory for high-stakes clinical decisions.

The study used a double-blind, randomized-sequence evaluation with a 2X2 factorial design. Researchers compared a domain-specific model called MedGo against a general-purpose model, DeepSeek V3.5. Both models were tested under standard direct prompting and chain-of-thought (CoT) prompting. They generated five parallel regimens for 59 complex inpatient infection cases, which three senior clinicians graded on a 1-to-5 scale.

How the models ranked

The clinical grades revealed a clear hierarchy in decision-making quality. Human expertise still reigns supreme, but the way the models fell in line tells a deeper story about AI development.

• Real physicians scored the highest, followed by MedGo-CoT, DeepSeek-CoT, MedGo, and base DeepSeek.
• In base mode, the medical model MedGo significantly outperformed DeepSeek with an adjusted p-value of 0.040.
• Chain-of-thought prompting improved both models, but MedGo-CoT significantly beat DeepSeek-CoT in individualized adjustment (p < 0.001) and dosing precision (p = 0.005).

This hierarchy proves that prompting is not a substitute for specialized training. While chain-of-thought reasoning reduced score dispersion and helped both models, it could not close the gap between general and medical AI. General-purpose models lack the clinical nuance required for precise dosing.

The automated evaluation trap

The researchers also tested ChatGPT 5.2 as an automated judge to see if AI could grade these prescriptions. The results were terrible. The correlation between ChatGPT’s grades and the human experts was negligible, showing an overall Kendall tau of just 0.153.

This is the most critical warning in the study. The industry is currently rushing to use LLMs to evaluate other LLMs to save time and money. This data suggests that automated clinical evaluation is highly unreliable. If an AI evaluator cannot align with senior clinicians, we cannot trust it to benchmark medical software.

We must also look at what the AI missed. Even the best-performing model, MedGo-CoT, showed notable deficiencies in antimicrobial stewardship ecological awareness. The AI struggled to weigh the broader environmental threat of drug resistance when treating individual patients.

This study is limited by its small sample of 59 cases and its focus on paper-based decisions rather than live clinical trials. However, the takeaway is clear. For now, senior clinical expertise remains entirely indispensable.

Read the full preprint in medRxiv.