Yusuf Yesil, MD, MIS 
Medical literature usually relies on peer review to weed out bad data. When large language models generate realistic scientific text, clinicians can no longer rely on superficial quality to spot errors. A new trial shows that well-structured abstract prose easily convinces doctors to act on complete fabrications.
That vulnerability challenges the trust clinicians place in automated summaries. Recent analysis on Why we need to be careful with LLMs in medicine stresses that readable text often hides deep inaccuracies. When speed matters in acute care, polished syntax becomes a dangerous proxy for truth.
Flawed signals in acute care
Researchers surveyed 20 physicians in emergency medicine and trauma surgery across 4 German hospitals. Each doctor evaluated 4 medical abstracts generated by GPT-5 Thinking mode, yielding 80 total ratings. Half of the abstracts contained accurate claims, while the other half contained intentionally false claims backed by plausible explanations.
Physicians reported an intent to act on 68% (54/80) of the abstracts they reviewed. When evaluated as a diagnostic test for catching fake science, the doctors performed poorly:
- Sensitivity for acting on valid evidence was 78% (31/40).
- Specificity against acting on false claims was just 43% (17/40).
- Positive predictive value reached 57% (31/54), while negative predictive value was 65% (17/26).
- The positive likelihood ratio was 1.35 (95% CI 0.98-1.85) and the diagnostic odds ratio was 2.55 (95% CI 0.96-6.73).
The numbers reveal a striking lack of discrimination. Doctors fell for false abstracts almost as often as they accepted true ones.
Polished prose hides false data
Physicians could not tell true and false abstracts apart using standard criteria. True and false texts received nearly identical scores for perceived accuracy (3.15 vs 2.88, P=.21), precision (3.15 vs 2.93, P=.29), structure (3.23 vs 3.20, P=.89), and grammar (3.40 vs 3.43, P=.86).
Instead, intent to act was driven entirely by how persuasive the abstract felt. Abstracts that doctors chose to act on scored significantly higher in perceived accuracy, precision, and structure (all P<.001). Grammar quality made no difference (P=.80).
This finding reinforces warnings about Evaluating ChatGPT in Medical Contexts and guarding against hallucinations. Flawless structure creates a false sense of security that overrides clinical skepticism.
Real-world limits and takeaways
This study evaluated intent in a vignette survey, not real patient care at the bedside. With 20 physicians and 80 evaluations, the small sample size limits broad conclusions across other specialties. Self-reported decisions on paper do not always mirror emergency room behavior.
Yet the core risk is clear. Hospitals cannot rely on individual clinician vigilance to catch fake AI output. Clinical workflows must require source verification against raw study data before synthetic summaries inform patient treatment.
Read the full study in the Journal of Medical Internet Research.
