A major new meta-analysis reveals that while large language models do boost diagnostic accuracy, the benefit is far smaller than the industry hype suggests.

If artificial intelligence is ready to take over clinical medicine, why are the actual gains so modest? Silicon Valley promises near-perfect diagnostic partners. Yet when real doctors use these tools, the reality is far more complicated.

This disconnect is the real story.

For years, developers have assumed that smarter models automatically make better doctors. This study challenges that assumption, proving that raw technology does not guarantee clinical success. Simply plugging an LLM into a clinic without a clear strategy is a waste of resources.

A systematic review published in medRxiv analyzed data from January 2020 to June 2026. The researchers pooled **42 studies** and **101 effect sizes** comparing physicians working with and without LLM help. They found a small but statistically significant improvement in diagnostic accuracy, with a Hedges g of **0.22** (95% CI [**0.18, 0.27**], p < 0.001).

The modest reality

This minor bump shows that AI is currently a helper, not a savior. The impact is not uniform. The benefits fluctuated wildly depending on the specific medical field and the exact model used.

This inconsistency aligns with findings from a Nature Medicine study on clinical decision-making, which warned that LLM limitations must be actively mitigated before deployment. If a tool works well in dermatology but fails in emergency triage, it is not ready for general use.

Why this matters

This finding matters because health systems are spending millions on software integration. If the return on investment is a mere 0.22 effect size, administrators must rethink their strategy. The bottleneck is no longer the technology itself, but how doctors interact with it.

Earlier research in the Journal of Medical Internet Research showed that AI utility changes across the clinical workflow. Doctors need training to spot AI hallucinations, otherwise, the tool becomes a liability rather than an asset.

Key study findings

• An overall diagnostic improvement of **Hedges g = 0.22**.
• Data pooled from **42 comparative studies** and **101 distinct effect sizes**.
• Highly variable performance across different medical specialties and specific LLM brands.

The study has clear limitations. Most included trials were preprints or conducted in simulated environments, not busy clinics. We still lack robust, real-world evidence on how these tools perform under pressure. Until we understand why AI helps in some cases and fails in others, widespread adoption remains a gamble.

This analysis is based on a study published in medRxiv.