A general-purpose AI model caught internal text errors in radiology reports almost ten times faster than human radiologists while outperforming them on recall.
Manual peer review of radiology reports rarely scales to meet rising imaging volume. That gap allows small typographical or factual slips to slip into permanent records. In a study published in the Journal of Medical Internet Research, researchers demonstrated that large language models can act as automated safety nets to flag internal text errors before they reach patients.
The trial analyzed 95,582 retrospective radiology reports across X-ray, CT, and MRI scans from five anatomical regions. Human annotators found that 4.6% of these reports contained at least one internal text error. After benchmarking five models, the researchers selected Qwen3 to evaluate the full dataset and an external hospital cohort.
Speed and Accuracy Breakdown
Qwen3 achieved high overall detection rates on the primary dataset. Its performance remained consistent across imaging modalities and body regions, though it proved better at identifying explicit factual errors than nuanced interpretive ones.
Key benchmark findings from the study include:
- Qwen3 reached a report-level sensitivity of 0.99, specificity of 0.99, and an F1-score of 0.94 on the full primary dataset.
- In a head-to-head challenge set, the model achieved an exact category recall of 0.96, compared to 0.45-0.67 for radiologists with three to ten years of experience (P<.001).
- The model processed reports in a mean time of 5.80 seconds, compared to 55.80-67.80 seconds per report for human reviewers.
- In an independent secondary center, the tool maintained a balanced accuracy of 0.94.
Radiologists rated the model’s generated error explanations an average of 4.57 out of 5 on a Likert scale. This indicates high clarity for clinical workflows.
What This Means for Clinical Workflows
These metrics challenge the idea that general-purpose language models are too unreliable for clinical text processing. Recent studies like zero-shot error evaluations in radiology reports highlighted significant limitations in older setups. Conversely, work in postdeployment monitoring for pulmonary embolism AI shows that structured text oversight can protect patient safety at scale.
That distinction matters for hospital IT teams. Automating basic text audits frees up clinician hours without replacing clinical judgment.
Crucial Limitations
This evaluation was strictly a text-only retrospective review. The model looked for internal text inconsistencies, not discrepancies between the written text and the actual pixel images. A report that sounds internally coherent but completely misinterprets a scan image would pass this filter unnoticed.
Health systems should view these tools as real-time spellcheckers for clinical logic rather than autonomous diagnostic supervisors. Operational deployment will require testing whether real-time alerts reduce reader fatigue or simply cause cognitive overload.
Read the full findings in the Journal of Medical Internet Research.
