Human oversight fails to catch specific AI errors before emergency discharge papers reach patients.
The illusion of human oversight
Hospitals rely on clinician signatures as the ultimate safety net for software-generated medical records. When doctors review drafts produced by artificial intelligence, regulators assume that human expertise filters out machine hallucinations before documents enter patient records.
That safety net has holes.
A trial at a paediatric emergency department tested whether doctor oversight actually catches drafting errors. The results reveal that while clinicians successfully prune obvious fabrications, subtle errors pass directly through human review into final medical records.
What the trial revealed
Researchers randomised 153 children discharged from a paediatric emergency department to either standard template assembly or drafting by an on-premises open-weight model. A total of 117 cases completed analysis, comparing 59 conventional documents against 58 LLM-assisted drafts.
The trial evaluated patient comprehension, document safety, and doctor edit behavior across both workflows:
- Family comprehension did not change, showing a primary outcome difference of -0.01 (95% CI -0.12 to 0.10) in record-concordant understanding at follow-up.
- No discrepancy in physician-facing final records was rated as seriously consequential (0/117).
- Clinicians edited 48 of 58 AI drafts before final sign-off.
- Reviewers stripped out 18 of 22 unsupported additions made by the model.
- Reviewers caught only 6 of 22 incorrect abbreviation expansions.
The flaw in edit tracking
This selective vigilance creates a specific operational hazard. Doctors easily spot extra claims that lack support in the chart, but skim past plausible yet false expansions of shorthand medical jargon. Studies on EHR-integrated discharge tools often highlight overall output quality, but fast sign-offs can mask systematic oversight gaps.
Edit counts alone do not prove safety. As outlined in broader discussions of clinical language model evaluation, tracking how many words a doctor changes tells us nothing about the errors they failed to notice.
What health systems must change
The study carries key limitations. It was a single-center trial focused on paediatric emergency discharges, and it did not measure long-term health outcomes after discharge.
Health systems cannot treat clinician sign-offs as proof of safety validation. Auditing software must directly compare unedited model outputs against finalized records to detect persistent errors like bad abbreviation mapping. Operational governance requires measuring what human reviewers miss, not just what they change.
Read the complete study in medRxiv.
