A new human-AI framework shows that the best way to clean up messy electronic health records is to let algorithms and doctors correct each other.

Can we trust the medical records used to train clinical algorithms? For years, researchers assumed human chart review was the ultimate gold standard. A new preprint challenges this assumption by showing that experts frequently mislabel complex patient data when working alone.

This is not just a minor clerical issue. If our baseline data is flawed, the clinical evidence we build on top of it is compromised.

Correcting the gold standard

Researchers developed SHARE, a human-AI co-learning framework, to label rheumatoid arthritis disease activity. They tested it on a multi-institutional cohort of 3,167 patients. Expert reviewers initially labeled 626 notes from 273 patients, but the collaborative process forced a rewrite.

The human-AI adjudication revised 127 of the initial labels. This means experts changed their minds on 20% of the notes. The process also added 60 newly labeled notes, resulting in a 686-note co-produced reference standard. Through this co-learning loop, the AI’s mean absolute error dropped from 0.406 to 0.291, while its ambiguity flags matched human experts with 92.1% accuracy.

This matters because manual chart review is the primary bottleneck in clinical research. While electronic health records hold massive potential, as outlined in earlier research on challenges in cardiovascular research, the sheer volume of unstructured text makes manual scaling impossible.

The cost of scale

To scale the system, researchers used a tiered AI setup. They combined GPT-5 Nano for high-volume evidence extraction and o4-mini for final reasoning, rather than running a heavy GPT-5.4 model for every step.

This tiered approach labeled 101,691 notes across the cohort. It matched the accuracy of the heavy model while slashing estimated costs by 69% and compute time by 70%. This proves that smart routing, rather than raw computing power, is the key to processing massive clinical datasets.

• Human-AI collaboration revised 20% of initial expert-assigned labels.
• The final system reduced mean absolute error to 0.291.
• The budget-tiered configuration cut costs by 69% and compute time by 70%.

Rethinking expert data

The real takeaway is that AI should not just automate human labor. It must audit it. If human experts are wrong one-fifth of the time, pure manual review is an unstable foundation for real-world evidence.

This collaborative approach could help manage other complex chronic diseases, mirroring similar data-driven efforts in predicting and managing obesity-associated diabetes. However, we must remain cautious. This study is a preprint and focused on a single disease. We do not yet know how this tiered model handles rarer conditions with less structured data.

Read the full study on medRxiv.