Title: LLMs Extract Breast Cancer Data Like Human Oncologists
Off-the-shelf artificial intelligence can now parse thousands of pages of messy medical records to build research-grade cancer datasets, matching the accuracy of expert oncologists.
Can an algorithm read a 3,000-page medical chart better than a trained clinical researcher?
For years, clinical research has stalled because human experts must manually dig through unstructured clinical notes to track cancer recurrence and treatment histories. This study challenges the assumption that we need highly specialized, custom-trained medical AI models to solve this bottleneck. By using unmodified, off-the-shelf large language models (LLMs), researchers proved that standard software can extract complex longitudinal data almost as well as oncologists.
Researchers tested this pipeline on **100** breast cancer patients with highly complex cases. The data was massive. The median patient chart spanned more than **3,100** pages of text, tracking patients who received a median of **7** lines of therapy over **6.5** years of follow-up. The input data was completely raw, consisting of unedited clinical notes, pathology reports, and medication administration records.
How the models performed
The best-performing LLM achieved high concordance with expert oncologists across several critical variables:
- 99% concordance for cancer recurrence status
- 100% for germline BRCA1/2 pathogenic variant detection
- 99% for hormone receptor status
- 96% for HER2 status
- 91% for clinical stage and PIK3CA mutation status
- 90% for ESR1 mutation status
The limits of automation
The AI did not perform perfectly across all tasks. When reconstructing exact lines of therapy, the best LLM scored **9 percentage points** lower than a second human oncologist. However, all **four** tested LLMs still outperformed human research coordinators. Crucially, the survival curves and hazard-ratio estimates calculated from the LLM-derived data matched those from the expert-derived datasets.
To prove the system’s reliability, researchers tested the unmodified pipeline on an external cohort of **97** young patients with early-stage breast cancer. The AI maintained its high performance for detecting recurrence and tracking adjuvant endocrine therapy. This suggests the tool is ready for real-world deployment, echoing similar successes in tracking real-world outcomes of neoadjuvant HER2-directed therapy using AI-enhanced data pipelines.
This finding matters because it removes the expensive barrier of custom AI training. If researchers can use standard, off-the-shelf models to build accurate survival curves from messy, multi-thousand-page records, the speed of observational oncology research will accelerate. We can finally move past small, curated clinical trials and analyze real-world outcomes for millions of patients in real time.
This study was published in medRxiv.
