Yusuf Yesil, MD, MIS 
Evaluating risk of bias in randomized controlled trials is a notorious bottleneck in evidence-based medicine. While rigorous trial appraisals protect clinical guidelines from skewed data, manual extraction of study flaws takes hundreds of reviewer hours. Raw AI models often hallucinate quotes or lack traceable reasoning, making them unusable for formal systematic reviews.
A new workflow shows that tailored AI design changes this balance. Instead of replacing human oversight, fine-tuning an open-weight model to extract evidence-backed quotes provides a reliable preliminary draft that human experts can quickly verify.
fine-tuned accuracy outperforms base models
Researchers developed TRoB-1 by fine-tuning Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set relied on 4,321 RoB records drawn from systematic reviews across chronic obstructive pulmonary disease, diabetes, and hypertension. Researchers then tested the system on an external validation set of 161 records from recently published trials across broader disease areas.
The unassisted model far outpaced its base version, but human-in-the-loop verification yielded the strongest results. When two blinded clinicians reviewed the AI’s preliminary judgments through a local web interface, overall assessment quality spiked.
- Raw TRoB-1 achieved an unassisted accuracy between 73.3% (n=118) and 73.9% (n=119), compared to 52.2% (n=84) to 60.2% (n=97) for the base Gemma 2 workflow.
- After clinician verification, final overall accuracy climbed to between 82.0% (n=132) and 84.5% (n=136), with macro-F1 scores reaching 0.709 to 0.776.
- Human reviewers actively refined the output: Clinician 1 fixed 13 incorrect judgments while Clinician 2 corrected 18. Crucially, human verification did not turn a single correct AI judgment into an incorrect one.
- Detection of high-risk bias saw the largest jump. The F1-score for high-risk judgments rose from 0.600 to 0.784 for Clinician 1, and from 0.571 to 0.821 for Clinician 2.
That targeted bump in catching flawed trials is the real win.
Traceable workflows protect clinical evidence
This approach shifts how healthcare teams should evaluate AI in methodology work. The value lies in grounded extraction—forcing the model to pull supporting passages directly from reviewer-selected text rather than generating free-text summaries. Because the workflow uses open-weight architecture hosted locally, review teams keep sensitive data on-site while avoiding vendor lock-in.
The operational takeaway is immediate for teams running living systematic reviews. Generating structured, traceable RoB drafts allows medical librarians and clinicians to focus on verifying edge cases rather than building reviews from scratch.
Study limitations and practical reality
The system still carries clear boundaries. TRoB-1 was designed specifically around Cochrane RoB 1 rather than the newer RoB 2 framework, which uses different domain structures and signaling questions. Furthermore, the initial validation relies on text segments pre-selected by reviewers, meaning the model is not yet reading full-length trial PDFs completely unprompted.
AI health software cannot replace methodical human appraisal. But structured models that surface direct evidence offer a validated shortcut to maintain living guidelines.
Read the complete study in the Journal of Medical Internet Research.
