A new study reveals that while AI easily spots prostate cancer across different imaging hardware, the actual grading of those tumors falls apart depending on which scanner a hospital uses.

Can we trust diagnostic AI when a hospital changes its microscope camera? Most pathology algorithms are trained on pristine, high-end Western scanners. When deployed in the real world, a simple hardware swap can render these tools dangerously unreliable.

This diagnostic drift challenges the industry assumption that a single, end-to-end model can handle both detection and grading. Instead, we must decouple these steps. AI works beautifully as a robust triage tool to flag cancer, but the actual grading remains highly sensitive to hardware. This mirrors challenges in other oncology domains, where researchers use techniques like xMagNet to handle magnification and scanner variability in breast cancer imaging.

Testing the hardware shift

Researchers tested an AI model developed on North American and European tissue on the PAR dataset from Erbil, Iraq. This cohort provided a rigorous double-test of both population and scanner drift. The dataset included 1017 whole-slide images from 339 slides representing 185 patients, with 49.6% of the slides showing clinically significant cancer. Crucially, every single physical slide was digitized on three different scanners: a compact Grundium, a Hamamatsu, and a Leica.

The AI evaluated these slides with no local tuning or scanner-specific adjustments. The results showed a stark divide between finding cancer and grading it.

The split in performance

• Robust detection: Cancer detection was virtually identical across all hardware, with an area under the ROC curve (AUC) of 0.987 to 0.991.
• Gated pipeline success: A “detect-then-grade” pipeline reached slide-level agreement scores of 0.86 on Grundium, 0.81 on Hamamatsu, and 0.86 on Leica, matching the human pathologist ceiling.
• Ungated failure: Without gating the model to focus only on confirmed cancer tissue, performance collapsed to agreement scores between 0.23 and 0.62.
• Grading volatility: On confirmed cancers, grading accuracy dropped to 0.44 on the Hamamatsu scanner, compared to 0.67 on Leica and 0.63 on Grundium.

Why this matters

This finding exposes a critical vulnerability in how we evaluate clinical AI. If an engineer boasts about a model’s high accuracy, you must ask which scanner generated the images. A model that excels on a Leica scanner might fail on a Hamamatsu setup down the street, even when analyzing the exact same tissue slide.

The clinical stakes are high. The two slides of a single patient case disagreed in grade in 43% of cases, highlighting how volatile prostate grading remains. If AI is to assist pathologists safely, developers must stop shipping monolithic models. They must deploy “detect-then-grade” pipelines where the grading component is calibrated and validated for each specific local scanner.

Read the full preprint study on medRxiv.