New data reveals that the AI tools explaining cancer diagnoses to pathologists cannot agree on what matters most.

Pathologists are told to trust AI, but they cannot see inside the black box. If an algorithm flags a lymph node as cancerous, how does a doctor know it looked at the actual tumor cells and not a speck of dust on the slide? This trust gap is the real barrier to clinical adoption.

A new study puts explainable AI to a rigorous test. The results expose a hard truth: there is no single best way to explain an AI’s decision. Instead, developers must choose between explanations that look right to human eyes and those that are mathematically true to the model’s inner workings.

Researchers evaluated four popular explanation methods—LIME, GradCAM, GradCAM++, and SHAP—across three neural networks trained on the PatchCamelyon benchmark of 220,026 image patches. To see if the AI’s reasons matched human logic, they compared the explanations against 2,847 expert pathologist annotations. They also measured mathematical faithfulness to ensure the explanations were not just pretty pictures.

The Explanation Divide

The data shows a clear split in performance. Some tools excel at mimicking human intuition, while others excel at representing the raw math. Developers cannot have both in a single tool.

• GradCAM++ achieved the highest spatial agreement with pathologist annotations, scoring a mean Intersection over Union of 0.52 ± 0.14 using the EfficientNetB3 model.
• SHAP proved to be the most mathematically faithful to the model, leading with an Area Over the Perturbation Curve score of 0.61 ± 0.08.
• A squaregrid variant of LIME offered a strong middle ground with a spatial agreement score of 0.44 ± 0.17, while running at 3.8 times lower computational cost than standard LIME.
• The study found a strong Spearman correlation of 0.81 between how accurate a model is and how well its explanations align with human experts.

The Regulatory Reality

This finding complicates the push for regulatory approval under frameworks like the EU IVDR and the FDA SaMD Action Plan. Regulators want transparency, but this study proves that transparency is not a single metric. If you optimize for what a pathologist wants to see, you might lose track of what the AI is actually calculating.

For clinical deployment, a single explanation tool is no longer enough. Hospitals will need a tiered strategy. Use GradCAM++ for fast, high-throughput daily workflows, but keep SHAP in reserve for audits and deep research.

We must be honest about the boundaries of this data. This was a single-benchmark study. Before these tools can be trusted in real clinics, they require prospective, multi-site validation and a formal reader study with practicing pathologists.

Read the full analysis in Diagnostics.