A new benchmark reveals that even calibrated image-recognition models struggle to safely hand off difficult skin cancer cases to human specialists.

What happens when a medical AI does not know what it does not know? In dermatology, an overconfident algorithm can mean a missed melanoma.

Developers often point to high AUROC scores to prove their models are ready for the clinic. But this study of the HAM10000 dataset exposes a critical gap. High overall accuracy does not guarantee the model knows when to ask for help.

If an AI cannot reliably flag its own uncertainty, clinics cannot use it for safe triage. This forces us to shift our focus from raw diagnostic accuracy to calibration and referral safety.

The Calibration Problem

The researchers evaluated an image-only ResNet50 model trained on the HAM10000 dataset using inverse-frequency weighted cross-entropy. On the internal test set, the model achieved an accuracy of 0.745, a balanced accuracy of 0.673, and a macro-F1 of 0.625. Its macro one-vs-rest AUROC reached a seemingly impressive 0.941.

Yet, these standard metrics hide a deeper issue of probability inflation. To fix this, the team applied temperature scaling. This adjustment did not change the top predictions, but it successfully dropped the 15-bin expected calibration error from 0.111 to 0.034. It also reduced the Brier score from 0.362 to 0.337 and the negative log-likelihood from 0.731 to 0.636.

For years, the machine learning community has treated AUROC as the gold standard. This paper shows why that is a mistake. A model can have an AUROC of 0.941 and still fail to safely route patients.

When Triage Fails

The real test of a triage AI is selective referral. The system should pass highly uncertain cases to a human doctor.

In this trial, researchers set an entropy threshold to target a selective risk no higher than 0.150. When tested, the system fell short. The actual selective risk rose to 0.171, while the held-out test coverage was 0.774.

Most concerningly, the system referred only 77 out of 219 melanoma cases. This means the algorithm kept the remaining 142 melanoma cases in its autonomous loop. If the AI’s overall accuracy is just under 75 percent, a worrying number of these highly aggressive cancers may have been misclassified without ever being seen by a human.

The Limits of Retrospective Data

To see how the model handled slightly different data, the researchers ran a related-domain test using 1,512 matched images from the ISIC2018 Task 3 dataset. The model maintained an accuracy of 0.731, a macro-F1 of 0.639, and a macro AUROC of 0.933.

These numbers show consistency, but they do not solve the safety problem. This study remains a retrospective benchmark, not a prospective trial in a real clinic. It proves that calibrating AI is a delicate balancing act, and we are still far from autonomous triage.

• Internal test accuracy reached 0.745 with an AUROC of 0.941.
• Temperature scaling successfully reduced calibration error to 0.034.
• The system missed its selective risk target of 0.150, hitting 0.171 instead.
• Only 77 of 219 melanoma cases were referred to human specialists.

Read the full study in Scientific Reports.