Hospitals paying premium prices for top-tier AI models to analyze patient feedback are throwing money away.

Why do health systems spend thousands on third-party vendors and premium AI models to read patient comments? A new validation study reveals that cheap, lightweight AI models read patient comments just as well as their expensive flagship counterparts. This finding upends the assumption that clinical language processing requires massive, costly computing power.

The study analyzed 512 free-text HCAHPS comments collected from two community hospitals in 2023. Six trained human reviewers, including medical students and practicing internists, graded the comments to establish a benchmark. They compared these human grades against two OpenAI models: the flagship GPT-5 and the cost-optimized GPT-5-nano. This builds on earlier efforts to map patient sentiments, such as a recent scoping review on automatic sentiment analysis in BMJ Health & Care Informatics.

The performance parity

The results show that paying for the premium model yields zero performance gains. Human reviewers achieved a substantial agreement score of 0.79 on the Cohen’s kappa scale. Remarkably, both AI models beat the human baseline, scoring an identical kappa of 0.85, an accuracy of 0.92, and a weighted F1-score of 0.93.

• Human agreement baseline reached a substantial Cohen’s kappa of 0.79.
• Both AI models exceeded humans with an identical kappa of 0.85.
• Overall accuracy for both models was 0.92 with a 0.93 weighted F1-score.
• The cost-optimized model processed all comments for $0.04 compared to $0.18 for the flagship.

The blind spots

The data reveals a critical limitation that developers must address. While both models excelled at identifying positive sentiments (F1 ~ 0.97) and negative sentiments (F1 ~ 0.90), they failed miserably on neutral comments, scoring an F1 of 0.19 or lower. This struggle with neutral feedback is a known issue in clinical NLP, as discussed in research on optimizing LLMs for medical sentiment in the IEEE Journal of Biomedical and Health Informatics.

This neutral-class failure is not just a statistical quirk. It means the AI struggles with mixed feedback, such as a patient praising the nurses but complaining about the food. If the system misclassifies these nuanced comments, hospital administrators will miss key operational pain points.

The bottom line

The real story is the price tag. The cost-optimized model analyzed the entire dataset for just $0.04, while the flagship model cost $0.18. That is a 4.2-fold cost difference for the exact same output. Scaled across a national health system receiving millions of comments, this difference represents massive savings.

For hospital administrators, the “so what” is clear. Do not buy the hype of expensive, specialized medical AI packages for basic feedback triage. Lightweight, API-driven models are ready today, and they run on pocket change.

Read the full study in medRxiv.