A new 10-point evaluation tool reveals that even expert clinicians struggle to agree on whether AI-generated health advice is acceptable.
Measuring AI Quality
Hospitals and clinics are adopting generative software faster than regulators can establish safety benchmarks. Without standardized testing, health systems risk deploying algorithms that produce plausible but unsafe clinical advice. To address this gap, researchers synthesized 69 publications and consulted an expert panel of 23 multidisciplinary specialists to build a standardized assessment framework called OMNI.
The system evaluates AI health outputs across 10 distinct domains using 4-point Likert scales. These domains include Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness. The framework provides detailed instructional guidelines to keep human evaluations consistent.
Testing the Rubric
To pilot the tool, 15 clinicians—including medical practitioners, pharmacists, and medical oncologists—assessed 250 input-output pairs across 4 healthcare scenarios. These scenarios covered health-related Reddit questions, AI-generated health images and videos, medical continuing education questions, and automated physician-to-physician clinical documentation. The dataset comprised 200 AI-generated responses and 50 physician-tagged Reddit replies.
The evaluation generated a massive dataset of clinician feedback across all four clinical use cases:
- Assessors produced 9,600 domain-level ratings and 4,010 qualitative comments.
- Mean pairwise agreement on exact 4-point scale scores was surprisingly low, ranging from 30.8% to 53.4%.
- When scores were collapsed into broader agreement categories, consensus improved to between 59.3% and 79.6%, with Gwet’s AC2 reliability coefficients spanning 0.49 to 0.80.
- Multimodal image and video outputs triggered the highest rate of negative scores across all tested categories.
Subjectivity in Evaluation
The low exact agreement among expert reviewers exposes a core vulnerability in clinical AI governance. Even when clinicians use the same structured grading criteria, individual judgment varies dramatically on what constitutes accurate or safe AI communication. Software makers cannot rely on loose expert spot-checks to prove safety.
Across all scenarios, AI models repeatedly failed in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy. While assessors noted OMNI was practical and worth re-using, this pilot relied on retrospective scenario testing rather than live workflow integration. Until tested in real-time environments against purpose-built clinical agents, structured rubrics can highlight model weaknesses, but they cannot single-handedly guarantee patient safety.
Read the full study in the Journal of Medical Internet Research.
