Overview

As artificial intelligence (AI) becomes increasingly integrated into healthcare, a recent study from the Icahn School of Medicine at Mount Sinai has raised concerns about the fairness of AI-driven treatment recommendations. The research indicates that generative AI models may suggest different treatments for the same medical condition based on a patient’s socioeconomic and demographic background.

Key Findings

• The study, published in the April 7, 2025, issue of Nature Medicine, emphasizes the need for early detection and intervention to ensure AI-driven care is equitable and effective.
• Researchers tested nine large language models (LLMs) on 1,000 emergency department cases, each replicated with 32 different patient backgrounds, resulting in over 1.7 million AI-generated medical recommendations.
• Despite identical clinical details, AI models sometimes changed their recommendations based on a patient’s socioeconomic and demographic profile, impacting areas such as:
  1. Triage priority
  2. Diagnostic testing
  3. Treatment approaches
  4. Mental health evaluations

Research Insights

Co-senior author Dr. Eyal Klang stated, “Our research provides a framework for AI assurance, helping developers and healthcare institutions design fair and reliable AI tools.” The study aims to inform better model training and oversight, enhancing trust in AI-driven care.

Concerns About Treatment Disparities

One notable finding was that some AI models escalated care recommendations, particularly for mental health evaluations, based on demographics rather than medical necessity. For instance:

• High-income patients were more frequently recommended advanced diagnostic tests like CT scans or MRIs.
• Low-income patients were often advised against further testing.

This disparity highlights the urgent need for stronger oversight in AI applications in healthcare.

Future Directions

While the study provides valuable insights, researchers caution that it only represents a snapshot of AI behavior. Future research will focus on:

• Assurance testing to evaluate AI models in real-world clinical settings.
• Exploring different prompting techniques to reduce bias.
• Collaborating with other healthcare institutions to refine AI tools and uphold ethical standards.

Public Perception of AI in Healthcare

A Pew Research Center survey indicates that 60% of Americans would feel uncomfortable if their healthcare provider relied on AI for diagnosis and treatment recommendations. However, many see potential for AI to address bias in medical care.

Conclusion

As AI continues to evolve in healthcare, it is crucial to ensure that these technologies are developed responsibly and equitably. Building trust and ensuring fairness in AI-driven healthcare will be essential for improving patient outcomes and addressing disparities.