A pioneering review conducted by researchers at Adelaide University has highlighted the absence of clear guidelines for the initial testing of AI tools in healthcare settings, a process referred to as silent trials.

Key Findings from the Review:

• The review revealed significant inconsistencies in how silent trials are conducted.
• There is a lack of standardized measures to evaluate the effectiveness of AI tools.
• Some trials focused on metrics that are not clinically relevant, while others provided minimal information on model performance.

According to Lana Tikhomirov, a PhD candidate and corresponding author, “This lack of guidance around silent trials is concerning as AI models can be unpredictable and difficult to use in real-world settings if they haven’t been tested thoroughly.”

Understanding Silent Trials

Silent trials involve testing AI models in their intended clinical environments without impacting patient care, as the results are not shared with the clinical team during treatment. Currently, there are no formal guidelines governing these trials, which are essential for ensuring that AI tools are effective and beneficial in specific healthcare contexts.

Importance of Comprehensive Testing

Co-author Associate Professor Melissa McCradden emphasized the importance of silent trials, stating, “Silent trials are a low-risk way to test technology without compromising patient outcomes.” She noted that many AI models fail when implemented in real-world settings, and an AI tool that is effective in one hospital may not perform well in another.

Future Directions

The review, published in Nature Health, is part of a broader initiative called Project CANAIRI (Collaboration for Translational Artificial Intelligence Trials), which aims to develop guidelines for silent trials. The goal is to ensure that healthcare environments are adequately prepared to implement AI tools effectively.

Ultimately, the researchers advocate for making silent trials a mandatory step in the adoption of AI tools in medicine. “Conducting comprehensive silent trials that adhere to a clear set of international guidelines is critical if we want to successfully take AI tools from bench to bedside,” said McCradden.