โก Quick Summary
This study compared code-based and AutoML methods for pill recognition in clinical settings, revealing that while Google Vertex AI achieved the highest accuracy of 91.60%, YOLO11 provided the best flexibility and cost-effectiveness.
๐ Key Details
- ๐ Dataset: 26,880 images from 30 commonly dispensed medications
- โ๏ธ Technology: Ultralytics YOLO11 and 3 AutoML platforms (Amazon Rekognition, Google Vertex AI, Microsoft Azure Custom Vision)
- ๐ Performance: Accuracy ranged from 80.83% (YOLO11) to 91.60% (Google Vertex AI)
- ๐ฐ Costs: YOLO11 (open-source), Google Vertex AI ($69.30), Amazon Rekognition ($5.43-$43.89), Microsoft Azure ($9.50-$28.60)
๐ Key Takeaways
- ๐ค YOLO11 showed consistent performance improvement with increased training data.
- ๐ก Google Vertex AI reached above 90% accuracy but had unpredictable performance declines.
- ๐ Amazon Rekognition had high false negative rates, missing many pills despite high precision.
- ๐ Custom Vision lagged behind other platforms, likely due to its older architecture.
- ๐ Performance varied significantly across different clinical datasets.
- ๐ Training times varied widely, with some platforms taking nearly 40 hours for full datasets.
- โ๏ธ Each platform has distinct advantages and trade-offs, requiring careful selection based on needs.
- ๐ก๏ธ Patient safety is paramount, necessitating rigorous validation of chosen models.
๐ Background
In clinical settings, the visual identification of medications is crucial yet prone to human error, especially under pressure. Misidentification can lead to serious medication errors, impacting patient safety and straining healthcare systems. Recent advancements in computer vision and object detection present promising avenues for automating pill recognition, yet comprehensive comparisons of different methodologies remain scarce.
๐๏ธ Study
This study aimed to evaluate the performance, cost, usability, and deployment feasibility of pill recognition models developed using Ultralytics YOLO11 and three cloud-based AutoML platforms: Amazon Rekognition, Google Vertex AI, and Microsoft Azure Custom Vision. The researchers utilized five training subsets of varying sizes and multiple datasets, including real-world clinical images, to assess the models’ effectiveness.
๐ Results
The findings indicated that no single platform excelled across all test environments. On the verification dataset, accuracy ranged from 80.83% for YOLO11 to 91.60% for Google Vertex AI. YOLO11 demonstrated a significant performance boost with larger training datasets, achieving near-perfect precision scores. In contrast, Amazon Rekognition exhibited high precision but also the highest false negative rates, while Custom Vision showed steady improvements but remained behind the other platforms.
๐ Impact and Implications
The results of this study highlight the potential of automated pill recognition technologies to enhance patient safety in clinical workflows. By selecting the appropriate platform based on specific performance requirements and budget constraints, healthcare providers can significantly reduce the risk of medication errors. This research underscores the importance of rigorous validation using real-world data to ensure the reliability of these technologies in practice.
๐ฎ Conclusion
This comparative performance study illustrates the diverse capabilities of both code-based and AutoML methods for pill recognition. While YOLO11 offers flexibility and lower costs, AutoML platforms like Google Vertex AI can deliver high accuracy, albeit with some unpredictability. As healthcare continues to embrace technology, careful consideration of these tools will be essential for improving patient safety and operational efficiency.
๐ฌ Your comments
What are your thoughts on the use of AI in medication identification? We would love to hear your insights! ๐ฌ Share your comments below or connect with us on social media:
Code-Based Versus AutoML Methods for Pill Recognition in Clinical Settings: Comparative Performance Study.
Abstract
BACKGROUND: Visual identification and verification of medications during dispensing and administration are prone to human error, particularly in high-pressure and high-volume clinical settings. Misidentification can lead to medication errors, posing risks to patient safety and placing a burden on health care systems. Recent advances in computer vision and object detection offer promising solutions for automated solid oral dosage form (pill) recognition. However, comprehensive studies comparing code-based and no-code (automated machine learning [AutoML]) approaches for pill recognition are lacking.
OBJECTIVE: This study aimed to evaluate and compare the performance, cost, usability, and deployment feasibility of pill recognition models developed with Ultralytics YOLO11 and 3 cloud-based AutoML platforms (Amazon Rekognition Custom Labels, Google Vertex artificial intelligence [AI] AutoML Vision, and Microsoft Azure Custom Vision) using multiple datasets, including real-world clinical images.
METHODS: Five training subsets of increasing size (1230, 3450, 7380, 14,400, and 26,880 images) from 30 commonly dispensed medications were used to train models on YOLO11 and 3 AutoML platforms. Models were evaluated on 6 datasets from different environments: clinical images from 3 hospitals, a verification dataset, a laboratory dataset, and an exhaustive testing set. Performance metrics, including accuracy, precision, recall, and mean average precision, were calculated. We evaluated the impact of training data size on performance and benchmarked training time, platform costs, and limitations.
RESULTS: No single platform dominated across all test environments. On the verification dataset (optimal conditions), accuracy ranged from 80.83% (YOLO11) to 91.60% (Google Vertex AI) when trained with the full training dataset. YOLO11 showed consistent performance improvement with increasing training data (accuracy: 63.06%-80.83%) and achieved near-perfect precision and mean average precision scores (0.95-1.00). Google Vertex AI reached above 90% accuracy on 3 training subsets but showed unpredictable declines. Amazon Rekognition maintained near-perfect precision (0.92-1.00) but had the highest false negative rates (up to 0.74), missing many pills. Custom Vision demonstrated steady performance improvements (77.08%-85.62% accuracy) but lagged behind other AutoML platforms, probably due to its older YOLOv2-based architecture. On clinical datasets, accuracy fluctuated (20.62%-90%) depending on the dataset and platform. Training costs and time varied: YOLO11 (open-source), Microsoft Azure (US $9.50-US $28.60, allowed user-predefined training duration), Google Vertex AI (US $69.30 with consistent 2.5-3-hour training times), and Amazon Rekognition (US $5.43-US $43.89 with size-dependent training time scaling, reaching nearly 40 hours on the full 26,880-image dataset).
CONCLUSIONS: Each platform offers distinct advantages and trade-offs: YOLO11 provides the highest flexibility and lowest platform costs but requires technical expertise, while AutoML platforms can offer high performance at a higher cost but with limited user control, introducing unpredictability. The performance variations demonstrate that successful clinical deployment requires careful platform selection based on specific performance requirements, budget constraints, and available technical resources, followed by rigorous validation using real-world, representative data to ensure patient safety in clinical workflows.
Author: [‘Ashraf AR’, ‘Rรกdli R’, ‘Vรถrรถshรกzi Z’, ‘Fittler A’]
Journal: JMIR Med Inform
Citation: Ashraf AR, et al. Code-Based Versus AutoML Methods for Pill Recognition in Clinical Settings: Comparative Performance Study. Code-Based Versus AutoML Methods for Pill Recognition in Clinical Settings: Comparative Performance Study. 2026; 14:e79160. doi: 10.2196/79160