โก Quick Summary
This study explored the use of AI-generated medical diagnoses to fine-tune the Llama-2-13B model for optimizing ICD10 coding in gynecologic oncology. The results demonstrated a remarkable increase in accuracy, with the best model achieving an accuracy of 0.95 and a Kappa score of 0.94.
๐ Key Details
- ๐ Dataset: 2415 discharge records, 83 records for validation
- ๐งฉ Focus: Gynecologic oncology and ICD10 coding
- โ๏ธ Models tested: Four variations of Llama-2-13B
- ๐ Best performance: Model 4 with 20 AI-generated statements per code
๐ Key Takeaways
- ๐ค AI-generated diagnostics significantly enhance model performance.
- ๐ Model 4 achieved an impressive accuracy of 0.95.
- ๐ก Kappa score for Model 4 was 0.94, indicating strong agreement.
- ๐งช Methodology offers a cost-effective strategy for ICD coding.
- ๐ Potential for broader applications in healthcare beyond gynecologic oncology.
- ๐ Study period: Data collected from January 1, 2020, to June 30, 2023.
- ๐ Validation involved rigorous confirmation of AI-generated texts.
๐ Background
The field of gynecologic oncology faces challenges in accurately coding diagnoses, which is crucial for patient management and healthcare statistics. The introduction of Large Language Models (LLMs) like Llama-2 presents an opportunity to enhance the accuracy of ICD coding through innovative AI techniques. This study aims to leverage these advancements to improve coding efficiency and reliability.
๐๏ธ Study
Conducted by a team of researchers, this study focused on fine-tuning the Llama-2-13B model using AI-generated diagnostic texts. The researchers established four models, each with varying levels of AI-generated data, to assess their impact on the accuracy of ICD10 coding in gynecologic oncology. The validation process involved a comprehensive review of 83 discharge records.
๐ Results
The results were striking. The original Llama-2-13B model (Model 1) had an accuracy of only 0.06 and a Kappa score of 0.04. In contrast, Model 4, which utilized 20 AI-generated statements per ICD10 code, achieved an accuracy of 0.95 and a Kappa score of 0.94. This demonstrates a significant leap in performance, highlighting the effectiveness of AI-generated data in enhancing model capabilities.
๐ Impact and Implications
The findings from this study have the potential to revolutionize the way ICD coding is approached in gynecologic oncology. By integrating AI-generated diagnostic descriptions, healthcare providers can achieve higher accuracy in coding, which is essential for patient care, billing, and research. This methodology not only optimizes model performance but also opens doors for broader applications of LLMs in various medical fields.
๐ฎ Conclusion
This study underscores the transformative potential of AI in healthcare, particularly in the realm of diagnostic coding. The successful fine-tuning of the Llama-2-13B model using AI-generated data marks a significant step forward in optimizing ICD coding processes. As we continue to explore the capabilities of LLMs, the future looks promising for enhancing accuracy and efficiency in medical coding and beyond.
๐ฌ Your comments
What are your thoughts on the integration of AI in medical coding? We would love to hear your insights! ๐ฌ Leave your comments below or connect with us on social media:
Fine-tuning Llama-2-13B with AI-generated medical diagnoses: A novel strategy for optimizing ICD coding in gynecologic oncology.
Abstract
OBJECTIVE: Given the substantial advancements in Large Language Models (LLMs), this study aimed to explore the effectiveness of using AI-generated medical diagnoses in the fine-tuning of the Llama-2 model, with the objective of optimizing the ICD10 coding process for gynecologic oncology. This study aimed to fine-tune the Llama-2-13B model using AI-generated diagnostic texts based on ICD10 descriptors, focusing on gynecologic oncology for initial validation.
MATERIALS AND METHODS: AI-generated diagnostic texts were rigorously confirmed to ensure medical coherence and reliability for fine-tuning. Four models were established: The original Llama-2-13B (Model 1); a model fine-tuned with basic ICD10 codes (Model 2); a model trained with an additional set of 10 AI-generated diagnosis statements per ICD10 code (Model 3); and the forth model trained with an additional set of 20 AI-generated statements per code (Model 4). Validation involved a set of 83 discharge records related to gynecologic oncology, derived from 2415 discharge records collected from January 1, 2020, and June 30, 2023.
RESULTS: Validation results for the models showed significant improvement in the accuracy rates and Kappa scores: Model 1 (native Llama-2-13B) had an accuracy of 0.06 and a Kappa score of 0.04, Model 2 achieved 0.24 and 0.19, Model 3 reached 0.90 and 0.89, and Model 4 greatly improved to 0.95 and 0.94.
CONCLUSION: The use of prompts to generate diagnostic descriptions, coupled with AI-generated data for model fine-tuning, resulted in a substantial enhancement in the Llama-2-13B model’s capability to accurately determine ICD diagnostic codes from medical records. This methodology offers a cost-effective strategy, optimizes model accuracy, and underscores the potential for broader applications due to the LLM’s generative capabilities.
Author: [‘Liang YL’, ‘Chen CF’, ‘Wu MH’, ‘Hsu KF’, ‘Chang PT’, ‘Chuang YJ’, ‘Chen PF’]
Journal: Taiwan J Obstet Gynecol
Citation: Liang YL, et al. Fine-tuning Llama-2-13B with AI-generated medical diagnoses: A novel strategy for optimizing ICD coding in gynecologic oncology. Fine-tuning Llama-2-13B with AI-generated medical diagnoses: A novel strategy for optimizing ICD coding in gynecologic oncology. 2025; 64:978-984. doi: 10.1016/j.tjog.2025.02.006