By capturing what surgeons say and where they look, a new AI system automates surgical video review without requiring manual data entry.

How do you teach an AI to understand what a surgeon is thinking during a complex operation? For years, the bottleneck in surgical data science has been annotation. Human experts must spend hours manually labeling video frames, a tedious process that often strips away the real-time decision-making of the operating room.

A new study challenges this manual paradigm by capturing the natural exhaust of clinical work. Instead of forcing surgeons to write reports after the fact, researchers built a system that translates spoken commentary and eye-gaze tracking directly into structured data. This shift suggests that the future of medical AI lies in passive capture, not active documentation.

Capturing the surgeon’s mind

The system works by breaking down transcribed verbal commentary into video-anchored semantic chunks. A large language model then classifies these chunks, while eye-gaze or cursor tracking provides the spatial context. This means the AI knows exactly what the surgeon is looking at when they describe a specific anatomical structure.

This approach changes how we think about clinical workflows. If an AI can reliably map a surgeon’s implicit reasoning, we no longer need to choose between detailed registries and surgeon burnout. The documentation happens automatically as the surgeon works and speaks.

The performance metrics

To test the method, researchers evaluated its performance on both structured and unstructured surgical tasks. The AI demonstrated high fidelity when compared to human experts. The key results include:

• A mean cosine similarity of 0.95 (SD: 0.01) for chunking verbal feedback during full-length colorectal procedures.
• A mean Cohen’s kappa of 0.71 (SD: 0.07) for semantic classification across surgical observations.
• An agreement score of 0.67 (SD: 0.14) for identifying evaluative triggers.
• Agreement scores of 0.83, 0.49, and 0.81 across three critical view of safety criteria in gallbladder removals.

These numbers prove that the system can match human precision in structuring complex dialogue. The high similarity score in chunking shows the AI rarely misses the boundaries of a surgeon’s thought process.

The safety catch

However, the data also reveals a critical limitation. The wide variance in safety assessment—specifically the low score of 0.49 on one safety criterion—proves that implicit tracking is not a perfect substitute for explicit review. Some safety-critical decisions are silent and do not register on gaze trackers.

Relying solely on what a surgeon says or looks at might miss the moments of passive hesitation or unspoken doubt. For high-stakes safety audits, human oversight remains non-negotiable. This tool is an assistant for scaling dataset creation, not an independent arbiter of surgical safety.

The broader implication is clear. We can now build massive, annotated surgical video libraries without hiring armies of annotators. This will accelerate how we train surgical AI, even if the algorithms still need a human safety net.

Read the full study in medRxiv.