A new study shows video foundation models can grade violent sleep movements, but their tendency to overestimate severity reveals the limits of clinical AI.
How do you measure a disease that only shows up when a patient is asleep and thrashing in the dark? For years, tracking REM sleep behavior disorder (RBD) has required expensive, in-lab sleep studies and subjective human grading. If we want to test new drugs to stop these patients from injuring themselves, we need a cheaper, automated way to measure their symptoms.
This study challenges the idea that simple, hand-crafted computer vision rules are enough for complex medical movement analysis. Instead, it shows that massive video foundation models can step in. But the real story is not just that the AI worked. It is that the AI still struggles with scale, systematically overestimating how bad a patient’s night actually was.
Researchers analyzed infrared video-polysomnography recordings from 86 patients with isolated RBD. They split the video data into 3,329 mild clips and 284 moderate-to-severe clips. To test how well AI could grade these movements, they compared a basic heuristic classifier using optical flow against V-JEPA2, a self-supervised video foundation model.
How the models performed
- V-JEPA2 achieved 93% accuracy and a Macro F1 of 0.76 on clip-level data.
- When tested on entirely new patients, the model kept 85% accuracy and a 0.68 F1 score.
- The foundation model cut the whole-night severity error rate to 25%, compared to 52% for the heuristic model.
- For the simpler heuristic model, clip duration was the single biggest predictor of severity.
The foundation model clearly outperformed the basic math. By using advanced frame sampling based on optical flow, the deep learning model captured the nuance of sudden, violent thrashing. Simple tracking algorithms get fooled too easily by basic motion duration, whereas the foundation model actually understands the context of the movement.
The overestimation problem
Yet, the AI has a major clinical blind spot. It consistently overestimated whole-night severity scores across the cohort.
While the model successfully kept the patients in the correct relative order of severity, the absolute scores were inflated. In a clinical trial, this kind of systematic bias could mask whether a drug is actually reducing symptom severity or just shifting the baseline. We cannot yet trust these models to give an absolute diagnostic score on their own.
What clinicians should rethink
Moving RBD monitoring from the lab to the home is the ultimate goal. This study proves that video-based AI can handle the messy reality of dark, infrared sleep footage without requiring patients to wear bulky sensors. However, developers must fix the calibration of these foundation models before they can serve as trusted endpoints in clinical trials.
Read the full study on medRxiv.