Video Language Models Fail at Simple Event Bookkeeping, Study Finds
A new study shared on arXiv (ID: 2608.06361) shows that video language models (VLMs) struggle with counting events accurately, exposing a timing issue. The researchers introduced a method called trace-grounded parametric profiling to tackle event counting in three specific video tasks: bouncing-ball wall hits, visual blinks, and state transitions. They analyzed a total of 2,190 videos, adjusting the event count (N) and frequency (F) while keeping the rendering the same. The results reveal that Gemini 3.6 Flash can correctly count up to 12 state transitions at 0.5 and 1.0 Hz with an accuracy of 80%, but it struggles beyond that. The study highlights that existing benchmarks make it hard to pinpoint where models fail, as they mix different factors like event count and visual complexity. This research proposes a more straightforward evaluation method to improve VLMs, particularly for tasks requiring precise event tracking.
Key facts
- Study introduces trace-grounded parametric profiling for event counting in video language models.
- Three controlled video tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions.
- 2,190 videos used, varying event count (N) and frequency (F) with fixed rendering.
- Each video includes an executable event trace for timestamp-level evaluation.
- Gemini 3.6 Flash reliably counts up to 12 events at 0.5 and 1.0 Hz at 80% reliability threshold.
- No reliable positive performance beyond 12 events at those frequencies.
- Existing benchmarks entangle event count, rate, duration, and visual complexity.
- Programmatic benchmarks often score only final answers, not auditing reported events.
- Study aims to bridge gap by providing controlled and auditable evaluation.
- Findings highlight staged temporal failure in VLMs for event bookkeeping.
Entities
Institutions
- arXiv