Audio Hallucinations in Egocentric Video Understanding
A recent study published on arXiv indicates that advanced audio-visual language models (AV-LLMs) are susceptible to audio hallucinations while analyzing egocentric videos. These models frequently deduce sounds from visual elements that are present but not actually audible, resulting in misleading multimodal interpretations. To address this issue, researchers have introduced a comprehensive evaluation framework based on a focused question-answering method. They assembled a dataset comprising 300 egocentric videos and formulated 1,000 sound-related questions to assess the models' outputs. A structured taxonomy differentiates between sounds from user activities and ambient background noises. This research emphasizes a significant flaw in existing AV-LLMs, especially in egocentric contexts where visual data may be inconsistent or obstructed due to ongoing camera movement, highlighting the necessity for enhanced audio-visual synchronization in AI systems.
Key facts
- arXiv paper 2604.23860v1 explores audio hallucinations in egocentric video understanding.
- State-of-the-art large audio-visual language models (AV-LLMs) are prone to audio hallucinations.
- Models infer sounds from visual cues that are visible but not heard.
- A systematic evaluation framework using a question-answering protocol is proposed.
- A dataset of 300 egocentric videos and 1,000 sound-focused questions was curated.
- A grounded taxonomy distinguishes foreground action sounds from background ambient sounds.
- Egocentric videos feature unstable or occluded visual information due to camera movement.
- The study highlights a critical limitation in current AV-LLMs.
Entities
Institutions
- arXiv