BAT: New AI Model for Spatial Sound Reasoning with LLMs
A groundbreaking AI system named BAT has been unveiled by researchers, merging spatial sound perception with natural language reasoning to analyze and interpret environmental sounds. This innovative system employs a binaural acoustic scene analysis model alongside a large language model (LLM), allowing it to respond to inquiries regarding spatial audio. For BAT's training, the team created a binaural audio dataset utilizing AudioSet and SoundSpaces 2.0, and crafted SpatialSoundQA, a question-answering dataset that explores various dimensions of spatial sound perception and reasoning. BAT's acoustic front end features a new spatial audio encoder known as the Spatial Audio Spectrogram Transformer (Spatial-AST), which excels in sound event detection, spatial localization, and distance estimation. This research addresses the scarcity of real-world spatial sound datasets and seeks to emulate humans' innate abilities to navigate and interpret their auditory environment. The paper can be found on arXiv with the identifier 2402.01591.
Key facts
- BAT combines spatial sound perception with LLM reasoning
- Uses binaural acoustic scene analysis model and LLM
- Synthesized binaural audio dataset using AudioSet and SoundSpaces 2.0
- Developed SpatialSoundQA dataset for QA tasks
- Introduced Spatial-AST, a spatial audio encoder
- Spatial-AST performs well in sound event detection, spatial localization, and distance estimation
- Paper available on arXiv (2402.01591)
Entities
Institutions
- arXiv