LAIP: Audio-Informed Pooling Enhances Spatial Grounding in Audio-Visual Retrieval

Researchers have introduced a novel framework named LAIP (Localization via Audio-Informed Pooling) that enhances spatial grounding in extensive audio-visual retrieval models, eliminating the need for dense spatial annotations. Their findings indicate that while the upper layers of retrieval backbones tend to lose spatial detail due to global pooling, intermediate visual tokens maintain organized spatial information. By substituting traditional global aggregation with a streamlined Audio-informed Spatial Pooling (AiSP) module, LAIP facilitates precise sound source localization using weakly supervised data. This approach utilizes latent representations from models trained on an unprecedented scale, demonstrating that global alignment can effectively aid spatial tasks. The study tackles the difficulty of identifying sound sources from temporally synchronized audio-visual data without requiring pixel-level supervision.