SKIP: Sparse Cross-Modal Routing for Efficient Multimodal QA
Researchers have introduced a new inference system named SKIP (Salient Knowledge-Injected Pathways) aimed at reducing the computational load in knowledge-intensive multimodal question answering (KI-MMQA). This innovative approach addresses issues related to long visual token sequences, extensive data retrieval from large databases, and the necessity for thorough cross-modal integration for each query, even when only a fraction of the visual and retrieved data is relevant. SKIP enhances efficiency through question-based sparse pathways, which include features like visual token pruning guided by questions, region-specific sparse retrieval, and a dynamic budget controller for verifying knowledge. The authors also present an information-bottleneck bound that shows how to achieve optimal visual sparsity. You can check out the paper on arXiv, reference 2607.25422.
Key facts
- SKIP is a unified inference architecture for KI-MMQA.
- It uses sparse pathways conditioned on question, image, and difficulty estimate.
- Components include visual token pruning, sparse retrieval, sparse cross-attention, and knowledge verification.
- An adaptive budget controller allocates compute based on predicted difficulty.
- An information-bottleneck bound shows optimal visual sparsity is achievable.
- The paper is on arXiv:2607.25422.
- The system reduces computational costs by processing only relevant content.
- It targets the intersection of long visual sequences, dense retrieval, and cross-modal fusion.
Entities
Institutions
- arXiv