VoxZip: Semantic-Anchored KV Cache Compression for Long-Context Audio Inference

A recent study published on arXiv (2608.08569) presents VoxZip, a two-stage framework that operates without training to compress key-value (KV) caches in Speech Large Language Models (Speech LLMs), facilitating efficient inference for long-context audio. The initial stage utilizes automatic speech recognition (ASR) transcriptions as semantic anchors to temporally align, compress, and merge audio tokens, thereby decreasing the KV cache size while enhancing token information density. The subsequent stage implements a dynamic filtering technique that relies on temporally decayed accumulated attention to remove non-essential tokens, reducing early-token bias. This approach tackles the challenge of high KV cache memory requirements in long-context audio tasks, which traditional text-focused compression methods struggle with due to disruptions in speech continuity or loss of semantic information. The authors of the paper are researchers who have shared their findings on arXiv, a preprint platform, indicating that it has not yet been peer-reviewed. This research is crucial for improving the applicability of Speech LLMs in real-world scenarios that demand long audio contexts, including meeting transcription, audio summarization, and voice assistant technologies.