RetroAttention: KV Cache Update for Long-Context LLMs
A new technique called RetroAttention has been introduced by researchers to enhance the efficiency of large language models (LLMs) in generating long contexts. This approach tackles the memory limitations posed by the Key-Value (KV) cache, which increases linearly with the length of the sequence and significantly impacts decoding speed. While current KV cache compression techniques concentrate on input contexts, they do not address the accumulation of attention errors during prolonged decoding. RetroAttention revises earlier attention outputs by utilizing newly generated KV entries from later decoding phases, keeping a lightweight output cache that adds minimal latency. This innovation disrupts the fixed-attention-output model, allowing for more effective long-context inference. The findings are available on arXiv under ID 2508.09001.
Key facts
- RetroAttention is a KV cache update technique for LLMs.
- It addresses memory bottlenecks in long-context generation.
- Existing compression methods ignore cumulative attention errors during decoding.
- RetroAttention revises past attention outputs with new KV entries.
- It uses a lightweight output cache for efficiency.
- The technique incurs minimal latency overhead.
- The paper is on arXiv: 2508.09001.
- It targets tasks like reasoning, code generation, and multi-turn dialogue.
Entities
Institutions
- arXiv