Decoupled Attention Fusion: Accelerating RAG with Efficient KV Cache Reuse
A new approach named Decoupled Attention Fusion (DAF) has been introduced to minimize latency in Retrieval-Augmented Generation (RAG) for extensive language models. In lengthy context situations, RAG experiences significant Time-To-First-Token (TTFT) delays. The reuse of pre-calculated document KV caches leads to a mismatch in distribution, as offline caches do not capture inter-document attention patterns. While CacheBlend lessens recomputation through selective attention, it compromises accuracy in longer contexts. DAF addresses this by separating attention into three phases: self-attention for important tokens, self-attention for question-document interactions, and state fusion, successfully maintaining high accuracy while cutting down on recomputation costs.
Key facts
- Retrieval-Augmented Generation (RAG) mitigates hallucinations in LLMs
- RAG suffers from prohibitive Time-To-First-Token (TTFT) latency in long-context scenarios
- Reusing pre-computed document KV caches introduces a distribution mismatch
- Offline caches lack inter-document attention patterns required for coherent reasoning
- CacheBlend reduces recomputation via selective attention
- CacheBlend suffers severe accuracy degradation at longer contexts
- Decoupled Attention Fusion (DAF) maintains high accuracy while reducing recomputation overhead
- DAF decouples attention into three integrated stages
Entities
—