AgentKVShift: Training-Free KV Cache Reuse for Agentic Memory Systems

The recently introduced AgentKVShift technique tackles the significant inference expenses associated with memory-augmented LLM agents by reusing Key-Value (KV) cache states. These agents utilize structured memory units enriched with LLM-generated metadata, such as summaries and tags, to sustain context throughout numerous interactions. Each retrieval necessitates a complete re-encoding of these units into KV states, which significantly increases prefill latency. Current training-free KV reuse strategies, intended for RAG-style raw passages, perform poorly with structured agentic memories. AgentKVShift features a probe-guided KV residual correction that functions for each retrieved memory unit. It reveals that the residuals from per-memory KV reuse can be broken down into a shared memory-level offset and minor token-wise variations. By estimating this offset from a small probe set, it enables the correction of all reused KV states, thereby minimizing computational demands without the need for retraining. This method is detailed in a paper available on arXiv (2607.21604).