KV-Rescue: A Training-Free Framework to Recover Lost Accuracy in KV-Cache Eviction for Long Reasoning Traces

A recent paper on arXiv (2608.15797) presents KV-Rescue, an inference framework that operates without training to address the information loss stemming from KV-cache eviction in large language models dealing with extensive reasoning tasks. The authors describe the impact of eviction as an information gap rather than a capability gap, highlighting that a 7B model that has been evicted and a full-context 1.5B model produce complementary errors. By selecting the best responses from both models, 79% of the accuracy deficit relative to the full-KV 7B model can be recovered. KV-Rescue integrates reasoning steps from both models into a unified trajectory, utilizing a lightweight full-context helper and an online detector to compensate for the lost context. This research is crucial for managing the memory demands of long reasoning traces in production settings, and its training-free nature allows for immediate application to current models without the need for fine-tuning. The results indicate that losses from eviction can be significantly mitigated using a small helper model, providing a viable approach for implementing large reasoning models within memory limits.