Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover
A new research paper on arXiv (2608.10545) proposes an importance-aware KV cache transfer method for multi-user edge LLM handover. The method prioritizes transmitting the most informative fraction of each user's KV cache, based on importance ordering, to maximize average accuracy across users under bandwidth constraints. The approach models the transfer as a multi-user backhaul allocation problem, using a sigmoid utility function that fits RULER benchmark measurements with R²>0.99. The proposed allocator aims to improve inference continuity during handovers in edge computing environments.
Key facts
- The paper is titled 'ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover'.
- It is available on arXiv with ID 2608.10545.
- The method addresses the challenge of preserving inference continuity when users hand over between edge nodes.
- Simultaneous handovers can saturate the backhaul, preventing full cache delivery.
- The approach orders each user's KV cache by importance and transmits only the most informative fraction.
- The transfer is cast as a multi-user backhaul allocation problem maximizing average accuracy.
- The utility function is a sigmoid that fits RULER benchmark measurements with R²>0.99.
- The proposed allocator keeps the concave region of the accuracy curve spanning nearly the entire cache.
Entities
Institutions
- arXiv