QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

The newly introduced QKVShare framework facilitates the quantized KV-cache transfer among agents in multi-agent LLM systems operating on edge devices. It integrates mixed-precision allocation at the token level, a self-sufficient CacheCard representation, and a cache injection method compatible with HuggingFace. When tested on 150 GSM8K problems using Llama-3.1-8B-Instruct, adaptive quantization proves to be effective during repeated handoffs, particularly outperforming uniform quantization in deeper-hop and higher budget scenarios. Regarding handoff latency, QKVShare achieves a reduction in TTFT compared to complete re-prefill across all tested contexts: 130.7 ms versus 150.2 ms at a nominal 1K context and 397.1 ms versus 1029.7 ms at a nominal 8K context. Timing analysis indicates that latency is primarily influenced by post-injection generation rather than cache transfer.