vToken: Token-Level Virtualization for Reclaimable KV Caches

A recent study published on arXiv (2608.13263) presents vToken, a streamlined token-level virtualization layer aimed at mitigating memory inefficiencies in large language model (LLM) serving. The key-value (KV) cache, which holds attention-related key-value pairs, expands with both sequence length and batch size, leading to significant memory constraints. While existing solutions like PagedAttention utilize fixed-size memory blocks to minimize allocator-level fragmentation, newer KV eviction strategies work at a more granular token level, resulting in intra-block fragmentation and leaving a substantial portion of allocated KV memory unrecoverable. vToken separates logical token liveness from physical block allocation via token-table indirection, allowing for asynchronous repacking of active tokens while keeping a consistent logical token view. This approach remains compatible with PagedAttention kernels and CUDA Graphs. The authors integrated vToken into vLLM and assessed it using eviction algorithms H2O, Random, and Scissorhands across multiple models. In comparison to a paired Naive-Evict baseline, vToken demonstrated a reduction in retained KV blocks, signifying enhanced memory efficiency. The paper is authored by researchers and is accessible on arXiv.