KVDiagnosis: Benchmark for KV-Cache Compression in Long-Context LLMs
A new standard for diagnosing KV-cache compression in long-context language models has been unveiled, named KVDiagnosis. This benchmark overcomes the shortcomings of aggregate task scores by identifying which correct executions fail and the reasons behind these failures. It features a taxonomy of 25 methods categorized into five mechanism families, linked to eight verified implementations and their corresponding diagnostic metrics. Each method setting is assessed against a FullCache control for every fixed split, with FullCache-correct/compressed-wrong (C-to-W) rows chosen individually, preventing overlap in test sets. On Qwen3-8B, four evidence-aware workloads generated 59,800 compressed runs across 2,600 sources and 12,520 C-to-W rows, enhancing the understanding of KV-cache compression failures essential for optimizing memory efficiency in long-context scenarios.
Key facts
- KVDiagnosis is a diagnostic dataset and benchmark for KV-cache compression.
- It includes a 25-method taxonomy grouped into five mechanism families.
- Eight verified implementations are linked to the taxonomy.
- Evaluation uses per-source FullCache control and selects C-to-W rows separately for each method-setting.
- A common record format links outputs and run metadata to cache, likelihood, attention, and decoding measurements.
- On Qwen3-8B, four evidence-aware workloads yield 59,800 supported compressed runs over 2,600 sources.
- The benchmark produces 12,520 C-to-W rows.
- The work is available on arXiv with ID 2608.09412.
Entities
Institutions
- arXiv