KV Cache Quantization as Transform Coding with Attention-Preserving Transforms

A recent paper published on arXiv introduces a technique for quantizing the key-value (KV) cache during the decoding process of long-context large language models (LLMs), tackling the challenges associated with bandwidth limitations in cache loading. The authors present KV cache quantization as a transform coding issue, where the goal is to maintain attention product integrity. They critique current methods that utilize orthogonal transforms, such as random or Hadamard rotations, for being less effective due to their focus on equalizing variances rather than concentrating energy, and their reliance on fixed-width scalar quantizers that fall short at lower bit rates. Their innovative approach derives transforms based on a distortion criterion, seeking to minimize cache size while preserving attention accuracy with a consistent bit count per token. At two bits per element, this method aspires to surpass both data-oblivious and query-statistics-based techniques. The paper can be found on arXiv under the identifier 2608.04074.