OptR: Output-Aware Rotation for INT2 KV-Cache Quantization
A novel technique named OptR, as presented in arXiv paper 2608.02691, tackles the limitations of memory and bandwidth in key-value (KV) caches during long-context large language model inference. Previous INT2 quantization methods based on rotation focused on optimizing cache statistics or proxy errors prior to the full attention readout. However, these methods still suffer from errors propagated through attention and the output projection W_O. OptR addresses this by breaking down post-W_O attention-output error into key- and value-induced components, learning orthogonal corrections per head via the complete INT2 quantization and attention process. Additionally, it employs a key reparameterization equivalent to attention, minimizing large channel-wise offsets while preserving the softmax distribution. The approach was evaluated on three models and five reasoning and coding benchmarks, demonstrating enhancements over current techniques.
Key facts
- OptR is an output-aware rotation method for INT2 KV-cache quantization.
- It minimizes post-W_O attention-output error.
- It decomposes the error into key- and value-induced terms.
- It learns per-head orthogonal corrections through full INT2 quantization and attention path.
- It applies attention-equivalent key reparameterization to reduce channel-wise offsets.
- Tested across three models and five reasoning and coding benchmarks.
- Paper available on arXiv with ID 2608.02691.
Entities
Institutions
- arXiv