HiKV: Hardware-Accelerated KV Cache Compression for LLMs
A new method called HiKV has been introduced by researchers for compressing key-value caches in large language models (LLMs) with a focus on hierarchical importance. The method functions in two phases: the first phase removes less important tokens while adhering to a set budget, and the second phase selectively loads only the crucial components of each token that remains, allowing for compression ratios that are not possible with a single granularity approach. An innovative accelerator featuring a reconfigurable importance sorter integrates both phases into a single circuit. When tested on standard LLMs, HiKV demonstrates a speed increase of up to 7.95 times and a 90% reduction in energy consumption during attention computations.
Key facts
- HiKV is a novel algorithm-hardware co-design for KV cache compression.
- It operates at two granularities: token eviction and element selection.
- A reconfigurable importance sorter switches between two sorting datapaths.
- Achieves up to 7.95x speedup and 90% energy reduction.
- Published on arXiv with ID 2607.22389.
- Targets long-context LLM decoding memory bottleneck.
- Stage I evicts unimportant tokens within a fixed budget.
- Stage II loads only significant elements of each retained token.
Entities
Institutions
- arXiv