TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
A novel framework named TaskPress has been unveiled to tackle the issue of long-context inference in large language models, which faces limitations due to the linear expansion of the key-value (KV) cache relative to sequence length. Unlike conventional pruning techniques that assess the significance of tokens for specific queries—without the ability to generalize to new queries—TaskPress develops a reusable memory representation based on a high-level task guide. This guide serves as a meta-query during the prefill stage, enabling the filtration of irrelevant tokens prior to the execution of downstream queries. Furthermore, TaskPress utilizes quantization scale factors as a no-cost indicator for identifying significant representation outliers, thus offering an effective measure for token relevance. Experiments across various long-context tasks reveal that TaskPress successfully generates a compact, reusable cache. The research can be found on arXiv with the identifier 2608.03276v1.
Key facts
- TaskPress is a framework for task-guided, query-agnostic KV cache eviction.
- It uses a high-level task guide as a meta-query during prefill to filter irrelevant tokens.
- It leverages quantization scale factors as a zero-cost signal for detecting influential representation outliers.
- Experiments on various tasks with long context input demonstrate efficiency.
- The paper is announced on arXiv with identifier 2608.03276v1.
- The approach addresses the linear growth of KV cache with sequence length.
- Traditional pruning methods are query-specific and not reusable across unseen queries.
Entities
Institutions
- arXiv