TAM: Thought-Aware KV Cache Compaction for Reasoning Models

A novel approach known as Thought-Aware Attention Matching (TAM) tackles the memory limitations in reasoning language models by leveraging the hierarchical nature of chain-of-thought (CoT) sequences. In contrast to current compaction techniques that view reasoning paths as flat token sequences with uniform compression, TAM breaks down the trajectory into reasoning blocks through thought segmentation. It allocates compression budgets adaptively, considering the significance and size of each segment, while safeguarding crucial high-attention tokens. The authors demonstrate that the allocation strategy is optimal within a convex error framework and that cumulative errors from sequential compaction remain limited. Experiments were carried out on the AIME 2024 and MATH datasets, though specific findings are not included in the abstract. The paper can be found on arXiv with the identifier 2608.12331.