Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

A novel approach known as Autonomy-of-Heads (AoH) has been introduced to tackle the limitations in long-context LLM inference, which stem from the quadratic nature of attention calculations and escalating KV-cache expenses. Current techniques for sparse attention and KV-compression depend on real-time attention scores, observation windows, calibration prompts, or learned gates, which make head diagnosis reliant on input and expensive to implement. In contrast, AoH is a data-free technique that discerns retrieval and streaming heads through the spectral geometry of query-key projections. It formulates the kernel attention operator M_h = W_K^{hT}W_Q^h and employs its effective rank as a measure of head function: concentrated spectra signify retrieval heads, while diffuse spectra denote streaming heads. Additionally, the method provides an efficient computation for d_he, likely the effective rank. This research can be found on arXiv with the identifier 2608.06849.