Random Matrix Theory Reveals Dominant Learned Structure in Transformer Attention
A recent preprint on arXiv (2608.07921) utilizes Marchenko-Pastur (MP) random matrix theory to analyze pre-trained attention weights in transformers, distinguishing each projection matrix into a random-like bulk and distinct spectral outliers. The authors demonstrate this causal decomposition: when the MP-identified outliers (signal) in Mistral-7B are set to zero, performance on benchmarks such as HellaSwag, MMLU, and PIQA drops to near-random levels, while zeroing a matched subset of bulk singular values results in minor but notable performance decline. Among 11 pre-trained transformers, five consistent patterns emerge: spectral outliers represent a key aspect of learned structure; Q projections exhibit the most outliers; V projections under grouped-query attention show poor signal/noise differentiation; entry-level outliers create structured row-bands in Q and column-bands in O; and certain residual-stream dimensions remain as band outliers in K and O across layers. The paper concludes by discussing how these insights may enhance future research on model interpretability and compression. The study is also cross-listed and accessible via the provided arXiv link.
Key facts
- Applies Marchenko-Pastur random matrix theory to pre-trained attention weights
- Separates each projection matrix into a random-like bulk and spectral outliers
- Zeroing MP-identified outliers in Mistral-7B drives HellaSwag, MMLU, and PIQA to near random-chance performance
- Zeroing bulk singular values causes smaller but non-negligible degradation
- Analysis covers 11 pre-trained transformers
- Identifies five recurring patterns in spectral outliers
- Q projections carry the most outliers
- V projections under grouped-query attention lack clean signal/noise separation
- Entry-level outliers form row-bands in Q and column-bands in O
- Specific residual-stream dimensions persist as band outliers across layers in K and O
Entities
Institutions
- arXiv