MLA Bottleneck Separates Content from Position in LLMs
A recent study on mechanistic interpretability has uncovered that Multi-head Latent Attention (MLA), which was presented in DeepSeek-V2, effectively compresses key-value pairs using a common low-rank bottleneck (cKV), resulting in an 81% reduction of KV-cache during inference. The research involved training a transformer with 114 million parameters on a mix of web, code, and math data, followed by fine-tuning on TinyStories. Through techniques like SVD, attention head taxonomy, linear probing, and disruption-attribution, it was found that the cKV bottleneck retains a pure content representation with 98% entity identity retention, while eliminating positional data, confirming MLA's ability to separate content from position via RoPE. Induction heads cluster within a single layer. This marks the first detailed analysis of the information preserved or discarded by the bottleneck and its impact on transformer circuits.
Key facts
- MLA compresses key-value pairs through a shared low-rank bottleneck (cKV).
- Achieves 81% KV-cache reduction during inference.
- Study trained a 114M-parameter transformer.
- Model pretrained on web/code/math mixture, fine-tuned on TinyStories.
- cKV bottleneck preserves entity identity with 98% retention.
- cKV bottleneck discards positional information.
- Induction heads co-locate at a single layer.
- First comprehensive mechanistic interpretability study of MLA.
Entities
Institutions
- DeepSeek