Gradient SVD Reveals Linear-Centroid Coupling in Transformer Training

A recent study available on arXiv (2604.25143) indicates that applying singular value decomposition (SVD) to loss gradients, rather than AdamW updates, uncovers a significant relationship between SED directions and Linear Centroid Hypothesis (LCH) features in transformer models. The observed perturbative coupling escalates from 3–9× to 100–330× across four distinct single-task modular arithmetic operations, removing any visible dependence on operations. In a multitask transformer utilizing a shared encoder, update-based SED resulted in R_k ≤ 1, indicating a diagnostic issue, while per-operation gradient-based SED achieved R_k = 20–45× across all tasks. The primary challenge identified is gradient aggregation across competing tasks, which is addressed by conducting SVD on per-task gradients. A causal intervention reveals that limiting attention updates to a specific subspace can influence coupling. This research was published on April 25, 2026.