Attention Sinks Induce Gradient Sinks in Transformers

A new study on arXiv (2603.17771) investigates the relationship between attention sinks and massive activations in Transformer models, focusing on backpropagation. The authors show that under causal masking, attention sinks cause gradient concentration, termed gradient sinks. They argue that massive activations act as adaptive regulators of gradient pressure during training, as the RMSNorm Jacobian attenuates gradients inversely with input norm. The study predicts that reducing sink-induced gradients weakens massive activations, and proposes V-scale, a modification that adjusts backpropagated gradients to test this. The work provides a theoretical and empirical explanation for these phenomena, previously understood mainly from the forward pass perspective.