Study: Gradient Heterogeneity Explains Adam's Edge Over SGD in Transformer Fine-Tuning
New research posted on arXiv (2502.00213v5) addresses why adaptive optimizers outperform SGD when training Transformers. The authors examine the fine-tuning setting and focus on gradient heterogeneity, defined by how gradient-norm values differ between parameter blocks, alongside Hessian heterogeneity. Their theoretical analysis shows that both kinds of variation can hamper convergence for gradient-based approaches such as SGD, whereas sign-based methods prove substantially less vulnerable. Adam and SignSGD both rely on coordinate-wise update rules, meaning they are less affected by the scale of any individual gradient coordinate. The paper uses SignSGD as a tractable stand-in for Adam-like behavior. Prior work had left the precise source of Adam's advantage unresolved. This replace-cross preprint contributes a formal explanation built around blockwise variation in gradients.
Key facts
- Paper ID: arXiv:2502.00213v5; announce type: replace-cross.
- Investigation targets Transformer optimization in the fine-tuning setting.
- Gradient heterogeneity is described as differences in gradient norms across parameter blocks.
- The analysis also incorporates Hessian heterogeneity.
- Theory predicts that gradient and Hessian heterogeneity degrade convergence of SGD.
- Sign-based methods are much less sensitive to this degradation.
- Adam and SignSGD use coordinate-wise rules and are less sensitive to individual coordinate scale.
- SignSGD serves as a computationally tractable proxy for Adam-like algorithms.
Entities
Institutions
- arXiv