Stability of Transformers under Layer Normalization
A new arXiv paper (2510.09904v2) investigates the stability of deep Transformers under various layer normalization placements. The study provides a theoretical analysis of both forward (hidden states) and backward (gradient) stability, offering insights into training dynamics. It derives explicit bounds on hidden state growth and explains how layer normalization affects gradient backpropagation, guiding the scaling of residual steps to improve stability. The paper addresses the ad-hoc placement of layer normalization, a common issue in Transformer training.
Key facts
- Paper arXiv:2510.09904v2
- Announce Type: replace-cross
- Study on forward and backward stability of Transformers
- Layer normalization placement affects training stability
- Explicit bounds on hidden state growth derived
- Analysis of gradient backpropagation under layer normalization
- Guidance for scaling residual steps in Transformer blocks
- Published on arXiv
Entities
Institutions
- arXiv