Post-Norm Outperforms Pre-Norm in Curriculum Depth Growing for LLMs

A new study from arXiv (2608.13156) challenges the conventional preference for pre-normalization (pre-norm) in Transformer architectures, particularly when models are trained via curriculum depth growth. The research, which uses a Qwen3-8B teacher and a nine-layer student model, finds that while pre-norm and post-norm perform nearly identically under joint training (differing by only 0.0004 in validation cross-entropy), post-norm shows a significant advantage under curriculum growth, improving by 0.0328 over pre-norm—an order of magnitude larger difference. The study also includes a post-joint control matched by student active-layer tokens, which remains worse than post-grow, ruling out compute as the sole explanation for the improvement. The authors suggest that normalization placement interacts with the training curriculum, and that post-norm may be more effective when depth is introduced progressively. The findings have implications for the design of large language models, potentially influencing future training strategies. The paper is available on arXiv under the identifier 2608.13156.