Scale Vectors in LLMs: Negligible Yet Critical for Training

A new study from arXiv systematically analyzes scale vectors in large language models (LLMs), revealing their outsized role in training despite comprising a negligible fraction of parameters. The research shows that removing scale vectors significantly degrades pre-training performance. In Pre-Norm architectures, scale vectors do not increase expressivity but improve optimization via a self-amplifying preconditioning effect on subsequent linear mappings. The study also examines the role of weight decay for scale vectors, distinguishing between Input-Norm and Output-Norm layers. This work provides theoretical and empirical insights into a previously poorly understood component of LLMs.