MACRO Optimizer Demystifies Manifold Constraints in LLM Pre-training

A recent study published on arXiv (2605.04418) thoroughly investigates how explicit manifold constraints influence the pre-training of large language models. The research presents MACRO (Msign-Aligned Constrained Riemannian Optimizer), a single-loop optimization framework that guarantees convergence while separating weight regularization strategies from processes such as RMS normalization and decoupled weight decay. Both theoretical insights and empirical assessments demonstrate that manifold constraints effectively regulate forward activation scales and maintain stable rotational equilibrium, surpassing the benefits of conventional stabilization methods. This study elucidates the reasons behind the enhancement of numerical stability and performance through the use of constraints, moving past heuristic approaches.