MXFP4 Pretraining on Native FP4 Hardware: Wgrad Quantization Key to Convergence

A recent paper on arXiv (2605.09825v4) explores the reasons behind the divergence observed in full-pipeline FP4 training of large language models, even when forward activations and activation gradients remain stable. This research, presented as a replace-cross announcement, systematically implements MXFP4 quantization across various transformer training components—forward propagation (Fprop), activation gradients (Dgrad), and weight gradients (Wgrad)—while controlling other variables. In the full pretraining of Llama 3.1-8B using the C4 dataset, the authors identify Wgrad quantization as the main factor leading to convergence issues, whereas Fprop and Dgrad quantization only slightly raises token demands. Their evaluation of structured and stochastic methods reveals that while stochastic rounding and randomized Hadamard rotations do not stabilize training post-Wgrad quantization, deterministic Hadamard rotations successfully restore convergence. This study sheds light on the complexities of low-precision training and proposes a viable approach for effective FP4 pretraining on native hardware.