Muon-Optimized Transformers Exhibit Post-Grokking Collapse at Representation-Readout Interface

A recent preprint on arXiv (2608.07436) reveals that transformer models utilizing the Muon optimizer for modular addition tasks face a post-grokking collapse, resulting in the loss of previously achieved generalization. The research employs a configuration where Muon optimizes hidden matrices while AdamW focuses on embeddings and the output head. It shows that all nine setups for (a+b) mod 113 initially grok but subsequently lose their generalization. Among five seeds, the AdamW reference drops below the threshold in four instances, achieving only 27.59% accuracy. This instability is evident across various moduli, widths, training fractions, subtraction, and depth. The failure occurs at the representation-readout interface, which is only identifiable jointly through an invertible map not selected by the loss. After mastering the training set, the gradient diminishes to order 10^-6, with distinct responses from the optimizers: Muon exhibits a step-size elasticity of -0.03 compared to +1.5 for AdamW, and the Muon group progresses 8.0 times faster per parameter. Preventing failure can be achieved by freezing either group from bit-identical states, while freezing embeddings/readout eliminates the issue across five runs over 451,400 post-grokking steps and five p.