Dion3 Optimizer Cuts Muon Overhead by 6x

A new optimizer called Dion3, detailed in a preprint on arXiv (2608.11612), significantly reduces the computational and communication overhead of the Muon optimizer. Muon, known for its effectiveness, suffers from a cubic-time Newton-Schulz orthogonalization step, which becomes especially costly when weights are sharded across devices. Dion3 addresses this at multiple levels: it introduces a Gram Newton-Schulz algorithm to lower FLOP costs, uses CuteDSL kernels that exploit symmetry for faster computation, and employs a megabatching strategy to reduce communication overhead. Additionally, the update rule is modified to orthogonalize only a fraction of the momentum matrix's rows each step, cutting costs further. This approach improves upon a previous compressed version called Dion, achieving better speed and performance. Overall, Dion3 matches or improves the loss achieved by Muon while reducing optimizer step time by up to 6 times. The method is available via the provided link.