CMuon Optimizer Enhances Diffusion Transformer Training Efficiency
A new optimization method called Chunked Muon (CMuon) has been developed by researchers to enhance and stabilize the training process of Diffusion Transformers (DiTs), which are leading the way in visual generative modeling. This innovative approach is detailed in a paper available on arXiv (ID: 2608.02502). The study tackles a significant issue in DiT training: the use of the Momentum Orthogonalization (Muon) optimizer, which, while a viable alternative to AdamW, results in less effective convergence in later stages. The authors pinpoint the problem to the merging of distinct weights within AdaLN and QKV layers into single tensors, causing implicit subspace coupling that disrupts optimization. CMuon resolves this by dividing these matrices into separate components before orthogonalization. Extensive tests show that CMuon markedly enhances training efficiency and stability, particularly with a 675M-parameter model. This breakthrough could lower the computational demands of training large generative models, increasing their accessibility. The paper's authors are linked to an undisclosed institution, and its announcement occurred on an unspecified date.
Key facts
- CMuon is a new optimizer for Diffusion Transformers (DiTs).
- It addresses suboptimal late-stage convergence when using Muon optimizer on DiTs.
- The root cause is implicit subspace coupling from fused weight tensors.
- CMuon partitions matrices into independent sub-components before orthogonalization.
- Experiments show improved training efficiency and stability.
- A 675M-parameter model demonstrated notable gains.
- The paper is available on arXiv with ID 2608.02502.
- The research aims to reduce computational cost of training large generative models.
Entities
—