GO-MUON Optimizer Improves Muon with Data-Dependent Geometry
The recent paper on arXiv, identified by ID 2608.09763, presents GO-MUON, an innovative optimizer leveraging Muon technology. GO-MUON uniquely incorporates a matched data-dependent geometry, which is utilized across multiple optimization steps. It excels in polar updates for unweighted spectral geometry and provides exact solutions for weighted spectral oracles. The findings indicate that the observed-label backward factor for softmax cross-entropy aligns closely with the model Fisher and generalized Gauss-Newton factor. Additionally, a four-step refresh effectively maintains tracking delay in slowly changing geometries, while lazy geometry is framed as a compute-statistics tradeoff.
Key facts
- Paper introduces GO-MUON, an optimizer based on Muon.
- GO-MUON uses a matched data-dependent geometry reused across optimization steps.
- Muon's polar update is exact for unweighted spectral geometry.
- GO-MUON's raw update exactly solves the weighted spectral oracle.
- For softmax cross-entropy, the observed-label backward factor approaches the model Fisher and generalized Gauss-Newton factor.
- Four-step refresh nearly preserves tracking delay of slowly changing geometry.
- Lazy geometry is a compute-statistics tradeoff, not a denoising mechanism.
- Paper is on arXiv with ID 2608.09763.
Entities
Institutions
- arXiv