LMO-IGT: Accelerating LMO-Based Optimization with Implicit Gradient Transport
A new research paper proposes LMO-IGT, a class of stochastic optimization methods that leverage implicit gradient transport to accelerate linear minimization oracle (LMO) based optimizers like Lion and Muon. The authors introduce a unified framework for stochastic LMO-based optimization and a new stationarity measure called the regularized support function (RSF), which bridges gradient-norm and Frank-Wolfe-gap concepts. By evaluating stochastic gradients at transported points, LMO-IGT aims to reduce computational overhead compared to variance reduction techniques that require additional gradient evaluations. The paper addresses fragmented theoretical understanding across unconstrained and constrained formulations. The work is published on arXiv under identifier 2605.05577.
Key facts
- LMO-IGT is a new class of stochastic LMO-based methods using implicit gradient transport.
- The paper proposes a unified framework for stochastic LMO-based optimization.
- A new stationarity measure, the regularized support function (RSF), is introduced.
- RSF bridges gradient-norm and Frank-Wolfe-gap notions.
- LMO-IGT evaluates stochastic gradients at transported points.
- The method aims to accelerate LMO-based optimizers like Lion and Muon.
- Variance reduction typically incurs computational overhead; LMO-IGT seeks to reduce it.
- The paper is published on arXiv with identifier 2605.05577.
Entities
Institutions
- arXiv