MESH: Memory-Efficient Sinkhorn Optimization for MoE Training
A recent study published on arXiv (2608.04407) presents MESH, a hidden-momentum Sinkhorn update aimed at enhancing memory-efficient optimization in Mixture-of-Experts (MoE) training. Conducted within a controlled environment using a 110M-parameter nanowhale DeepSeek-style MoE pretraining framework, the research revealed that applying Sinkhorn gradient descent directly to MoE training proved inconsistent. A combination of SAGE and Sinkhorn reduced the optimizer's state from 0.883GB to 0.331GB, yet led to an increase in evaluation loss to 3.8265, in contrast to AdamW baselines ranging from 3.58 to 3.64. The study pinpointed routed MoE expert matrices as the main failure source due to their conditional and temporally fluctuating gradients. MESH addresses this by restoring a temporal first-moment signal throughout the gradient-buffer lifecycle without requiring the expert first moment to be stored as optimizer state, and it includes a block-preconditioning option. This paper is also a cross-announcement and can be accessed on arXiv.
Key facts
- Paper: arXiv:2608.04407
- MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
- 110M-parameter nanowhale DeepSeek-style MoE pretraining setting
- SAGE/Sinkhorn hybrid reduces optimizer state from 0.883GB to 0.331GB
- Evaluation loss degrades to 3.8265 with hybrid, vs AdamW baselines 3.58–3.64
- Routed MoE expert matrices are the dominant failure point
- MESH uses hidden-momentum Sinkhorn update
- MESH restores temporal first-moment signal without storing expert first moment as optimizer state
Entities
Institutions
- arXiv