Horizontal Scaling Laws for Mixture-of-LLMs
A new arXiv paper from 2607.22577 proposes cMoLLM, a method to scale large language models by applying mixture-of-experts (MoE) style layers throughout the entire LLM pipeline, not just the feed-forward network. The authors reformulate MoE mixture layers as variable-kernel dynamic convolutions, where each expert is a 1x1 convolutional kernel and routing performs input-conditioned kernel aggregation. This approach aims to decouple capacity from computation, addressing the bottleneck of dense Transformers where every parameter is activated per token, causing linear cost growth. Prior pipeline-level approaches like ParaScale and AltUp suffered from overhead, homogenized routing, gradient collapse, limited adaptivity, and slow convergence. The paper establishes scaling laws for horizontal mixture of LLMs.
Key facts
- arXiv paper 2607.22577 introduces cMoLLM.
- cMoLLM applies MoE-style mixture throughout the LLM pipeline.
- Dense Transformers couple capacity and computation, activating all parameters per token.
- MoE mixture layers are reformulated as variable-kernel dynamic convolutions.
- Each expert corresponds to a 1x1 convolutional kernel.
- Routing implements input-conditioned kernel aggregation.
- Prior approaches ParaScale and AltUp had significant drawbacks.
- The paper focuses on horizontal scaling laws for mixture-of-LLMs.
Entities
Institutions
- arXiv