Mixture of Heterogeneous Grouped Experts for Efficient Language Modeling

A recent paper published on arXiv introduces the Mixture of Heterogeneous Grouped Experts (MoHGE) to tackle the shortcomings of traditional Mixture-of-Experts (MoE) models in Large Language Models (LLMs). Conventional MoEs impose uniform sizes on experts, leading to inflexibility that does not match computational demands with the varying complexity of tokens. Although heterogeneous expert designs aim to diversify expert sizes, they struggle with uneven GPU usage and poor parameter efficiency. MoHGE features a two-tier routing system for adaptable, resource-conscious expert combinations and suggests a Group-Wise Auxiliary Loss to effectively direct tokens to the most efficient experts, enhancing inference performance. The paper can be found on arXiv with the ID 2604.23108.