LorExperts: A New Method for Compressing Mixture-of-Experts Models

A recent study published on arXiv (2608.07814) presents LorExperts, a novel approach to compressing Mixture-of-Experts (MoE) language models. While MoE models provide substantial capacity with minimal per-token computation, their numerous expert weight matrices can lead to high deployment costs. Current compression methods, such as expert pruning (e.g., REAP) and merging, lower expenses but often compromise accuracy and necessitate router retraining. Although low-rank delta decomposition (e.g., D^2-MoE) maintains all experts and the router, its effectiveness diminishes as the number of experts increases due to the limitations of a single shared component. The researchers demonstrate that MoE expert weights are nearly orthogonal, which hampers scaling. However, experts form functional co-activation communities independent of weight similarity. LorExperts addresses this by clustering experts, retaining one full-precision dominant expert per cluster, and using low-rank updates for the others. This method, which preserves the router, seeks to minimize memory usage without sacrificing accuracy. The authors announced this work as a cross-type submission on arXiv, highlighting its importance for deploying large language models in environments with limited resources.