FreeBalance: Pre-Routing Online Load Balancing for MoE Inference

A recent study published on arXiv (2608.14205) presents FreeBalance, a technique designed for online load balancing during Mixture-of-Experts (MoE) model inference. This research tackles the issue of load imbalance in expert parallelism, where a heavily loaded rank can hinder global execution due to uneven routing distributions, leading to increased latency. While offline expert placement can help with persistent imbalance, real-world multi-task serving workloads demonstrate layer- and batch-dependent routing behaviors, highlighting the need for online load balancing. Current methods depend on routing statistics gathered post-routing, which delays expert weight load or migration until routing decisions are made, adding overhead to the inference process. The authors suggest that if routing distributions are accurately predicted beforehand, online balancing can occur simultaneously with earlier computations, thus minimizing latency. The paper is authored by a team of researchers and can be accessed on arXiv with the identifier 2608.14205.