TEMPO: New Load Balancing for Expert-Parallel MoE Serving

A new research paper on arXiv (2608.13057) introduces TEMPO, a makespan-aware expert-parallel load balancing method for Mixture-of-Experts (MoE) serving. The paper challenges existing load balancing approaches by showing that expert execution time is not linear in token count or activated-expert count alone. Measurements on two datacenter GPU generations reveal two distinct regimes: below approximately 156–168 tokens, HBM weight streaming dominates, making cost proportional to activated replicas; above that, grouped GEMM rounds tokens to 128-tile M-tiles, so splitting an expert adds padded compute. The authors propose a max-affine profile t = max(a + bG, c + βN) to capture both regimes. They formalize per-batch dispatch as a fixed-charge makespan problem, which is NP-hard on two machines. The paper demonstrates that proxy dispatches can differ by 1.4–1.6× in modeled block time (p95 up to 1.7×), and the optimal proxy depends on the regime. The work is relevant to improving efficiency in large-scale AI serving systems.