Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

A recent paper on arXiv (2608.00574v1) presents a novel geometry-guided strategy for load balancing within vision-language mixture-of-experts (MoE) models. The researchers point out that the conventional token-level Switch auxiliary loss (Std-Aux) only manages the mixed load of image and text tokens, which can lead to significant errors being offset against one another. Their findings reveal that the same trained router exhibits over a fivefold variation in load imbalance when tested across various image resolutions. By maintaining fixed image and text load profiles, they derive the precise load curve as the token mix changes, highlighting that the gap between image and text loads influences sensitivity to the token mix. Additionally, they observe that physical preprocessing can alter conditional profiles, a factor overlooked by the fixed-profile law. To tackle this, they analyze the router input structure, discovering that image and text occupy separate areas, with visual tokens clustering strongly by source image. This distinction prompts the inclusion of separate image and text components in the auxiliary loss. Their geometry-guided approach seeks to enhance load balancing by acknowledging the unique contributions of image and text tokens, potentially improving the efficiency of training and inference in vision-language MoE models. The paper has been released on arXiv and is noted as a cross-type submission.