Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in MoE Models

A recent paper on arXiv (2608.07890) explores the potential of lightweight fine-tuning to pinpoint prunable experts within Mixture-of-Experts (MoE) models. These models, which are expansive language frameworks, activate only a fraction of their parameters for each token while necessitating the storage of all experts. The researchers introduce a technique that involves a brief fine-tuning phase using a parameter-efficient adapter, ranking experts based on the changes in the L2 router, and pruning the least altered experts in a single step. On Mixtral-8x7B-Instruct (44.83% MMLU-Pro), router-only LoRA, which utilizes just 0.002% of parameters, surpasses all-module LoRA with half the experts removed (27.54% versus 24.42%). As adaptation extends to attention and expert weights, signal quality diminishes. Accuracy consistently increases with LoRA rank, peaking at 28.76%. IA3, which keeps router weights static, achieves results comparable to direct router adaptation, while unconstrained additive adaptation negatively impacts the signal. The findings indicate that router sensitivity during lightweight fine-tuning can effectively reveal prunable experts, presenting a viable method to lessen storage demands in MoE models.