Depth-Dependent Sensitivity in Mixture-of-Experts Models: Qwen3.6-35B-A3B Analysis
A recent preprint on arXiv (2608.13565) details a sensitivity analysis conducted layer by layer on the Mixture-of-Experts (MoE) framework within the Qwen3.6-35B-A3B model, which consists of 40 MoE layers, each with 256 experts and utilizing top-8 routing. By employing magnitude-based expert masking, researchers evaluated how the removal of low-magnitude experts affected performance, using the XLCoST benchmark across 100, 300, and 500 prompts on three H100 GPU servers. Results show that the early layers (0-9) and middle layers (10-29) are sensitive to expert masking, whereas the later layers (30-39), particularly the very-late layers (35-39), can withstand significant masking, indicating opportunities for model compression. This research underscores the significance of MoE layer sensitivity for efficient implementation.
Key facts
- The study analyzes the Qwen3.6-35B-A3B model with 40 MoE layers, 256 experts per layer, and top-8 routing.
- Magnitude-based expert masking is used to assess layer sensitivity.
- The XLCoST cross-lingual code translation benchmark is used for evaluation.
- Evaluation scales include 100, 300, and 500 prompts.
- Three H100 GPU servers were used for the experiments.
- Early layers (0-9) and middle layers (10-29) are highly fragile to expert masking.
- Late layers (30-39) and very-late layers (35-39) tolerate aggressive masking of low-magnitude experts.
- The findings suggest depth-dependent sensitivity, with later layers being more compressible.
Entities
Institutions
- arXiv