UniMoMo: Post-Training MoE Compression for Recommendation Models

A recent study presents UniMoMo, a framework designed for post-training compression of sparse mixture-of-experts (MoE) layers within extensive recommendation models. This approach tackles the challenge of transforming a trained checkpoint featuring a complete expert bank into a more compact standard MoE that adheres to a specified expert budget, all without introducing an online module dedicated to compression. UniMoMo treats this as a constrained graph coarsening issue, organizing experts by their functional similarities instead of their parameter distances. An unlabeled calibration set assesses how similarly experts react to common recommendation states. To mitigate performance loss, a layer-adaptive protection mechanism limits the merging of frequently accessed experts based on their routing exposure. The framework was tested on the Amazon Beauty, KuaiRec, and TenRec datasets, utilizing 2, 4, and 6 MoE blocks and achieving a final configuration of four experts. The paper can be found on arXiv under the identifier 2608.08627.