Dense2MoE: Unified Pruning and Upcycling for Efficient On-Device LLMs

Researchers propose Dense2MoE, a framework that combines pruning and upcycling to create efficient Mixture of Experts (MoE) models for on-device deployment. The method, called Layer Fusion UpCycling (LF-UC), prunes bandwidth-heavy attention modules from redundant layers and repurposes their MLPs as MoE experts. This preserves core model capabilities while limiting active parameters via selective token routing. Dense2MoE is guided by hardware Roofline theory to overcome the inference memory wall. The approach addresses the trade-off between parameter redundancy and model accuracy, achieving better Pareto frontier for on-device LLMs.