S2-MoE: Efficient Self-Speculative Decoding for MoE on Edge Devices
A novel approach named S2-MoE has been presented in an arXiv paper (2608.15018) to enhance the efficiency of inference for large language models (LLMs) on edge devices. This technique integrates self-speculative decoding with a Mixture-of-Experts (MoE) framework, tackling issues related to memory and bandwidth limitations. By utilizing routing-aware adaptive speculative expansion, S2-MoE minimizes verification overhead, while reuse-aware expert gating boosts verification efficiency. Additionally, it synchronizes draft and target execution through shared context. When implemented in llama.cpp, this method demonstrates a speed increase of up to 5.3 times (approximately 2.0 times on average) compared to standard autoregressive decoding across various MoE models and datasets on edge devices. The code is accessible on GitHub.
Key facts
- S2-MoE is a self-speculative decoding framework for MoE inference on edge devices.
- It reduces redundant verification through routing-aware adaptive speculative expansion.
- It improves verification efficiency with reuse-aware expert gating.
- It aligns draft and target execution via shared context.
- Implemented in llama.cpp.
- Achieves up to 5.3x speedup (about 2.0x on average) over standard autoregressive decoding.
- Tested on diverse MoE models and datasets on edge devices.
- Code available at https://github.com/angerybob/
Entities
Institutions
- arXiv
- llama.cpp