APEX: Adaptive Expert Prefetching for Edge MoE Inference

A newly developed framework known as APEX (Adaptive Expert Prefetching) aims to tackle memory constraints in Mixture-of-Experts (MoE) models utilized on edge devices. While MoE models are favored for edge applications because of their substantial capacity and computational efficiency, the large expert parameters typically stored in off-chip memory can hinder expert loading. APEX functions as a predictive resource management system that synchronizes expert loading with productive computation. It features a lightweight prefetch router that anticipates candidate experts prior to the attention block, dynamically retrieving additional experts based on a learned confidence model. This adaptive approach achieves over 99% overlap accuracy, surpassing traditional fixed top-k prefetching methods. The framework, which includes two execution modes—one that ensures correctness and another that may sacrifice accuracy for speed—is discussed in a paper available on arXiv (2608.11688), emphasizing its significance in edge AI and efficient inference.