MemSpec: Memory-Aware Runtime for Adaptive Speculative Decoding on Edge Devices

A recent paper published on arXiv (2608.10362) presents MemSpec, a runtime system aimed at enhancing speculative decoding for large language models (LLMs) on edge devices with limited memory. This technique accelerates LLM inference by utilizing a lightweight draft model to predict several tokens, thereby minimizing costly decoding steps of the target model. The success of this method relies on the selection of drafts, with adaptive strategies that adjust draft models according to input and generation stages showing promise. However, these strategies often struggle to boost end-to-end throughput on memory-constrained devices due to the overhead from switching models. The authors highlight a significant issue: the disconnect between draft selection and availability under memory limitations. MemSpec resolves this by separating draft selection from execution via proactive management of the resident working set. A lightweight predictor assesses draft effectiveness based on the prompt and generation stage, determining which draft models to retain in memory. This strategy seeks to lower switching overhead and enhance throughput. The paper serves as a cross-type announcement, suggesting it may have been shared at a conference. This research is crucial for deploying LLMs on devices like smartphones and IoT gadgets, where memory is at a premium, and aims to improve practical efficiency, potentially leading to more agile AI applications on consumer hardware.