OnlineSPEC: Evolving Draft Models via Online Learning for Speculative Decoding

A new framework, OnlineSPEC, has been proposed to enhance speculative decoding in large language model inference. Speculative decoding is a widely used method to accelerate inference, where a lightweight draft model generates candidate tokens that are verified in parallel by a larger target model. However, the draft model often fails to approximate the target distribution due to limited capacity, leading to shorter acceptance lengths and reduced speedup. The key insight of OnlineSPEC is that the verification feedback inherent in speculative decoding quantifies the deviation between draft and target models at no extra cost. This feedback forms an iterative loop—draft commits, feedback provides, draft adapts—which aligns with the online learning paradigm. OnlineSPEC is a unified framework that systematically leverages this interactive feedback to continuously evolve draft models. The framework is grounded in dynamic regret analysis, providing theoretical guarantees. The paper is available on arXiv under the identifier 2603.12617, with the announcement type 'replace-cross'. The work was announced on arXiv, and the abstract outlines the motivation and approach, though the full details of the method and experimental results are not included in the provided content. The research addresses a critical bottleneck in speculative decoding, offering a potential path to more efficient LLM inference.