Approximate Speculative Decoding: A New Verifier for Faster LLM Inference
A recent paper on arXiv presents Approximate Speculative Decoding (ASD), a verifier that does not require training and aims to enhance the speed of autoregressive generation in large language models. Unlike conventional speculative decoding, which checks a draft block against a target model in parallel but halts at the first token that diverges from the target's argmax, ASD employs a budgeted longest-prefix selection approach. It permits certain mismatches based on a local target-logit regret gate, a cap on per-block exceptions, and a sustained request-level regret budget. This innovation enables the reuse of a contiguous target-greedy suffix without needing extra approximate decisions or forward passes of the target model. No new draft model is necessary, making it a valuable improvement for current inference systems. The paper can be found on arXiv with the identifier 2608.03447.
Key facts
- ASD is a training-free verifier for speculative decoding.
- It uses budgeted longest-prefix selection instead of binary first-mismatch truncation.
- ASD accepts mismatches based on a local target-logit regret gate, per-block exception cap, and request-level regret budget.
- It reuses contiguous target-greedy suffixes without extra target-model passes.
- No new draft model is required.
- The paper is published on arXiv with ID 2608.03447.
- The method accelerates autoregressive generation.
- The approach is described in the paper's abstract.
Entities
Institutions
- arXiv