Training-Free Speculative Decoding Enhanced with Semantic Drafting

A new arXiv paper (2608.03839) proposes a method to improve training-free speculative decoding by addressing a failure in exact-match drafting. The authors diagnose that exact-match drafters, which work by matching suffixes of the current context against earlier contexts, often miss correct drafts already present in the pool, especially in tool-calling traffic where requests repeat most content but vary a few values. They find that about half of what the strongest exact-match drafter misses on their densest tool-calling benchmark is actually in the pool but unreachable by exact matching. To solve this, they introduce a second, semantic draft source: the same pool re-keyed by the hidden state that the verifier has already computed at each committed token, merged to work within an existing lexical drafter's tree. The method is tested on three published drafters at matched pool and budget, showing improvements. The paper is available on arXiv.