Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023)

The primary research paper behind speculative-decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias (Google), ICML 2023 Oral (arXiv:2211.17192, first posted Nov 2022). The speculative-decoding page is anchored to a Hugging Face explainer (speculative-decoding cites the assisted generation blog); this page anchors the method to its peer-reviewed origin and the correctness argument that makes it a free speedup.

The two observations it rests on

Hard tasks contain easy subtasks. Much of what a large model generates (whitespace, obvious continuations) can be approximated well by a far cheaper model.
Parallel verification is nearly free. Because decode is memory-bandwidth-bound, not compute-bound, the large model can score several draft tokens in one forward pass for roughly the cost of scoring one — so the wins from a good draft come at little extra compute.

Speculative sampling — why the distribution is exact

The contribution is a novel sampling method (“speculative sampling”) that lets a draft model propose tokens and a target model accept or reject them while keeping the output distribution identical to sampling from the target alone. Each draft token is accepted with a probability tied to the ratio of the two models’ token probabilities; on rejection the token is resampled from an adjusted distribution. The net effect is provably the same distribution as the target model — so it is an exact accelerator, not an approximation. Requires no retraining and no architecture change: it wraps off-the-shelf models.

Numbers

2×–3× wall-clock acceleration on T5-XXL versus the standard T5X implementation, with identical outputs. The gain scales with how often draft tokens are accepted, which is highest when the draft model approximates the target well.

Why it matters for the spoke

This is the peer-reviewed primary the spoke’s speculative-decoding page leaned on via a vendor blog. It supplies the formal correctness argument (distribution-preserving) and a neutral benchmark, and it is the concrete instance of the synthesis’s sampling × serving coupling — the draft model couples token-sampling to the serving loop. Chen et al.’s contemporaneous DeepMind paper (Accelerating LLM Decoding with Speculative Sampling) reached the same result independently; this Google paper is the ICML-published anchor.

Tier

T1 — peer-reviewed primary (ICML 2023 Oral) with a formal distribution-preservation argument and a reproducible T5-XXL benchmark. Figures here are from the abstract; the arXiv PDF body was not fully extracted.

speculative-decoding · token-sampling · prefill-decode-kv-cache · llm-inference · vllm