Speculative / assisted decoding
Speculative decoding (a.k.a. assisted generation) speeds up the latency-bound decode phase by having a small fast “draft” model propose several tokens, which the large model verifies in a single parallel forward pass, accepting the longest correct prefix. It directly exploits the spoke’s root fact — decode is memory-bandwidth-bound, not compute-bound — so verifying many tokens in one pass is nearly free. Source: Hugging Face.
Why it’s free correctness-wise
The big model still decides which tokens are accepted, so the output distribution is unchanged — it
validates proposals rather than generating each token serially, turning latency “from O(n) to
O(1)” in the ideal case. The peer-reviewed primary, speculative-decoding-paper (Leviathan et al.,
ICML 2023), makes this exact: its speculative sampling rule accepts each draft token with a
probability set by the ratio of the draft’s and target’s token probabilities, and resamples from an
adjusted distribution on rejection — provably reproducing the target model’s distribution. So the
“unchanged distribution” claim is a theorem, not an empirical observation, and it needs no retraining
or architecture change.
Numbers & requirements
HF (the assisted-generation framing): “Up to 3× with INT8, ~2× otherwise”; “up to 10×” with memory offloading. The primary, speculative-decoding-paper, reports 2×–3× wall-clock on T5-XXL vs the standard T5X implementation with identical outputs — the speedup scales with the draft model’s acceptance rate, so it is best when the draft is “at least an order of magnitude smaller” yet still a good approximation. Hard constraint: the assistant must share the exact same tokenizer. This is the sampling × serving interaction the open questions flagged — the draft model couples token selection (token-sampling) to the serving loop.
Related
speculative-decoding-paper · prefill-decode-kv-cache · token-sampling · llm-inference · kv-cache · vllm