From Prompt to Prediction: Prefill, Decode, and the KV Cache

A code-first walk-through (MachineLearningMastery, intermediate; assumes familiarity with attention) of the two phases of transformer llm-inference and the cache that makes the second phase cheap.

What it covers

• Prefill phase. All prompt tokens are processed in parallel — each attends to itself and prior tokens via scaled dot-product attention, building contextual representations in one pass.
• Decode phase. Output tokens are generated one at a time, autoregressively. Each new token attends to the already-computed context; the model avoids recomputing it.
• kv-cache. Stores the keys and values computed during prefill; during decode the system appends only the new token's K/V instead of recomputing K/V for all prior tokens, cutting per-step attention cost from O(n²) toward O(n).

Concrete example

PyTorch code demonstrating causal masking, simplified attention heads with explicit selection rules, context-vector computation during prefill, and KV-cache appending during decode.

Takeaway

"Prefill warms up the KV cache and decode updates it." This phase split explains why LLMs ingest long prompts efficiently (parallel prefill) yet emit output token-by-token (sequential decode), and why caching is essential to practical serving — the foundation continuous-batching builds on.

The measurement consequence (added 2026-09-10)

The same asymmetry is why a single "tokens per second" figure is untrustworthy: prefill and decode run at different speeds, by as much as 10× in one vendor's measurements, so a combined number mostly reports the prompt-to-output ratio the benchmarker chose. See tokens-per-second and openmetal-tokens-per-second.

Tier: T4 → T2 (2026-08-25)

Same reading as the two sibling MLM pages. Prefill/decode and the KV cache are public mechanism, the walk-through implements them in PyTorch, and no vendor's claim is being relayed. T2 on the established-concept branch. Flagged in prose, as before: the attention heads are simplified for teaching, so the code is an illustration of the mechanism and not a reference implementation.