KV Cache

The KV cache (key–value cache) is the core optimization of the decode phase of llm-inference prefill-decode-kv-cache. During attention, each token produces a key (K) and value (V); without caching, generating token n would recompute K and V for all n prior tokens every step.

How it works

• Prefill computes K/V for all prompt tokens in parallel and stores them — “prefill warms up the KV cache.”
• Decode computes K/V for only the new token and appends it to the cache — “decode updates it.” Each step then attends over the cached K/V plus the new entry.

This drops the per-step attention cost from roughly O(n²) to O(n), which is what makes autoregressive generation practical.

Cost it shifts, not removes

The cache trades compute for memory: it grows linearly with sequence length × layers × heads and must be held in GPU memory for the whole generation. That memory pressure is a key reason serving needs smart scheduling — see continuous-batching, which manages many such per-request caches at once.

The pressure isn’t only the cache’s size but the fragmentation around it: when each request gets one contiguous allocation, dynamic growth/shrink wastes memory and caps the batch size. The PagedAttention paper (Kwon et al., SOSP 2023) is the primary source on this — it borrows OS paging to store the cache in non-contiguous blocks, cutting waste to near zero and letting requests share identical KV (a shared prefix, or parallel-sampling candidates). vllm is its production embodiment.

How big it actually gets (2026-08-03)

The wiki asserted this memory pressure for two months without a number attached. cloudflare-kimi-glm-serving supplies one: serving Moonshot’s Kimi K2.6, a BF16 cache holds roughly 686,000 tokens before GPU memory runs out; halving the precision to FP8 (e4m3) takes that to about 1.37 million. The post states the ranking directly — for a long-context model “it is usually the KV cache, not the model’s weights, that fills up GPU memory first.”

Quantizing the cache is therefore not a marginal saving but the thing that decides how many concurrent requests a GPU can hold at all, and it buys that capacity at a small per-token speed cost (see quantization for the trade). Because the win is capacity rather than speed, it lands on decode and not on prefill, which is one of the two reasons Cloudflare splits the phases into separate pools (prefill-decode-disaggregation).

The cost of sharing it

Once many requests occupy one physical cache, the paging and reuse that make it efficient become a correctness surface: a mis-mapped page returns another request’s tokens, silently. See kv-cache-isolation.

Related

quantization · continuous-batching · paged-attention-paper · vllm · prefill-decode-disaggregation · kv-cache-isolation · cloudflare-kimi-glm-serving · llm-inference