KV Cache

The KV cache (key–value cache) is the core optimization of the decode phase of llm-inference prefill-decode-kv-cache. During attention, each token produces a key (K) and value (V); without caching, generating token n would recompute K and V for all n prior tokens every step.

How it works

• Prefill computes K/V for all prompt tokens in parallel and stores them — "prefill warms up the KV cache."
• Decode computes K/V for only the new token and appends it to the cache — "decode updates it." Each step then attends over the cached K/V plus the new entry.

This drops the per-step attention cost from roughly O(n²) to O(n), which is what makes autoregressive generation practical.

Cost it shifts, not removes

The cache trades compute for memory: it grows linearly with sequence length × layers × heads and must be held in GPU memory for the whole generation. That memory pressure is a key reason serving needs smart scheduling — see continuous-batching, which manages many such per-request caches at once.

The pressure isn't only the cache's size but the fragmentation around it: when each request gets one contiguous allocation, dynamic growth/shrink wastes memory and caps the batch size. The PagedAttention paper (Kwon et al., SOSP 2023) is the primary source on this — it borrows OS paging to store the cache in non-contiguous blocks, cutting waste to near zero and letting requests share identical KV (a shared prefix, or parallel-sampling candidates). vllm is its production embodiment.

How big it actually gets (2026-08-03)

The wiki asserted this memory pressure for two months without a number attached. cloudflare-kimi-glm-serving supplies one: serving Moonshot's Kimi K2.6, a BF16 cache holds roughly 686,000 tokens before GPU memory runs out; halving the precision to FP8 (e4m3) takes that to about 1.37 million. The post states the ranking directly — for a long-context model "it is usually the KV cache, not the model's weights, that fills up GPU memory first."

Quantizing the cache is therefore not a marginal saving but the thing that decides how many concurrent requests a GPU can hold at all, and it buys that capacity at a small per-token speed cost (see quantization for the trade). Because the win is capacity rather than speed, it lands on decode and not on prefill, which is one of the two reasons Cloudflare splits the phases into separate pools (prefill-decode-disaggregation).

The formula, and the architectures that attack it (2026-08-17)

serret-attention-primer writes the arithmetic this page had been describing in words. Cached floats:

2 · N_layers · N_heads · N_tokens · d

times the bytes per float — the 2 for K and V — and the compute side is the complement: recomputing keys and values every step is O(N³·d) over a conversation, caching them makes it O(N²·d). It also names the regime: streaming attention, where the query tokens are a subset of the key/value tokens, as against full attention. Streaming is only possible because the cache exists.

Written that way, the formula names its own attack surface — every factor is something an architecture has tried to shrink, and the spoke now holds three of the four:

• N_heads → grouped-query-attention shares one KV head across several query heads, so the head count in the K and V terms drops (Llama 3 70B: 64 query heads, 8 KV heads).
• the whole per-head K and V → latent-attention replaces them with a single shared low-rank latent per token, which is why DeepSeek V2 has no KV-head count to state.
• bytes per float → quantization, measured at FP8 by cloudflare-kimi-glm-serving above.
• N_tokens → sliding-window attention, which this corpus still has no source on.

The reason keys and values are kept while queries are discarded also falls out of the algebra rather than being a convention: the last token's output depends on the K and V of every prior token, and prior queries affect nothing downstream. See attention-mechanism.

The cost of sharing it

Once many requests occupy one physical cache, the paging and reuse that make it efficient become a correctness surface: a mis-mapped page returns another request's tokens, silently. See kv-cache-isolation.

It does not have to die with the process (2026-09-04)

Everything above treats the cache as memory: something built during prefill, reused during decode, evicted under pressure, and gone when the server stops. omlx adds a rung below eviction. Its paged cache has two tiers — a hot tier in RAM and a cold tier on SSD, where evicted blocks are written out as safetensors. A later request whose prefix matches restores those blocks from disk instead of recomputing them, including after the server has been restarted.

The consequence is about when prefill is paid, not how fast it runs. Every other lever in this spoke makes prefill cheaper per occurrence; persistence changes the number of occurrences, and for a workload that re-sends a large stable prefix — a coding agent with a repository in context, which is the case oMLX names — the cost can approach once ever rather than once per boot. It is also the first appearance here of the disk as a rung on the memory ladder, one step past the RAM-vs-HBM offloading the datacenter sources describe.

Two caveats travel with it. A cache on disk is a cache that outlives the process boundary that used to bound the sharing problem — the isolation argument above was written about pages inside one running server, and safetensors files under ~/.omlx/cache are a longer-lived surface than that. And the benefit is entirely unmeasured: oMLX publishes no hit rate, no restore latency and no end-to-end saving for the cold tier, so what the corpus holds is a mechanism with a plausible story and no number.

Related

attention-mechanism · grouped-query-attention · latent-attention · serret-attention-primer · quantization · continuous-batching · paged-attention-paper · vllm · prefill-decode-disaggregation · kv-cache-isolation · omlx · cloudflare-kimi-glm-serving · llm-inference