KV cache isolation

Keeping one request’s tokens out of another request’s attention when hundreds of requests share the same physical kv-cache. The first correctness-and-safety mechanism in this spoke; every other page here is about speed or memory.

Why the problem exists at all

It is created by the optimizations this wiki has been cataloguing. Paged attention stores each request’s cache in non-contiguous blocks and lets requests share identical pages; continuous-batching keeps dozens of requests resident at once; cache reuse across a shared prefix is a deliberate feature. All three are bookkeeping over a pool of physical pages that no longer belongs to any one request, and cloudflare-kimi-glm-serving states the consequence plainly:

“The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly.”

A mis-mapped page is not a crash. It is a request reading tokens it never sent — stale content from a finished request, or another tenant’s prompt. The failure mode is silent and looks like a correct answer.

The mechanism as described

Three parts, from cloudflare-kimi-glm-serving:

1. Generation tags. Every physical cache page carries a tag that changes whenever the page is reallocated, so a stale reference to a recycled page no longer matches.
2. Expected-mapping records. The server records which pages and which tags each request should be reading.
3. Validation before the read. Supported decode operations check the mapping first. “If anything doesn’t match, the affected request is aborted rather than allowed to return data from the wrong page.” Failing the request is the designed outcome; returning wrong data is not.

The check runs as a separate batch check, not fused into the attention kernel, because fusing it “would have introduced a race between GPU thread groups.” Measured overhead at concurrency 1–8 with 8,192-token inputs and 1,000-token outputs: throughput down 0.38–0.79%, p95 latency up 0.42–0.80%.

What this wiki does and doesn’t claim

The source describes no incident, no bug found, and no attack. Nothing here shows that cross-request leakage has occurred in any engine, and the tag scheme is presented as defence in depth against the operator’s own bookkeeping. The post’s stated next step — “working toward making integrity checks something we can leave on everywhere at negligible cost” — implies the validation is not on for all operations today, even at under 1% cost.

Untouched by any source here: whether validation catches a malicious attempt to reference another request’s pages, or only accidental mismapping. The mechanism as described is a consistency check against expected state, so it would catch either — but the source argues from error rates, not from a threat model, and this page does not extend it into one.

Related

kv-cache · paged-attention-paper · continuous-batching · cloudflare-kimi-glm-serving · sglang · synthesis