Prefill/decode disaggregation

Running the two phases of llm-inference as separate server pools rather than as two stages inside one process, so each can be configured for the constraint that actually binds it. Sourced here from cloudflare-kimi-glm-serving, which uses it in production on H200s.

Why the phases want different machines

The founding asymmetry in this wiki’s synthesis — prefill is parallel and compute-bound, decode is sequential and bandwidth-bound — has a consequence the early sources never drew out. An optimization that helps one phase can hurt the other, and if both phases share a process you have to pick one number to lose.

cloudflare-kimi-glm-serving shows both inversions on the same hardware:

• FP8 kv-cache doubles cache capacity, which decode needs to reach high concurrency, but it is a few percent slower per token at low concurrency. Prefill doesn’t benefit from the bigger budget, so prefill keeps BF16 and its slightly higher throughput.
• INT4 weights speed decode up substantially (60 → 92 tok/s at concurrency 1) because decode is limited by streaming weights out of memory. Prefill is compute-bound and must decompress those weights before multiplying, so INT4 costs it about 15% (10,160 → 8,660 tokens/sec).

Split the pools and both trades resolve the right way: “we run INT4 for decode, where it wins, and FP8 for prefill, where it wins.”

What it changes about the serving picture

continuous-batching and paged attention both improve utilization within one engine handling both phases. Disaggregation is a level above: the phases stop competing for the same memory and the same precision choice, and the KV cache produced by prefill has to travel to the decode pool that will consume it. The transfer cost is real and cloudflare-kimi-glm-serving does not quantify it, which is the gap on this page.

It also reframes what “quantization” means operationally. Precision stops being a property of a deployment and becomes a property of a phase — the same model can be resident twice, at two precisions, in the same cluster.

Open on this page

• What the KV handoff costs. No source here measures the prefill→decode cache transfer, the network it crosses, or how pool sizing is chosen between the two.
• When it isn’t worth it. Disaggregation presumes enough traffic to keep two pools busy. No source here states the volume below which one mixed pool wins.

Related

kv-cache · quantization · continuous-batching · cloudflare-kimi-glm-serving · sglang · llm-inference · synthesis