Smaller, faster, safer: running Kimi and GLM at scale (Cloudflare, 2026-08-03)

Cloudflare’s engineering account of serving two open-weight frontier models — Moonshot’s Kimi K2.6 and Z.ai’s GLM 5.2 — on Workers AI, published 2026-08-03 with no individual byline. Three levers, each with a before/after table: quantize the kv-cache, quantize the weights, and then guard the cache that all those packed-together requests now share.

This is the first source in the spoke that separates the levers and measures each one on the same hardware. Every prior benchmark here either bundled everything (vllm) or measured one mechanism against no baseline.

The framing: the cache fills first, not the weights

“it is usually the KV cache, not the model’s weights, that fills up GPU memory first.”

Both models are long-context mixture-of-experts, which is what makes them expensive to serve rather than merely large. The second constraint named is bandwidth, not compute:

“generating each token means streaming the model’s weights out of GPU memory, so decode speed is limited by memory bandwidth.”

Those two sentences set up the two quantization moves: shrink the cache to fit more requests, shrink the weights to move fewer bytes per token.

Lever 1 — FP8 KV cache

The cache moves from BF16 to FP8 (e4m3), which halves it. For Kimi K2.6 that takes the budget from roughly 686,000 tokens to about 1.37 million on the same GPUs.

Decode throughput, tokens/sec:

The shape is the whole argument. FP8 is slower at every concurrency both configurations can run — about 9% down at a single request — and it wins anyway, because BF16 runs out of memory at 64 concurrent requests and FP8 does not. Peak-to-peak that is 2,192 against 1,558, which the post puts at 41% more throughput for roughly 30% less cost per token.

Accuracy across GSM8K, ARC-Easy, ARC-Challenge, MMLU and MMLU-Pro moves between −1.0 and +0.77 points; the post calls FP8 and BF16 caches indistinguishable.

Lever 2 — INT4 weights

GLM 5.2’s checkpoint drops from 705 GB in FP8 to 421 GB in INT4, about 40%, and per-GPU memory from roughly 88 GB to 52 GB across an 8-way tensor-parallel deployment.

Decode throughput, tokens/sec:

INT4 wins everywhere here, and by most at concurrency 1 — the memory-bandwidth-bound extreme, where fewer bytes per weight translates directly into tokens per second. Accuracy stays “within 0.8 points of the FP8 model across every benchmark” (GSM8K exact match 94.39% FP8 vs 93.56% INT4; MMLU 86.60% vs 86.54%).

Prefill goes the other way. Prefill is compute-bound, and INT4 weights have to be decompressed before they can be multiplied: about 10,160 tokens/sec of prefill in FP8 versus 8,660 in INT4. That inversion is what forces the third structural move.

Lever 3 — separate the pools

Because FP8-vs-BF16 cache and INT4-vs-FP8 weights each win in one phase and lose in the other, Cloudflare runs prefill and decode as separate pools and picks a different configuration for each: BF16 cache and FP8 weights for prefill, FP8 cache and INT4 weights for decode. Paged as prefill-decode-disaggregation.

The “safer” third: guarding a shared cache

Packing more requests onto one GPU means more of them share one physical cache, and the post is direct about what that costs:

“hundreds of requests are reading and writing pages of the same physical KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly.”

The mitigation tags every physical cache page with a value that changes whenever the page is reallocated, records which pages and tags each request expects, and validates the mapping before supported decode reads:

“If anything doesn’t match, the affected request is aborted rather than allowed to return data from the wrong page.”

They kept the check out of the attention kernel deliberately — fusing it “would have introduced a race between GPU thread groups” — and run it as a separate batch check instead. Measured cost at concurrency 1–8 with 8,192-token inputs and 1,000-token outputs: throughput −0.79% to −0.38%, p95 latency +0.42% to +0.80%. Paged as kv-cache-isolation.

Stack and what’s next

Serving runs on sglang, and Cloudflare says it works “closely with the SGLang team to upstream patches and new features to make our work available to the open-source community.” Hardware named is H200 for the disaggregated deployment. Next on their list: validating NVFP4 weights on Blackwell, and “working toward making integrity checks something we can leave on everywhere at negligible cost” — which implies the check is not on everywhere today.

With FP8 cache and INT4 weights together, GLM reaches “around 1.18 million tokens of KV cache on the same hardware.”

Tier and what to distrust

T2 — first-party engineering blog from the operator, with a product to sell (Workers AI). The numbers are self-reported, single-vendor, and no configuration details are given beyond the model, precision and concurrency, so nobody outside can reproduce them. What raises it above the usual vendor post is that each table has a baseline the vendor’s own choice loses to (FP8 cache is slower; INT4 prefill is slower), and the post keeps those losses in rather than reporting only the wins. Accuracy is measured on standard benchmarks rather than asserted.

The claim to hold loosely is the security one. “Even a one-in-a-billion mistake would show up regularly” is a statement about their volume, not evidence that such a mistake occurred, and no incident, bug or attack is described. The check is presented as defence in depth, and this wiki records it that way.

Related

kv-cache · quantization · prefill-decode-disaggregation · kv-cache-isolation · sglang · continuous-batching · paged-attention-paper · vllm · synthesis · cloudflare · glm-52 · z-ai