GLM-5 serving parameter tuning (China Unicom)

Hua, Wang, Yang, Wang & Lian, China Unicom — GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads, arXiv:2607.02518, 27 April 2026. Ten configurations, one cluster, one workload, three knobs. Found by the research pass hunting growth edge 1.

T2. China Unicom did not build sglang and did not build GLM-5; they measured somebody else's engine running somebody else's model on hardware they operate, and published the sweep. That is a third party measuring a product it does not sell, which is what the T2 rung describes — the same grade this wiki gives cloudflare-kimi-glm-serving for the same reason.

The setup, which is the reason this counts

Two nodes, sixteen H100s, eight per node, 2× Xeon Platinum 8468 and 2 TiB of RAM per node, ConnectX-7 InfiniBand. SGLang v0.5.9 in two-node distributed mode, flashinfer attention backend. GLM-5, FP8 checkpoint. Workload: OpenClaw agent traffic, roughly 28,000–30,000 input tokens and ~500 output tokens per request — long tool-augmented prefixes, which is prefill-heavy by construction.

Everything else is nailed down. The paper lists the fixed server flags (--context-length 202752, --mem-fraction-static 0.85, the parsers) and states that exactly three knobs vary: --chunked-prefill-size, the tp/pp split, and --max-running-requests. Same model artifact, same topology, same framework across all ten rows.

The sweep

Best against baseline: +10.7% total tokens/s, +11.6% requests/s, −25.5% TTFT, −18.9% P90, which the authors convert to 10.4% lower cost per request and 9.6% per token under a fixed footprint.

What is worth taking from it

The parallelism split is the biggest lever by a distance, and it is free. Holding chunk and concurrency fixed, tp4/pp4 gives 9,993 tok/s and tp8/pp2 gives 6,906 — a 45% throughput difference on the same sixteen GPUs, with tp2/pp8 in between at 8,705. Nothing was bought, nothing was quantized, no algorithm changed. This wiki's mechanism pages describe paging, batching and quantization as the levers; here the largest single delta comes from how the model is cut across devices, which no page here covers. TTFT moves with it too — 6.69 s against 9.48 s.

All three knobs are non-monotonic, and two of them peak in the interior. Chunk size: 2048 → 3072 helps, 4096 is the worst tested value on every axis (below even the 2048 baseline), 6144 partly recovers. Concurrency: 24 beats 16, then 32 and 48 both fall back. The authors state it directly — larger chunks and deeper queueing "do not further improve this workload." The default tuning instinct, raise the batch and raise the chunk, is wrong here in both directions.

Throughput and latency moved together, not against each other. The usual serving trade is throughput against tail latency. Going from max-run 16 to 24 improved both: 9,270 → 9,993 tok/s and TTFT 8.04 → 6.69 s and P90 40.07 → 32.64 s. That is a configuration that was simply below its knee, not a trade being made. It is worth naming because the shape of the usual argument makes this case easy to miss.

The knee is workload-specific and the paper says so. ~28–30k input tokens is prefill-dominated traffic; the reason 24 concurrent requests is a knee at all is that each one carries a very large kv-cache and a long prefill. No number here transfers to short-prompt chat serving, and the authors decline to generalize the result even to another precision of the same model.

What it does not do

It is not the ablation growth edge 1 asks for. The edge wants the feature levers separated — paging against batching against quantization against speculative-decoding — each turned off in turn with its own delta. This paper varies scheduling and partitioning parameters with every feature left on throughout. Chunked prefill is present in all ten rows; only its chunk size changes.

So it half-closes the edge along an axis the edge had not thought to name. The corpus has wanted a which-lever-bought-what decomposition since June and has now been given, twice, something adjacent: cloudflare-kimi-glm-serving separated the two precision levers, and this separates three configuration levers. The algorithmic levers are still bundled, and both sources that did isolate anything did it on levers the edge was not asking about.

Three further limits the authors state. The FP8 result must not be read across to BF16, INT4 or W4A8 without re-measuring, because weight footprint changes what is left for the cache. There is no resource telemetry — no GPU utilization, no KV-cache occupancy — so the interpretations (pipeline bubbles at 2/8, collective overhead at 8/2) are inference from throughput, not measurement. And the exact benchmark command lines were not archived, which they list as further work; nobody outside China Unicom can reproduce these runs, the same defect cloudflare-kimi-glm-serving has.

One thing it does supply for free: this is a second production deployment on sglang, and the first to publish a parameter surface for it rather than a feature list.

Related

sglang · cloudflare-kimi-glm-serving · continuous-batching · kv-cache · prefill-decode-kv-cache · vllm · llm-inference · synthesis