SGLang

An open-source inference serving framework, and the engine underneath the only production deployment this wiki has read a benchmark table from (cloudflare-kimi-glm-serving). It had been named twice in passing — local-llm-stack and llms-local-list list it beside Ollama, llama-cpp, vllm and MLX as a datacenter engine people also run locally — but nothing here described what it does until now.

What this wiki has evidence for

Cloudflare runs Workers AI inference on SGLang and describes working "closely with the SGLang team to upstream patches and new features to make our work available to the open-source community." The features exercised in that post, and therefore the ones attested here:

• FP8 (e4m3) kv-cache quantization, as a per-deployment choice.
• INT4 weight quantization, likewise, and separable from the cache precision.
• Disaggregated prefill and decode pools, configured at different precisions.
• Paged cache management with the page-tag validation described in kv-cache-isolation — which is the kind of change the upstreaming sentence refers to.

A second deployment, and the first parameter surface (2026-08-22). glm5-serving-parameter-tuning runs SGLang v0.5.9 in two-node distributed mode with the flashinfer attention backend, serving GLM-5 FP8 across sixteen H100s, and sweeps three of its flags with everything else pinned:

• --chunked-prefill-size — best at 3072, with 4096 the worst tested value on every axis.
• the tp/pp split — 4/4 at 9,993 tok/s against 8/2 at 6,906, a 45% swing on the same GPUs.
• --max-running-requests — best at 24, above and below which throughput and latency both worsen.

Fixed flags named in that deployment, which is the closest thing this wiki has to a picture of SGLang's operational surface: --context-length 202752, --mem-fraction-static 0.85, --tool-call-parser, --reasoning-parser, --enable-reasoning-tokens, --enable-cache-report.

That is the whole of the evidence. Nothing here covers SGLang's own distinguishing design (RadixAttention, its structured-generation frontend), its licence, or how it compares to vllm on the same workload.

Where it sits

Same regime as vllm: the datacenter serving stack, where the constraint is keeping expensive GPUs busy across many tenants. The two are the field's two main open serving engines, and this wiki now has a detailed benchmark run on one and a detailed feature inventory on the other, with no comparison between them — which is the obvious next source.

Related

vllm · kv-cache · prefill-decode-disaggregation · kv-cache-isolation · quantization · local-llm-stack · cloudflare-kimi-glm-serving · glm5-serving-parameter-tuning · synthesis