SGLang

An open-source inference serving framework, and the engine underneath the only production deployment this wiki has read a benchmark table from (cloudflare-kimi-glm-serving). It had been named twice in passing — local-llm-stack and llms-local-list list it beside Ollama, llama-cpp, vllm and MLX as a datacenter engine people also run locally — but nothing here described what it does until now.

What this wiki has evidence for

Cloudflare runs Workers AI inference on SGLang and describes working “closely with the SGLang team to upstream patches and new features to make our work available to the open-source community.” The features exercised in that post, and therefore the ones attested here:

• FP8 (e4m3) kv-cache quantization, as a per-deployment choice.
• INT4 weight quantization, likewise, and separable from the cache precision.
• Disaggregated prefill and decode pools, configured at different precisions.
• Paged cache management with the page-tag validation described in kv-cache-isolation — which is the kind of change the upstreaming sentence refers to.

That is the whole of the evidence. Nothing here covers SGLang’s own distinguishing design (RadixAttention, its structured-generation frontend), its licence, or how it compares to vllm on the same workload.

Where it sits

Same regime as vllm: the datacenter serving stack, where the constraint is keeping expensive GPUs busy across many tenants. The two are the field’s two main open serving engines, and this wiki now has a detailed benchmark run on one and a detailed feature inventory on the other, with no comparison between them — which is the obvious next source.

Related

vllm · kv-cache · prefill-decode-disaggregation · kv-cache-isolation · quantization · local-llm-stack · cloudflare-kimi-glm-serving · synthesis