BatchLLM (Microsoft, MLSys Industry Track)
Zheng, Ji, Fang, Zhou, Liu and Peng (Microsoft; one author interning from the Chinese Information
Processing Laboratory), published in the Industry Track of the 9th MLSys conference; code at
github.com/microsoft/MixLLM. Found by the 2026-09-23 research pass for growth edge 1, which lever
bought what, and read from the PDF.
The paper proposes its own system for large offline batches that share prompt prefixes: find shared prefixes across the whole batch up front, schedule requests that share one together, and mix decode tokens into prefill chunks. Its own speed-ups are the authors' claims about their system, and this page does not lean on them. The reason it is here is one small table that measures two of vllm's features, which the authors did not build, one at a time on fixed hardware.
T1 on that table: a peer-reviewed venue, stated hardware and software, and levers the authors have no stake in. Their own system's numbers are flagged as theirs.
The table that matters here
vLLM v0.6.4, one industry workload, Qwen-2.5-7B, throughput as reported (the paper does not name the unit in this table):
| Setting | A100 | MI200 |
|---|---|---|
| vLLM | 5.45 | 2.75 |
| + chunked prefill | 5.48 (+0.6%) | 2.84 (+3.3%) |
| + prefix caching | 6.20 (+14%) | 3.27 (+19%) |
| + both | 6.71 (+23%) | 3.36 (+22%) |
On this workload chunked prefill is worth almost nothing on its own, prefix caching is worth 14–19%, and together they add slightly more than the sum of their parts on A100. The gain from the two features differs between the NVIDIA and AMD parts. So a lever's value depends on both the hardware and the workload.
A lever outside the engine
The authors then sorted the requests with Python's sorted() so that requests sharing a prefix sat
next to each other, and changed nothing else. vLLM's throughput went from 6 to 13.02, more than
double, a larger gain than both engine features combined. BatchLLM still reached 18. This is the
spoke's second lever that works before the engine runs (after model-routing), and it is also the
cheapest.
Limits
- One workload, two levers. Paging, continuous-batching, speculative-decoding and quantization are not switched off anywhere, so the spoke's question stays open for them.
- Total throughput only. The table does not split prefill and decode, which openmetal-tokens-per-second showed can differ tenfold.
- The workload is one the authors chose because prefix sharing is heavy in it, which favours prefix caching.
- vLLM v0.6.4 is an old version by late 2026.
Related
vllm · sglang · kv-cache · continuous-batching · glm5-serving-parameter-tuning · model-routing · llm-inference · synthesis