Continuous Batching
Continuous batching (a.k.a. in-flight batching) is the serving-layer optimization of llm-inference that keeps a GPU busy when many users issue requests of differing lengths continuous-batching-serving.
The problem it fixes
Static batching groups requests into a fixed batch padded to the longest sequence; short requests idle until the longest in the batch finishes, wasting GPU cycles on padding tokens.
The mechanism
Two parts working together:
- Dynamic scheduling — the instant a sequence finishes it frees its slot and the next queued prompt is admitted on the same step, so the batch is continuously refilled.
- Ragged batching — all in-flight tokens are concatenated into a single unpadded row, with a block-diagonal attention mask ensuring sequences don’t attend across each other.
Result
Eliminating padding and idle slots gave ≈6.5× throughput in the source’s demo (9.54 s vs. 61.80 s, same hardware). It is the multi-request counterpart to the single-stream kv-cache optimization, and it manages one KV cache per in-flight sequence. (Caveat: single tutorial benchmark, not a rigorous evaluation.)
Provenance and the harder numbers
The technique’s academic origin is iteration-level scheduling, introduced in Orca (Yu et al., OSDI 2022) — “the first to our knowledge to tackle this problem” per continuous-batching-anyscale, the secondary that bridges to the USENIX-only paper. Anyscale’s controlled A/B run decomposes the win against a naive static-batching baseline: ~4× for optimized static batching (FasterTransformer), ~8× for continuous batching alone, and ~23× for continuous batching combined with PagedAttention KV-memory management (vllm). The jump from 8× to 23× is the empirical case that batching and KV paging compound rather than overlap — the same “levers stack in one engine” pattern the synthesis reads into vllm.
Related
continuous-batching-anyscale · kv-cache · paged-attention-paper · vllm · llm-inference · continuous-batching-serving