How Continuous Batching Enables 23x Throughput (Anyscale, 2023)
Anyscale engineering post — Cade Daniel, Chen Shen, Eric Liang, Richard Liaw, June 2023. The continuous-batching page was grounded in a single tutorial demo (the ≈6.5× figure); this is the benchmarked secondary that ties the technique to its primary source and gives a stack-decomposed set of numbers.
The primary it points to: Orca (OSDI 2022)
The post credits Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022) as “the first to our knowledge to tackle this problem” — i.e. the origin of iteration-level scheduling, the academic name for continuous / in-flight batching. (Orca is USENIX-only and was not fetchable here directly; this T2 secondary is the bridge to it.)
The core framing
“LLM inference is memory-IO bound, not compute bound” — so the dominant performance lever is keeping the batch full, not adding raw FLOPs. In static batching the batch holds its size until every sequence finishes, idling the GPU as shorter requests complete; continuous batching admits a new request the moment a slot frees, at iteration granularity.
The decomposed numbers
Measured against a naive static-batching baseline:
- ~4× — optimized static batching (NVIDIA FasterTransformer).
- ~8× — continuous batching alone (comparable across Ray Serve and HF text-generation-inference).
- ~23× — continuous batching combined with memory optimization (the vLLM PagedAttention implementation).
That last jump from 8× to 23× is the empirical case that batching and KV-memory management (PagedAttention) compound — consistent with the synthesis’s “levers stack inside the same engine” reading of vllm.
Tier
T2 — vendor engineering blog, named authors, controlled A/B benchmarks across multiple frameworks, explicitly attributed to the Orca primary. Not peer-reviewed; numbers are Anyscale’s own runs.
Related
continuous-batching · kv-cache · paged-attention-paper · vllm · llm-inference