vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (Kwon & Li et al., 2023)

The launch post for vllm, written by Woosuk Kwon and Zhuohan Li (equal contribution) with Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica — UC Berkeley Sky Computing Lab, 20 June 2023. It is the plain-language announcement that sits beside paged-attention-paper (the same authors' SOSP 2023 paper): the paper carries the peer-reviewed serving study, this post carries the mechanism explained for a general audience plus its own headline benchmark, run before the paper's numbers existed publicly.

The headline number, and what it's measured against

Up to 24x throughput over HuggingFace Transformers, up to 3.5x over HuggingFace Text Generation Inference (TGI) — no model changes required. Broken down by request shape:

One output per request: 14x-24x over Transformers, 2.2x-2.5x over TGI.
Three parallel outputs per request: 8.5x-15x over Transformers, 3.3x-3.5x over TGI.

Test setup: LLaMA-7B on an NVIDIA A10G and LLaMA-13B on an A100 (40GB), input/output lengths sampled from ShareGPT. No batch size, concurrency sweep, or hardware utilization figure is given beyond that — the same gap openmetal-tokens-per-second leaves in this spoke's other headline-number source.

This is a different comparison from paged-attention-paper. The paper reports 2-4x over FasterTransformer and Orca at equal latency; this post reports up to 24x over plain HuggingFace Transformers and up to 3.5x over TGI. Different baselines, different number — a reader citing "vLLM is 24x faster" and a reader citing "vLLM is 2-4x faster" are both accurate to a source, just not the same one. It is also easy to conflate with a third, close-but-distinct number already in this spoke: continuous-batching-anyscale (published two days after this post) measured vLLM's PagedAttention combined with continuous batching at up to 23x over naive static batching — a different baseline system and a different measuring team from this post's 24x-over-HuggingFace-Transformers, even though both numbers describe the same underlying engine. The two are independent measurements that happen to land close together, not one source citing the other — and this post itself never mentions continuous batching by name (see below).

The mechanism: PagedAttention

OS-style virtual memory and paging, applied to the kv-cache:

Each sequence's KV cache is split into fixed-size blocks, each holding a fixed number of tokens' keys and values.
Blocks do not need to sit in contiguous memory. A block table maps each sequence's logical blocks (contiguous, as the sequence sees them) to scattered physical blocks (where they actually live).
Physical blocks are allocated on demand, as tokens are generated, not reserved up front.
The post's own analogy: "blocks as pages, tokens as bytes, and sequences as processes."
Because only a sequence's last block can be partially empty, memory waste stays under 4% — against the 60-80% the post says existing systems lose to fragmentation and over-reservation. A single LLaMA-13B sequence's KV cache can reach 1.7GB.
Blocks can be shared across sequences (e.g. a common prompt prefix, or the parallel candidates in beam search / parallel sampling), tracked with reference counting and copy-on-write. The post credits this sharing with up to 55% memory reduction and up to 2.2x throughput on parallel-sampling and beam-search workloads specifically — narrower than the 24x headline, which is single-stream.

This is the same mechanism paged-attention-paper and vllm already describe; nothing here contradicts those pages, it is the more accessible restatement plus the specific numeric breakdown by request shape that neither the paper abstract nor the repo README states.

What this post does not say

The source material for this ingest described the post as covering continuous batching. It does not, by name. It says only that PagedAttention's memory efficiency "allows the system to batch more sequences together" — true, but it never uses the term "continuous batching" or describes iteration- level scheduling. continuous-batching's origin in this spoke stays continuous-batching-anyscale and, behind that, Orca (OSDI 2022); this post should not be cited for that mechanism.

Deployment numbers (Chatbot Arena / LMSYS, at time of writing)

Reported as vLLM's own production track record, not a controlled benchmark: two months live behind Chatbot Arena and the Vicuna demo, handling up to 5x more traffic than before; an LMSYS micro-benchmark showing up to 30x over LMSYS's initial HuggingFace backend; more than half of Arena requests (April-May 2023) served through vLLM; 50% fewer GPUs used for serving; average 30K requests/day, peak 60K. These are usage claims from the deploying team about their own system, not an independent measurement — recorded here as such.

Tier

T3. First-party announcement from the project's own authors; the mechanism description (blocks, block table, CoW sharing) is artifact fact and would stand as T1 on its own, but the page's load-bearing reason for inclusion is its self-run throughput benchmark (24x/3.5x), which per ../QUALITY.md's "maker's claims about how well it works" rule grades T3. The peer-reviewed serving study behind the same mechanism is paged-attention-paper (T1).

vllm · paged-attention-paper · kv-cache · continuous-batching · continuous-batching-anyscale · how-does-vllm-work · llm-inference