Synthesis — LLM Inference

The evolving thesis of this wiki. Sits above the schema.org pages. Records the current best understanding, open questions, and explicitly flagged contradictions.

Current understanding

llm-inference — running a trained model on a prompt — factors cleanly into three layers, and the founding three sources (all MachineLearningMastery code walk-throughs, all landing 2026-06-01) each take one layer:

1. Execution: two phases. Prefill processes the whole prompt in parallel; decode then emits tokens one at a time, autoregressively prefill-decode-kv-cache. This asymmetry is the root fact of inference economics: prefill is compute-bound and parallel; decode is sequential and latency-bound.
2. Token selection. At each decode step the logits become a distribution (softmax, scaled by temperature) and a token is chosen via token-sampling — top-k or top-p logits-softmax-sampling-walkthrough. This is the cheap part, layered on top of the heavy attention compute.
3. Serving at scale. continuous-batching keeps the GPU busy across many concurrent, variable-length requests by dynamically refilling batch slots and packing unpadded tokens with a block-diagonal mask continuous-batching-serving.

The unifying thread is the kv-cache. It is what makes decode cheap per step (O(n²)→O(n)) by storing past keys/values — and its memory footprint is precisely what makes serving hard, which is the problem continuous-batching exists to manage. So the three sources are not three topics but one pipeline seen at three altitudes: how a token is chosen → how the model runs to produce it → how that run is shared across users.

Open questions

• Quantified, neutral benchmarks. Most claims here are from intermediate tutorials with toy/demo setups (e.g. the 6.5× continuous-batching figure, the O(n) KV-cache claim). Partly closed (2026-06-15): flash-attention-paper (Dao et al., NeurIPS 2022) is the first peer-reviewed primary with hardware/model-specific numbers — 15% on BERT-large, 3× on GPT-2, 2.4× on long-range arena, plus a formal IO-complexity proof. It benchmarks training, though. Serving side now grounded (2026-06-16): paged-attention-paper (Kwon et al., SOSP 2023) is the peer-reviewed inference-serving study — 2–4× throughput over FasterTransformer/Orca at equal latency, attributed to near-zero KV fragmentation. What remains open is narrower: a which-lever- bought-what decomposition across vLLM’s full stack (paging vs batching vs quantization vs speculative decoding), not the existence of any controlled serving number. (A new T3 explainer, how-does-vllm-work (Amit Shekhar / Outcome School, 2026-06-17), restates the PagedAttention/continuous-batching mechanics cleanly but ships no numbers — a useful on-ramp to vllm, not movement on this gap.) Largely closed for the quantization levers (2026-08-03): cloudflare-kimi-glm-serving runs cache precision and weight precision as separate experiments on the same H200 hardware and publishes both tables — FP8-vs-BF16 kv-cache at five concurrency levels, INT4-vs-FP8 weights at five more, with accuracy on GSM8K/ARC/MMLU/MMLU-Pro beside each. That is the which-lever-bought-what decomposition this question has wanted since June, for two of the levers, from a first-party operator (T2) rather than a neutral study. Still open: the algorithmic levers (paging vs batching vs speculative decoding) remain bundled and unattributed, and nobody outside Cloudflare can reproduce these runs.
• Sampling × serving interaction. The sources treat sampling and batching independently. Does aggressive batching constrain per-request sampling (shared temperature, speculative decoding)? Unaddressed.
• Memory math. kv-cache growth (length × layers × heads × precision) and its cap on batch size / context length is asserted qualitatively but never quantified here. (The attention side of the memory story is now grounded: flash-attention-paper proves attention memory is linear, not quadratic, in sequence length — the KV-cache decode-step byte math remains unquantified here, but its serving cost is now grounded: paged-attention-paper shows the binding problem is fragmentation of the per-request cache, not just its raw size, and that OS-style paging recovers the wasted memory as batch capacity.) Now quantified (2026-08-03): cloudflare-kimi-glm-serving gives the byte math an actual budget — a BF16 cache for Kimi K2.6 holds ~686,000 tokens before the GPUs run out, FP8 holds ~1.37M, and with INT4 weights freeing further room GLM reaches ~1.18M. It also settles the ranking the earlier sources left vague: for a long-context model the cache, not the weights, is what fills memory first.
• Beyond the basics. PagedAttention/vLLM, speculative decoding, FlashAttention now added (2026-06-09): vllm (PagedAttention = OS-paging for the kv-cache, near-zero fragmentation + KV sharing), flash-attention (IO-aware exact attention — the prefill/decode compute core), and speculative-decoding (draft-and-verify; the concrete sampling × serving coupling). Still absent: quantization (int4/FP8). Now added (2026-06-10): quantization (the data-type lever — int8/int4/FP8, GPTQ/AWQ/bitsandbytes/GGUF; shrinks weights and the kv-cache, the fourth production lever after the three algorithmic ones) and llama-cpp (its flagship embodiment). The additions reframe the pipeline: the founding three explained what each layer does; the 06-09 trio explained how production engines make each layer cheap; quantization adds the orthogonal make-the-numbers-smaller axis that cuts across all layers.

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

1. Which lever bought what — the spoke’s oldest gap, now well-specified. vllm bundles paging, batching, chunked prefill, prefix caching, speculative decoding and quantization in one engine, so every throughput claim here is the stack’s, not a lever’s. — needs: a T1/T2 ablation on fixed hardware that turns levers off one at a time and reports each delta · hunted 2026-08-08 — nothing ≥ bar. Two searches, and the field’s output has the wrong shape twice over. The arXiv literature is method papers — ML-SpecQD, EAGLE, Ouroboros, SPEQ — each reporting its own technique against a baseline it configured (2.07×, 2.8×, 56.3% latency), which is a lever measured by its author, not levers measured against each other. The engine comparisons that do hold hardware fixed (Spheron, Yotta Labs, LeetLLM, markaicode on H100) are vendor and affiliate blog roundups, T3/T4, and they compare whole engines rather than levers, so they fail the bar and the question both. Not re-hunted before 2026-08-22.
2. vLLM against SGLang, by someone running neither. The spoke holds a detailed feature inventory of vllm and a detailed production benchmark on sglang, and no comparison between them. — needs: a T1/T2 head-to-head at stated concurrency and hardware.
3. The browser regime, benchmarked. litertjs‘s 3× and 5–60× figures are vendor claims and the regime has no independent measurement at all. — needs: a T2 WASM/WebGPU/WebNN comparison.
4. Quantization’s sign, per phase. cloudflare-kimi-glm-serving showed the lever helps decode and hurts prefill, which contradicts how the mechanism pages state it. One deployment reported it. — needs: a second independent measurement separating the phases.

Coverage edges (added 2026-08-08, at the curator’s request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

5. The attention variants that shrink the cache. kv-cache is the spoke’s central constraint and the architectural answers to it — multi-query and grouped-query attention, sliding-window attention, DeepSeek’s latent attention — have no page. — needs: the originating papers, T1.
6. Constrained decoding. Grammar- and schema-constrained generation (Outlines, XGrammar, llama.cpp’s GBNF) is a serve-time mechanism that changes the sampling step token-sampling describes, and it is absent. — needs: a paper or the implementation’s own design doc, plus any overhead figure.
7. Prefix caching. Reusing a shared prompt prefix across requests is the cheapest lever most servers ship, and it appears nowhere beside kv-cache-isolation. — needs: vLLM or SGLang design docs, plus a hit-rate or latency number.
8. Tokenization, the step before prefill. BPE and its variants set how many tokens a prompt costs, and the spoke starts one step later. — needs: the BPE/SentencePiece papers or a tokenizer’s docs.

Where inference runs, and what binds it there

Inference reads as distinct regimes, not one pipeline. The regimes sort by how the runtime reaches the hardware and what constraint binds there — and a fourth sorting axis, raw latency, turned out to reach past the serving layer into application design.

The ladder of access

Inference now reads as distinct regimes, not one. vllm + continuous-batching + flash-attention describe the datacenter regime, where the goal is maximizing GPU utilization across many concurrent users (the binding constraint is keeping an expensive GPU busy). llama-cpp

• quantization describe the on-device / single-user regime, where the binding constraint is fitting the model in limited memory at all — solved not by batching but by shrinking precision (GGUF 2–8-bit). Georgi Gerganov’s llama.cpp (the core of Ollama/LM Studio) is the canonical instance. So the same pipeline (prefill/decode, KV cache, sampling) runs at both ends of the hardware spectrum, with batching the datacenter lever and quantization the edge lever — and quantization is the one that appears in both (KV-cache + weight quantization help the GPU regime too).

A third regime — in-browser (added 2026-07-10). The edge splits again: beneath native on-device (llama-cpp/Ollama talking to CUDA/Metal) sits inference inside the web page, where the binding constraint is neither GPU utilization nor raw VRAM but the browser sandbox — no direct GPU access, so the runtime reaches the silicon through portable APIs (WASM/XNNPACK on CPU, WebGPU on GPU, emerging WebNN on NPU). litertjs (Google’s .tflite runtime for the web, ex- TensorFlow.js) is the founding instance, claiming ~3× over prior web runtimes and 5–60× GPU/NPU-over-CPU (vendor figures, unbenchmarked). Two things this regime teaches the spoke: (1) the pipeline is defined by how it reaches the hardware — datacenter GPU → native-edge → browser is a ladder of access, not just of size; and (2) its motivation is privacy + zero server cost + latency (nothing leaves the device), a different objective function from the datacenter’s utilization or the native edge’s fit-in-memory. Scope caveat: litertjs is general-ML (vision/audio/embeddings), not LLM-only — it enters the spoke as inference-execution mechanics on browser hardware (LLMs run on the same stack via the sibling MediaPipe), so the browser regime is broader than LLMs. This is the spoke’s first deliberate step past strictly-LLM serving into the general on-device inference runtime it shares a discipline with.

The edge regime’s tooling ecosystem, now mapped (2026-06-21). The on-device regime had only two engine pages (llama-cpp, vllm) and the claim that llama.cpp is “the core of Ollama/LM Studio.” A T4 curated awesome-list, llms-local-list, supplies the layer above: local-llm-stack separates platforms (turnkey apps — LM Studio, Jan, LocalAI) from engines (the runtimes — Ollama, llama.cpp, vLLM, SGLang, MLX). The useful structural fact is that most people “run a model locally” by picking a platform and never choosing an engine, while vllm/SGLang straddle both regimes (run on a single box as readily as a cluster). It’s a map, not a measurement — no benchmarks — so it broadens the spoke’s coverage of the edge regime without touching the standing which-lever-bought-what question.

vLLM as the datacenter regime’s convergence point (refreshed from the GitHub repo, 2026-06-11). The vllm page, previously grounded only in the docs, is now anchored to its primary source — and the repo makes plain that vLLM is not one lever but all of them in a single engine: PagedAttention + continuous-batching + chunked prefill + prefix caching + speculative-decoding (n-gram/suffix/ EAGLE/DFlash) + quantization (FP8/MXFP4/NVFP4/INT8/INT4/GPTQ/AWQ/GGUF) + flash-attention kernels

• tensor/pipeline/expert/context parallelism. So the wiki’s separate mechanism pages are not a list of alternatives — in production they stack inside the same system. vLLM also reaches well beyond NVIDIA (AMD, CPU, TPU, Gaudi) and 200+ model architectures incl. MoE & multimodal, so the datacenter regime is now a portable software stack, not a GPU-vendor story. This sharpens the standing benchmark question: with every lever bundled, an honest “which lever bought what” attribution needs a controlled, hardware- specified study — the README’s “state-of-the-art throughput” stays unquantified.

The axis the ladder does not capture — latency, and what it makes unnecessary

The ladder sorts by access and by binding constraint. One provider sorts by neither.

The regimes above are sorted by how the runtime reaches the hardware (datacenter GPU → native edge → browser) and what constraint binds (utilization → fit-in-memory → sandbox). cerebras-inference adds a data point that stresses a different axis: raw per-token latency. It is a hosted provider (not an installable engine like vllm/llama-cpp) running on a wafer-scale processor, and the vendor’s pitch is seconds-to-milliseconds. Same prefill/decode pipeline and kv-cache — a hardware substrate chosen to minimize latency rather than maximize multi-tenant throughput.

The genuinely new move is that the source, designing-for-cerebras, reasons from the serving layer up into the application: it argues that much of an LLM app’s architecture — job queues, always-on streaming, backgrounded agent loops, voice-filler UI — is latency compensation, built only to hide slow inference, and should be deleted once inference is fast. fast-inference-architecture captures that as a reusable, provider-agnostic idea with a one-line diagnostic (“was this built because LLM calls are slow?”). So the spoke now has a serving-side view (regimes/levers) and an application-side corollary: past a latency threshold, the simplest architecture becomes the fastest. Caveat: all speed figures here are vendor claims (T2) — this extends the standing which-lever-bought-what gap onto a hardware lever, still without an independent benchmark.

Put together, the ladder answers where can this model run and the latency axis answers what does the software around it stop needing. Both stay unresolved in the same way: every figure on either is a vendor claim, which is why the benchmark question below has outlived every source that touched it.

What the levers actually cost, once someone measures them

The mechanism pages describe levers in the abstract: batching, paging, quantization, speculative decoding. Two sources built or ran the whole stack instead of one piece of it, and both came back with a correction rather than a confirmation.

Built by hand — the bottleneck is not where the mechanism pages point

build-llm-runtime-from-scratch is the first source that isn’t about one mechanism but implements all of them at once: a hand-written decode-only engine for Qwen2.5-Coder-7B on an H100 (C++/CUDA PTX) that stacks INT4 quantization, a paged kv-cache, warp-specialized attention, fused GEMV and CUDA-graph capture into one runnable path. As a source it does two things for the spoke. It grounds the mechanism pages in concrete kernel-level reality — the paged KV cache isn’t a diagram, it’s 16 tokens per 4 KiB page with a real synchronization bug at the page boundary (a __syncthreads() in a warp-branch silently corrupting softmax) — and it puts a hard T2 number on the standing which-lever-bought-what gap that most pages fill with tutorial figures.

Its durable contribution is a fourth instance of the spoke’s recurring pattern: the bottleneck is per-step fixed overhead, not the modeled work. The single biggest win in the whole build was not a math kernel but wrapping decode in a CUDA graph — 280+ kernel launches per token collapsed into one driver submission, ~119 ms → ~17 ms, 7×. That rhymes exactly with the levers already here: continuous-batching amortizes per-request scheduling overhead, speculative-decoding amortizes the per-token forward pass, CUDA graphs amortize per-kernel launch cost — three altitudes of “stop paying a fixed cost once per small unit of work.” The honest coda is that the finished engine is slower than llama-cpp on the same task (60 vs 200 tok/s), and the author says so: the payoff claimed is ownership of the decode stack, not speed. So it enters as pedagogy and a legibility artifact, not a competitive runtime — and as one engineer’s single-model, single-GPU exercise with no independent check, the T2 numbers are illustrative, not a benchmark.

Run in production — a lever’s sign depends on the phase it lands in

The hand-built engine measures one model on one GPU for one user. The other correction comes from the opposite end: hundreds of tenants on a fleet, where the same levers behave differently enough to reverse.

cloudflare-kimi-glm-serving is the first source here written from inside a running multi-tenant deployment rather than from a paper, a tutorial, or a single-GPU exercise, and it changes two things in the picture above.

Precision becomes a property of a phase, not of a deployment. The founding asymmetry — prefill compute-bound, decode bandwidth-bound — had been treated as an explanation of why inference is expensive. Cloudflare treats it as a configuration boundary. FP8 kv-cache buys capacity that only decode needs, and costs about 9% per-token speed that prefill would rather keep; INT4 weights speed decode up by half at low concurrency and slow prefill down by 15%, because prefill has to decompress before it can multiply. Both trades point opposite ways in the two phases, so the phases run as separate pools at different precisions (prefill-decode-disaggregation). This is a level above continuous-batching and paging, which improve utilization within one engine doing both jobs.

It also corrects a simplification this wiki carried: quantization is not uniformly a speedup. Cache quantization is a capacity buy that costs per-token latency; the FP8 configuration loses to BF16 at every concurrency both can serve, and wins only because BF16 dies at 64 concurrent requests. The lever’s payoff is how many users fit, and reading it as “smaller means faster” gets the sign wrong half the time.

The optimizations created a correctness surface. Every mechanism in this spoke exists to pack more requests onto one GPU, and the endpoint of that is hundreds of requests reading and writing pages of one physical cache — where paging, batching and prefix reuse are all bookkeeping, and a mis-mapped page returns another request’s tokens silently rather than crashing. kv-cache-isolation is the spoke’s first mechanism page that is not about speed or memory at all: per-page generation tags, recorded expected mappings, validation before decode reads, and an aborted request when they disagree. It costs under 1% of throughput and tail latency, and Cloudflare says it is still working toward leaving it on everywhere — so even sub-1% is not yet free enough. The pattern worth carrying forward: multi-tenant efficiency and multi-tenant isolation are the same bookkeeping, seen from two directions.

Stack note: the deployment runs on sglang, which until now this wiki had only as a name in two lists. The spoke now holds a detailed feature inventory of vllm and a detailed benchmark run on SGLang, and no comparison between them.

The two sources agree on the uncomfortable part. In both, the win came from somewhere other than the modeled work — a driver submission in one, a phase boundary in the other — and in both, a lever this wiki had described as a speedup turned out to cost speed and buy something else (ownership in one, capacity in the other). The mechanism pages are not wrong; they are stated without the sign and without the phase, and those are exactly what a deployment decides.

Contradictions flagged

One, mild, and it is a sign error rather than a disagreement (2026-08-03). quantization, llama-cpp and build-llm-runtime-from-scratch all present lower precision as a memory-and-speed win, which holds for weights on a bandwidth-bound decode step. cloudflare-kimi-glm-serving measures the two other cases and both go the other way: FP8 KV cache is slower per token than BF16, and INT4 weights make prefill slower than FP8. Nobody is wrong — the earlier sources measured the single-user, weight-bound case and said so — but the generalization “quantization makes inference faster” does not survive, and the pages have been amended rather than one being overwritten.

Cross-spoke adjacency

• research-wiki holds the model-substrate thread (capability & cost of frontier models: claude-opus-4-8, Anthropic). This wiki is the mechanism layer beneath that business/capability story — how inference actually runs and what it costs in compute and memory. The hub router split these deliberately: research-wiki is tools-for-thought / agentic products; this spoke is inference internals. Watch for sources that bridge them (e.g. inference cost driving product economics).
• webperf-wiki shares a latency/efficiency sensibility (byte budgets there, GPU cycles here) but the domains don’t overlap in subject matter.

Index — LLM Inference Wiki

Catalog of all pages, grouped by @type. The spine: synthesis (thesis), log.md (history), this file (catalog).

DefinedTerm (concepts / mechanisms)

• llm-inference — umbrella: turning a prompt into tokens with a trained model; the three-layer pipeline. · domain
• token-sampling — logits → softmax → temperature → top-k / top-p; the per-step token choice. · mechanism
• kv-cache — key/value cache; the decode-phase optimization (O(n²)→O(n), at a memory cost). · mechanism
• continuous-batching — in-flight batching; serving-layer optimization for concurrent requests. · mechanism
• flash-attention — IO-aware exact attention; tiling/fusion avoids the N×N matrix in HBM · source · mechanism
• speculative-decoding — draft model proposes, big model verifies in parallel; same output distribution · source · mechanism
• quantization — lower-precision weights/activations/KV (int8/int4/FP8; GPTQ/AWQ/GGUF); the data-type lever, the fourth production optimization · source · mechanism
• local-llm-stack — the on-device execution stack: turnkey platforms (LM Studio/Jan/LocalAI) over engines (Ollama/llama.cpp/vLLM/SGLang/MLX) · landscape
• browser-inference — running inference client-side in the web page: WASM/WebGPU/WebNN as the browser’s hardware-access layer; the third regime (privacy/zero-server-cost), model-agnostic · regime
• prefill-decode-disaggregation — run prefill and decode as separate pools at different precisions, because FP8 cache and INT4 weights each win in one phase and lose in the other · source · mechanism
• kv-cache-isolation — per-page generation tags + validated mappings so one request can’t read another’s cached tokens; the spoke’s first correctness/safety mechanism, not a speed one · source · mechanism
• fast-inference-architecture — much of an LLM app is latency compensation (queues, always-stream, filler UI); fast inference makes deleting it the win — “was this built because LLM calls are slow?” · source · idea

SoftwareApplication (engines)

• vllm — high-throughput inference/serving engine; PagedAttention (OS-paging for the KV cache); bundles every datacenter lever in one stack (200+ models, multi-vendor HW); the datacenter regime · source
• llama-cpp — local/on-device C/C++ engine (Gerganov); GGUF 2–8-bit quantization; the edge regime (core of Ollama/LM Studio) · source
• litertjs — Google’s .tflite in-browser inference engine (WASM/XNNPACK + WebGPU/ML Drift + WebNN); succeeds TensorFlow.js (~3×); general-ML, the browser sub-regime of the edge · source · T2 · developers.googleblog.com
• sglang — open-source datacenter serving framework; the engine under Cloudflare’s Workers AI (FP8 cache, INT4 weights, disaggregated pools, page-tag validation upstreamed) · source
• cerebras-inference — hosted inference provider on a wafer-scale processor; the datacenter regime pushed to the latency extreme (seconds→ms, vendor claim) · provider

Organization

• cerebras-systems — wafer-scale AI-chip company behind cerebras-inference · org

ScholarlyArticle (source summaries, source: true)

• flash-attention-paper — Dao et al., FlashAttention (NeurIPS 2022); IO-aware exact attention, the primary paper + quantified speedups · source · T1 · arxiv.org
• flash-attention-2-paper — Dao, FlashAttention-2 (2023); the utilization sequel — 25–40%→50–73% of A100 peak FLOPs/s (~2× over FA-1) · source · T1 · arxiv.org
• paged-attention-paper — Kwon et al., PagedAttention/vLLM (SOSP 2023); OS-paging for the KV cache; 2–4× serving throughput over FasterTransformer/Orca — the inference-serving primary · source · T1 · arxiv.org
• speculative-decoding-paper — Leviathan et al., Fast Inference via Speculative Decoding (ICML 2023); speculative sampling preserves the exact distribution; 2–3× on T5-XXL · source · T1 · arxiv.org

Collection (source summaries, source: true)

• llms-local-list — 0xSojalSec’s LLMs-local awesome-list (via @DanKornas tweet); maps the on-device ecosystem (platforms/engines/UIs/tutorials) · source · T4 · github.com

TechArticle (source summaries, source: true)

• logits-softmax-sampling-walkthrough — MachineLearningMastery: logits, softmax & sampling. (url-only)
• prefill-decode-kv-cache — MachineLearningMastery: prefill, decode & the KV cache. (url-only)
• continuous-batching-serving — MachineLearningMastery: continuous batching for serving. (url-only)
• designing-for-cerebras — Cerebras docs: fast inference deletes latency-workaround architecture (queues, always-stream, filler UI); the “was this built because LLM calls are slow?” test · source · T2 · inference-docs.cerebras.ai
• build-llm-runtime-from-scratch — TDS (Banerjee): hand-written decode-only inference engine for Qwen2.5-Coder-7B on an H100 (C++/CUDA) — INT4 quant + paged KV cache + warp-specialized attention + fused GEMV + CUDA-graph decode (7×, ~119→17 ms/tok); every mechanism page built at once; slower than llama.cpp, sold as ownership not speed · source · T2 · towardsdatascience.com
• continuous-batching-anyscale — Anyscale: continuous batching benchmarked; ~4× / ~8× / ~23× decomposition, traces the technique to the Orca (OSDI 2022) primary · source · T2 · anyscale.com
• how-does-vllm-work — Amit Shekhar / Outcome School: from-scratch explainer of PagedAttention + continuous batching; accessible secondary to vllm, no numbers · source · T3 · outcomeschool.com
• cloudflare-kimi-glm-serving — Cloudflare (2026-08-03) serving Kimi K2.6 + GLM 5.2 on Workers AI: FP8 KV cache (686k→1.37M tokens, −9% per-token, +41% peak), INT4 weights (705→421 GB, 60→92 tok/s), separate prefill/decode pools, and page-tag cache-integrity validation at <1% cost. The spoke’s first lever-by-lever benchmark · source · T2 · blog.cloudflare.com

Notes

• Entity reuse (cross-wiki, not duplicated): cloudflare lives in cloud-wiki; glm-52 and z-ai live in llm-providers-wiki. Linked, never re-paged here — this spoke owns the mechanics, those spokes own the vendor and model nodes. Moonshot has no entity node in the hub yet; Kimi K2.6 is named in prose rather than paged, since this source says nothing about the model beyond its shape.