Spokes.wiki Search About

Synthesis — LLM Inference

The evolving thesis of this wiki. Sits above the schema.org pages. Records the current best understanding, open questions, and explicitly flagged contradictions.

Current understanding

llm-inference — running a trained model on a prompt — factors cleanly into three layers, and the founding three sources (all MachineLearningMastery code walk-throughs, all landing 2026-06-01) each take one layer:

  1. Execution: two phases. Prefill processes the whole prompt in parallel; decode then emits tokens one at a time, autoregressively prefill-decode-kv-cache. This asymmetry is the root fact of inference economics: prefill is compute-bound and parallel; decode is sequential and latency-bound.
  2. Token selection. At each decode step the logits become a distribution (softmax, scaled by temperature) and a token is chosen via token-sampling — top-k or top-p logits-softmax-sampling-walkthrough. This is the cheap part, layered on top of the heavy attention compute.
  3. Serving at scale. continuous-batching keeps the GPU busy across many concurrent, variable-length requests by dynamically refilling batch slots and packing unpadded tokens with a block-diagonal mask continuous-batching-serving.

The unifying thread is the kv-cache. It is what makes decode cheap per step (O(n²)→O(n)) by storing past keys/values — and its memory footprint is precisely what makes serving hard, which is the problem continuous-batching exists to manage. So the three sources are not three topics but one pipeline seen at three altitudes: how a token is chosen → how the model runs to produce it → how that run is shared across users.

Open questions

The biggest lever measured here is one the spoke never listed (added 2026-08-22)

Growth edge 1 has asked since June for a decomposition: which of paging, batching, chunked prefill, prefix caching, speculative decoding and quantization bought the throughput. Two sources have now answered adjacent questions instead, and the second one moves the frame.

glm5-serving-parameter-tuning holds sixteen H100s, sglang v0.5.9 and a GLM-5 FP8 checkpoint fixed, runs ~28–30k-token agent requests, and varies three knobs one at a time. The result the spine should keep: tp4/pp4 reaches 9,993 tok/s and tp8/pp2 reaches 6,906 — a 45% throughput difference on identical hardware, with identical software, from nothing but how the model is cut across devices. TTFT moves with it, 6.69 s against 9.48 s. Nothing was quantized, no algorithm changed, no feature was switched on.

Every lever this wiki argues about is smaller than that in the same paper. Chunked-prefill size across its whole tested range moves throughput by about 15% (7,886 at 4096 to 9,270 at 3072); concurrency from 16 to 24 moves it 8%. The parallelism decomposition — a subject with no page here at all — moves it 45%.

Two corollaries, both of which cut against the way serving advice is usually written:

The knobs are non-monotonic and peak in the interior. 3072 beats both 2048 and 4096, and 4096 is the worst tested value on every axis. Concurrency 24 beats 16, 32 and 48. "Raise the batch size, raise the chunk" is wrong in both directions on this workload, and the authors say so plainly.

Throughput and tail latency moved together. Going from 24 concurrent requests down to 16 lost throughput and raised TTFT and raised P90. That is not the trade the serving literature describes; it is a configuration sitting below its knee, where both curves are still improving. The familiar trade only starts past the knee, which here is around 24 requests — and where the knee sits is a property of a workload carrying 28–30k input tokens per request, not a constant.

What this does to edge 1. The edge is not closed and its wording is not wrong. But the corpus has now been handed lever separations twice — cloudflare-kimi-glm-serving separated cache precision from weight precision, and this separates chunk, split and queue depth — and neither touched the algorithmic levers the edge names. Two independent teams with the hardware to run an ablation both chose to ablate the things an operator can change on a running deployment. That may be the answer: the feature levers are not separable in practice because nobody turns paging off, while the configuration levers are the ones with a knob.

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

  1. Which lever bought what — the spoke's oldest gap, now well-specified. vllm bundles paging, batching, chunked prefill, prefix caching, speculative decoding and quantization in one engine, so every throughput claim here is the stack's, not a lever's. — needs: a T1/T2 ablation on fixed hardware that turns levers off one at a time and reports each delta. HALF-CLOSED 2026-08-22 (research pass) by glm5-serving-parameter-tuning (China Unicom, arXiv:2607.02518, T2): ten configurations, 16×H100 held fixed, SGLang v0.5.9 and GLM-5 FP8 held fixed, three knobs varied one at a time. It is a real ablation with real per-knob deltas — of configuration parameters, with every feature left on throughout. The algorithmic levers stay bundled, so the edge stands as written; what changed is that the corpus now knows a fourth lever class exists and outweighs the ones it was arguing about (see the section above). — new edge 1a: the parallelism decomposition is the largest measured lever in this wiki and no page here covers it — needs: a T1/T2 source on TP/PP/EP for inference, or a second cluster's split swept the same way. · earlier hunt 2026-08-08 — nothing ≥ bar. Two searches, and the field's output has the wrong shape twice over. The arXiv literature is method papers — ML-SpecQD, EAGLE, Ouroboros, SPEQ — each reporting its own technique against a baseline it configured (2.07×, 2.8×, 56.3% latency), which is a lever measured by its author, not levers measured against each other. The engine comparisons that do hold hardware fixed (Spheron, Yotta Labs, LeetLLM, markaicode on H100) are vendor and affiliate blog roundups, T3/T4, and they compare whole engines rather than levers, so they fail the bar and the question both. Not re-hunted before 2026-08-22. · 2026-09-10 adds a hazard to the edge rather than closing any of it: openmetal-tokens-per-second points out that a total TPS figure conflates prefill and decode, which its measurements put 10× apart. Any future ablation this edge asks for has to report the two phases separately or it answers nothing. HALF-CLOSED again 2026-09-23 (research pass, curator's request) by batchllm (Microsoft, MLSys Industry Track, T1). The first algorithmic levers measured one at a time here, by people who did not build them: on vLLM v0.6.4, chunked prefill +0.6% / +3.3% and prefix caching +14% / +19% (A100 / MI200), with sorting the requests before the engine worth more than both (6 → 13.02). Still missing: paging, batching, speculation and quantization, per phase, and on more than one workload. New edge 1b: the same one-at-a-time table for the remaining four levers, with prefill and decode reported separately — needs: T1/T2 on fixed hardware. The analytic side has a candidate method, arXiv 2607.05876 ("Floor First"), which assigns each lever to the hardware limit it moves but has not yet published its measured sweep; not ingested, noted here for the next hunt.
  2. vLLM against SGLang, by someone running neither. The spoke holds a detailed feature inventory of vllm and a detailed production benchmark on sglang, and no comparison between them. — needs: a T1/T2 head-to-head at stated concurrency and hardware.
  3. The browser regime, benchmarked. litertjs's 3× and 5–60× figures are vendor claims and the regime has no independent measurement at all. — needs: a T2 WASM/WebGPU/WebNN comparison.
  4. Quantization's sign, per phase. cloudflare-kimi-glm-serving showed the lever helps decode and hurts prefill, which contradicts how the mechanism pages state it. One deployment reported it. — needs: a second independent measurement separating the phases.

Coverage edges (added 2026-08-08, at the curator's request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

  1. The attention variants that shrink the cache. kv-cache is the spoke's central constraint and the architectural answers to it — multi-query and grouped-query attention, sliding-window attention, DeepSeek's latent attention — have no page. — needs: the originating papers, T1. Two thirds closed 2026-08-17, from a secondary: serret-attention-primer (T2, expository) derives both, and grouped-query-attention and latent-attention now exist alongside the spoke's first page on attention-mechanism itself. What is closed is the mechanism; what is not is the evidence, and the two halves should not be confused. No measurement of either lever is held anywhere here — no cache saving, no throughput, no quality ablation — so the pages state what the architectures do and nothing about what they buy. The originating papers are still the ask: Ainslie et al. (GQA, for the uptraining procedure and the quality/speed trade) and the DeepSeek-V2 report (MLA). Sliding-window attention remains untouched — no page, no source — and it is the one lever that attacks N_tokens rather than the per-token cost. Worth adding to the ask: TransMLA (arXiv:2502.07864), for converting trained GQA/MHA weights into the latent form.
  2. Constrained decoding. Grammar- and schema-constrained generation (Outlines, XGrammar, llama.cpp's GBNF) is a serve-time mechanism that changes the sampling step token-sampling describes, and it is absent. — needs: a paper or the implementation's own design doc, plus any overhead figure. (Sharpened 2026-08-15: it is now also the mechanism most likely to collide with text-watermarking, since one needs the freedom the other removes.)
  3. Prefix caching. Reusing a shared prompt prefix across requests is the cheapest lever most servers ship, and it appears nowhere beside kv-cache-isolation. — needs: vLLM or SGLang design docs, plus a hit-rate or latency number. (Its upstream sibling arrived first: okf-token-handoff reuses the integers rather than the cache computed from them.)
  4. Tokenization, the step before prefill. BPE and its variants set how many tokens a prompt costs, and the spoke starts one step later. — needs: the BPE/SentencePiece papers or a tokenizer's docs. Partly opened 2026-08-15, and not by a mechanism source: tokenization now exists with a measured cost (~19 ms fixed, okf-token-handoff) and the vocabulary-compatibility rule, but nothing on how a tokenizer is built or trained. The edge stands as written. Sidestepped rather than answered, 2026-10-07. byteification (Minixhofer et al., Nature, T1) retrofits an existing subword LLM to operate directly on bytes — no tokenizer step, no vocabulary — at under 1% of a pretraining budget, matching subword baselines on general benchmarks and clearly beating them on character-level tasks. It settles nothing about how BPE or SentencePiece are built; it's evidence that the step could be designed away instead. The edge as originally written — a mechanism source for subword tokenizer construction — still stands.
  5. Watermarking, from the people who measured it. text-watermarking rests entirely on a vendor announcement that publishes no detection rate at all. — needs: the SynthID-Text paper (Google DeepMind, Nature 2024) and/or Aaronson's 2022 proposal, T1, for the detection statistics and the distortion-free claim (added 2026-08-15).

Where inference runs, and what binds it there

Inference reads as distinct regimes, not one pipeline. The regimes sort by how the runtime reaches the hardware and what constraint binds there — and a fourth sorting axis, raw latency, turned out to reach past the serving layer into application design.

The ladder of access

Inference now reads as distinct regimes, not one. vllm + continuous-batching + flash-attention describe the datacenter regime, where the goal is maximizing GPU utilization across many concurrent users (the binding constraint is keeping an expensive GPU busy). llama-cpp

A third regime — in-browser (added 2026-07-10). The edge splits again: beneath native on-device (llama-cpp/Ollama talking to CUDA/Metal) sits inference inside the web page, where the binding constraint is neither GPU utilization nor raw VRAM but the browser sandbox — no direct GPU access, so the runtime reaches the silicon through portable APIs (WASM/XNNPACK on CPU, WebGPU on GPU, emerging WebNN on NPU). litertjs (Google's .tflite runtime for the web, ex- TensorFlow.js) is the founding instance, claiming ~3× over prior web runtimes and 5–60× GPU/NPU-over-CPU (vendor figures, unbenchmarked). Two things this regime teaches the spoke: (1) the pipeline is defined by how it reaches the hardware — datacenter GPU → native-edge → browser is a ladder of access, not just of size; and (2) its motivation is privacy + zero server cost + latency (nothing leaves the device), a different objective function from the datacenter's utilization or the native edge's fit-in-memory. Scope caveat: litertjs is general-ML (vision/audio/embeddings), not LLM-only — it enters the spoke as inference-execution mechanics on browser hardware (LLMs run on the same stack via the sibling MediaPipe), so the browser regime is broader than LLMs. This is the spoke's first deliberate step past strictly-LLM serving into the general on-device inference runtime it shares a discipline with.

The edge regime's tooling ecosystem, now mapped (2026-06-21). The on-device regime had only two engine pages (llama-cpp, vllm) and the claim that llama.cpp is "the core of Ollama/LM Studio." A T4 curated awesome-list, llms-local-list, supplies the layer above: local-llm-stack separates platforms (turnkey apps — LM Studio, Jan, LocalAI) from engines (the runtimes — Ollama, llama.cpp, vLLM, SGLang, MLX). The useful structural fact is that most people "run a model locally" by picking a platform and never choosing an engine, while vllm/SGLang straddle both regimes (run on a single box as readily as a cluster). It's a map, not a measurement — no benchmarks — so it broadens the spoke's coverage of the edge regime without touching the standing which-lever-bought-what question.

vLLM as the datacenter regime's convergence point (refreshed from the GitHub repo, 2026-06-11). The vllm page, previously grounded only in the docs, is now anchored to its primary source — and the repo makes plain that vLLM is not one lever but all of them in a single engine: PagedAttention + continuous-batching + chunked prefill + prefix caching + speculative-decoding (n-gram/suffix/ EAGLE/DFlash) + quantization (FP8/MXFP4/NVFP4/INT8/INT4/GPTQ/AWQ/GGUF) + flash-attention kernels

The axis the ladder does not capture — latency, and what it makes unnecessary

The ladder sorts by access and by binding constraint. One provider sorts by neither.

The regimes above are sorted by how the runtime reaches the hardware (datacenter GPU → native edge → browser) and what constraint binds (utilization → fit-in-memory → sandbox). cerebras-inference adds a data point that stresses a different axis: raw per-token latency. It is a hosted provider (not an installable engine like vllm/llama-cpp) running on a wafer-scale processor, and the vendor's pitch is seconds-to-milliseconds. Same prefill/decode pipeline and kv-cache — a hardware substrate chosen to minimize latency rather than maximize multi-tenant throughput.

The genuinely new move is that the source, designing-for-cerebras, reasons from the serving layer up into the application: it argues that much of an LLM app's architecture — job queues, always-on streaming, backgrounded agent loops, voice-filler UI — is latency compensation, built only to hide slow inference, and should be deleted once inference is fast. fast-inference-architecture captures that as a reusable, provider-agnostic idea with a one-line diagnostic ("was this built because LLM calls are slow?"). So the spoke now has a serving-side view (regimes/levers) and an application-side corollary: past a latency threshold, the simplest architecture becomes the fastest. Caveat: all speed figures here are vendor claims (T2) — this extends the standing which-lever-bought-what gap onto a hardware lever, still without an independent benchmark.

Put together, the ladder answers where can this model run and the latency axis answers what does the software around it stop needing. Both stay unresolved in the same way: every figure on either is a vendor claim, which is why the benchmark question below has outlived every source that touched it.

The rungs are converging, and one of them brought a disk (2026-09-04)

The ladder was built on an assumption the sources kept confirming: each rung has its own machinery because each has its own binding constraint. Datacenter gets batching and paging because a GPU must stay busy; the native edge gets quantization because the model must fit at all. omlx runs the datacenter machinery on the edge rung — paged block cache with prefix sharing and copy-on-write, continuous-batching, LRU model eviction, an OpenAI/Anthropic API — on one Mac, for one user, in a menubar app with 21k stars and a signed DMG that outsells its own Python wheels twenty to one.

That is not a new regime; it is the third rung's constraint changing. Unified memory on Apple Silicon removes the fit-in-memory cliff that made quantization the edge lever, and what binds instead is concurrency and re-prefill inside one long session — which are the datacenter's problems at one-user scale. local-llm-stack's platform/engine split is the first casualty: oMLX is both rows at once.

The genuinely new mechanism is persistence. oMLX's cold cache tier writes evicted KV blocks to SSD as safetensors and restores them on a matching prefix after a server restart (kv-cache). Every lever this spoke holds makes prefill cheaper per occurrence; this one attacks the number of occurrences, and it is the first time the disk appears as a rung on the memory ladder rather than as where the weights live. Worth watching for a second instance before it becomes a claim about the field.

And it supplies the sharpest instance yet of the standing "which lever bought what" problem — from the wrong direction. oMLX's one hard number is not a lever's benefit but a lever's absence: fused Metal kernels give GLM-5.2 prefill 845 tok/s against ~29 without them, ~30×, on an M3 Ultra, and the build that most people produce from source does not compile them and says nothing at runtime. A 30× regression with no error message is a serving-stack failure mode this corpus has no other example of, and it argues that on the edge rung the dominant variable may be the build rather than the architecture. Every figure in that sentence is the author's own, on one machine, with no published methodology — the same caveat that has attached to every performance claim above it.

What the levers actually cost, once someone measures them

The mechanism pages describe levers in the abstract: batching, paging, quantization, speculative decoding. Two sources built or ran the whole stack instead of one piece of it, and both came back with a correction rather than a confirmation.

Built by hand — the bottleneck is not where the mechanism pages point

build-llm-runtime-from-scratch is the first source that isn't about one mechanism but implements all of them at once: a hand-written decode-only engine for Qwen2.5-Coder-7B on an H100 (C++/CUDA PTX) that stacks INT4 quantization, a paged kv-cache, warp-specialized attention, fused GEMV and CUDA-graph capture into one runnable path. As a source it does two things for the spoke. It grounds the mechanism pages in concrete kernel-level reality — the paged KV cache isn't a diagram, it's 16 tokens per 4 KiB page with a real synchronization bug at the page boundary (a __syncthreads() in a warp-branch silently corrupting softmax) — and it puts a hard T2 number on the standing which-lever-bought-what gap that most pages fill with tutorial figures.

Its durable contribution is a fourth instance of the spoke's recurring pattern: the bottleneck is per-step fixed overhead, not the modeled work. The single biggest win in the whole build was not a math kernel but wrapping decode in a CUDA graph — 280+ kernel launches per token collapsed into one driver submission, ~119 ms → ~17 ms, 7×. That rhymes exactly with the levers already here: continuous-batching amortizes per-request scheduling overhead, speculative-decoding amortizes the per-token forward pass, CUDA graphs amortize per-kernel launch cost — three altitudes of "stop paying a fixed cost once per small unit of work." The honest coda is that the finished engine is slower than llama-cpp on the same task (60 vs 200 tok/s), and the author says so: the payoff claimed is ownership of the decode stack, not speed. So it enters as pedagogy and a legibility artifact, not a competitive runtime — and as one engineer's single-model, single-GPU exercise with no independent check, the T2 numbers are illustrative, not a benchmark.

Run in production — a lever's sign depends on the phase it lands in

The hand-built engine measures one model on one GPU for one user. The other correction comes from the opposite end: hundreds of tenants on a fleet, where the same levers behave differently enough to reverse.

cloudflare-kimi-glm-serving is the first source here written from inside a running multi-tenant deployment rather than from a paper, a tutorial, or a single-GPU exercise, and it changes two things in the picture above.

Precision becomes a property of a phase, not of a deployment. The founding asymmetry — prefill compute-bound, decode bandwidth-bound — had been treated as an explanation of why inference is expensive. Cloudflare treats it as a configuration boundary. FP8 kv-cache buys capacity that only decode needs, and costs about 9% per-token speed that prefill would rather keep; INT4 weights speed decode up by half at low concurrency and slow prefill down by 15%, because prefill has to decompress before it can multiply. Both trades point opposite ways in the two phases, so the phases run as separate pools at different precisions (prefill-decode-disaggregation). This is a level above continuous-batching and paging, which improve utilization within one engine doing both jobs.

It also corrects a simplification this wiki carried: quantization is not uniformly a speedup. Cache quantization is a capacity buy that costs per-token latency; the FP8 configuration loses to BF16 at every concurrency both can serve, and wins only because BF16 dies at 64 concurrent requests. The lever's payoff is how many users fit, and reading it as "smaller means faster" gets the sign wrong half the time.

The optimizations created a correctness surface. Every mechanism in this spoke exists to pack more requests onto one GPU, and the endpoint of that is hundreds of requests reading and writing pages of one physical cache — where paging, batching and prefix reuse are all bookkeeping, and a mis-mapped page returns another request's tokens silently rather than crashing. kv-cache-isolation is the spoke's first mechanism page that is not about speed or memory at all: per-page generation tags, recorded expected mappings, validation before decode reads, and an aborted request when they disagree. It costs under 1% of throughput and tail latency, and Cloudflare says it is still working toward leaving it on everywhere — so even sub-1% is not yet free enough. The pattern worth carrying forward: multi-tenant efficiency and multi-tenant isolation are the same bookkeeping, seen from two directions.

Stack note: the deployment runs on sglang, which until now this wiki had only as a name in two lists. The spoke now holds a detailed feature inventory of vllm and a detailed benchmark run on SGLang, and no comparison between them.

The two sources agree on the uncomfortable part. In both, the win came from somewhere other than the modeled work — a driver submission in one, a phase boundary in the other — and in both, a lever this wiki had described as a speedup turned out to cost speed and buy something else (ownership in one, capacity in the other). The mechanism pages are not wrong; they are stated without the sign and without the phase, and those are exactly what a deployment decides.

Measured in percentages — the denominator does the talking

A third measurement, from the same author as the first (2026-08-15). okf-token-handoff removes the downstream tokenizer from a three-agent chain by passing the token array itself, and reports 28.0% off time-to-first-token on a 3B model and 37.8% on a 1.5B. Subtract instead of divide and both are the same number: 19.4 ms and 18.7 ms. Skipping tokenization saves a constant, and the two percentages differ only because the forward passes they are measured against differ. Against the pipeline's own 41.3-second wall clock the constant disappears entirely.

Nothing here is wrong — the author states the cause, and a TTFT win is a real win where first-token latency is what a user feels. What the section keeps noticing is that the shape of a published number decides how it reads. A ratio without its baseline is the same failure as a speedup without its phase: it travels further than the measurement supports. That matters directly for the spoke's oldest question, which lever bought what, because every candidate answer to it will arrive as a ratio.

The third lesson from this thread, then: ask what was subtracted before believing what was divided.

The sampling step is being asked to carry things that are not speed

Sampling has sat in this wiki as the cheap last step: pick a token, get out of the way. Two sources now use it for something else. kv-cache-isolation made the packing bookkeeping do correctness work, and claude-text-watermark makes the choice itself do evidentiary work — the sampler's randomness is replaced by a keyed function of the preceding tokens, so the emitted sequence can later be tested against that key text-watermarking. Neither buys throughput. Both are obligations the deployment has and the mechanism absorbs, which is a second reading of the pipeline the founding sources did not offer: the levers are not all levers.

What puts the watermark in this spoke rather than a policy one is what it depends on. The mark only lands where several tokens would serve equally well, so its strength is a function of the entropy left in the distribution at each step — the same slack that temperature and top-p control token-sampling. Anthropic states two consequences and stops: factual prose carries a sparse mark, and code carries almost none because "in very many cases" it has to be exact. The questions past that are ours, and all three are open:

The last of these is the older question in a new costume. The first serve-time mechanism that needs per-request entropy has arrived while the wiki still cannot say what shared-batch serving does to per-request sampling, and the source does not close the gap.

And the number is missing. The announcement publishes no detection rate — no true-positive rate, no false-positive rate, no minimum passage length, only "less information to go on" for short samples. That is the same shape as the spoke's oldest gap: a mechanism described qualitatively by the party with the measurements. A promised detection API may or may not come with figures.

The mechanism under the constraint, finally written down (2026-08-17)

For two and a half months this spoke described the consequences of attention without holding attention. kv-cache is the corpus's central constraint, flash-attention is about its IO cost and paged-attention-paper about its memory management — all downstream of an operation that had no page. serret-attention-primer supplies it, and attention-mechanism now holds it.

The useful framing is that softmax is a choice, not the definition. Attention is a database lookup where equality is relaxed into a similarity: keys and values, a query, and an output that is the values weighted by how well their keys match. The similarity is an arbitrary kernel κ, and standard attention is the instance κ(v,w) = exp(⟨v,w⟩/√d_QK) — the √d_QK there because ⟨v,w⟩ has variance d_QK under unit-variance inputs. Separating the structure from the kernel is what makes the rest of this section legible: the efficiency variants do not change the operation at all. They change which matrices exist and which get stored.

The cache formula names its own attack surface. Cached floats are 2 · N_layers · N_heads · N_tokens · d, times bytes per float. Every factor is a lever somebody has pulled, and the spoke now holds three of four: N_heads → grouped-query-attention; the per-head K and V themselves → latent-attention; bytes per float → quantization, the one with a measurement attached (cloudflare-kimi-glm-serving's FP8 686k → 1.37M tokens). N_tokens is sliding-window attention, and it is still absent. Read this way the levers stop being a list of tricks and become a factorization — which is the thing the wiki wanted from growth edge 5 and did not have.

The best technical detail is a negative one. MLA is exactly equivalent to a GQA/MHA model in the absence of positional embeddings — the factorizations are algebraic identities. RoPE breaks it, because RoPE is applied after K is built and inserts a position-dependent matrix between precisely the two factors MLA needs to pre-merge; leaving it there would not merely lose the speedup but add overhead. DeepSeek's fix appends a non-latent portion to key and query and puts RoPE only there, keeping the compute win and discarding the exact equivalence. That is the spoke's cleanest case of an efficiency mechanism whose binding constraint is its interaction with an unrelated architectural choice, rather than anything about the mechanism itself — and it belongs beside cloudflare-kimi-glm-serving's finding that a lever's sign depends on the phase it lands in. Both say the same thing: these mechanisms do not compose as cleanly as their descriptions imply.

The caveat that governs all three new pages. The source is expository, unreviewed, and reports no experiment. Its complexity and storage results are counted, not timed. So the spoke now knows what GQA and MLA do and still cannot say what either buys on hardware — no cache saving, no throughput, no quality ablation. That is the same gap the wiki keeps recording from the other direction, and it is why growth edge 5 is marked two-thirds closed rather than closed.

The unit everything is quoted in finally has a page (2026-09-10)

Every claim in this wiki is denominated in tokens per second, and until now the spoke had no page on the metric — only on the mechanisms it measures. openmetal-tokens-per-second (OpenMetal, T3, a bare-metal vendor's explainer) is a weak source that nonetheless closes a real hole, and tokens-per-second now exists.

The argument worth keeping is about reporting, not mechanism. Prompt TPS and eval TPS are different quantities — the first is prefill, parallel and compute-bound; the second is decode, sequential and bandwidth-bound — and the source measures the first running up to 10× the second. So any total tokens-per-second figure is largely a statement about the input-to-output ratio the benchmarker picked, and a long prompt with a short answer flatters any stack. This is prefill-decode-kv-cache's asymmetry, which the spoke has held since its founding day, restated as a way benchmarks go wrong. Growth edge 1 has been complaining that every throughput claim here belongs to a stack rather than a lever; this adds that some of them may not even be measuring the phase they appear to.

It also puts the spoke's first CPU numbers on the table, and they are the interesting kind of unsatisfying. Llama 3.2 3B and Llama 3.1 8B on an Intel Xeon Gold 6530 run ~52 and ~33 eval TPS; on an A100, ~171 and ~122. The GPU is 3.3–3.7× the CPU, which is a smaller gap than this spoke's GPU-centric sources would lead a reader to expect — and 2.7× the parameters costs only ~1.5× the throughput on both, which is not what a purely bandwidth-bound decode predicts. No methodology is published: no batch size, no concurrency, no quantization level, no context length, no engine. The numbers are held as an order of magnitude and the page says so.

The source's shape is itself evidence about the field. It names OpenVINO, IPEX, TensorRT and Hugging Face Optimum and never mentions vllm, sglang or llama-cpp — the tool list of a vendor selling CPU and private-cloud capacity, not GPU serving. Growth edge 1's 2026-08-08 hunt already found that the sources willing to hold hardware fixed are vendor blogs; this is another, and it sells the hardware it measured. The prompt-versus-eval reasoning stands on its own; the table does not.

One thing it adds that no page here had: a requirement. Code completion is given a target of 50–100 TPS per user. Everything else in this wiki measures what a stack delivers; nothing until now stated what a workload needs.

A lever that runs before the engine (2026-09-14)

vllm-semantic-router is the first source here that saves compute by deciding which model, or which mode, runs rather than by making a run cheaper. Its founding paper routes on one bit — does this prompt need chain-of-thought — and reports −48.5% tokens and −47.1% latency with accuracy up 10.2 points on MMLU-Pro. Read against the rest of the spoke that is a strange number: every other lever trades something (memory for the cache, a draft model for speculation, precision for quantization); this one claims to trade nothing, because the tokens it removes were reasoning that hurt. The catch is that the classifier is a model too, sitting in the request path at a cost nobody in the source measures, and a misroute is a wrong answer or a wasted long one. model-routing holds the mechanism and the open questions. A second, cheaper pre-engine lever arrived on 2026-09-23: in batchllm, sorting requests so shared prefixes sit together doubled vLLM's throughput on a prefix-heavy batch, more than the engine's own prefix caching and chunked prefill together. The product's larger claim — a pool of cheaper models matching a frontier one — is asserted with a single unmethoded benchmark and stays T3 inside a T1 page. Cross-spoke: the router's second author owns inferoa in ../agentic-tooling-wiki, a harness built on it; the models in the pool belong to ../llm-providers-wiki; Envoy and the Kubernetes operator are ../platform-ops-wiki material.

Confidence, not just cost, decides which forward pass runs (2026-09-28)

vllm-semantic-router routes on the prompt alone, before any inference happens. lev (introducing-lev) is the first source here that routes on the classifier's own uncertainty after it has already run: a laya encoder answers every question in ~95 ms, and only the specific questions it scores below a confidence threshold go on to a local LLM (llama.cpp, scoring GGUF log-probabilities rather than generating text). On Lev's own 144-case adversarial set this lifted accuracy from 61.1% (encoder alone) to 92.4% (gated) against 95.1% (LLM on everything), while an easier 120-case mix never escalated a single question — the gate pays its latency cost only where the accuracy gap is real. That is the missing half of model-routing's own open question about classifier cost: here the "classifier" is the cheap tier itself, and its cost is measured directly rather than assumed.

The source also names a failure mode no other page here has: a binary (yes/no) question's confidence is max(p, 1-p), a quantity that can never read below 0.5, so one global threshold silently never escalates that whole question type. The fix — a threshold per question type, not one number for the whole request — is a small but concrete lesson for any confidence-gated system, not just this one.

Cross-spoke. The router's runner-up was ../agentic-tooling-wiki (jev-code-reviewer uses the same TypeSafe Jev model for PR triage); filed here because the substance is the inference architecture, not an agent harness. The ecosystem Lev answers is already split across two other spokes: ../llm-providers-wiki holds TypeSafe's own launch post (system-one-models-and-jev, typesafe-ai); ../machine-learning-wiki holds the training method behind the fast tier's checkpoints (laya, open-jev-9b, typed-decision). Three independent teams — TypeSafe, Convai Innovations, and now Sotnikov — have converged on the same choice/score/noul typed-answer shape; this spoke's angle on all three is narrower and specific to it: what it costs, in latency and forward passes, to get a calibrated answer instead of just an answer.

Contradictions flagged

One, mild, and it is a sign error rather than a disagreement (2026-08-03). quantization, llama-cpp and build-llm-runtime-from-scratch all present lower precision as a memory-and-speed win, which holds for weights on a bandwidth-bound decode step. cloudflare-kimi-glm-serving measures the two other cases and both go the other way: FP8 KV cache is slower per token than BF16, and INT4 weights make prefill slower than FP8. Nobody is wrong — the earlier sources measured the single-user, weight-bound case and said so — but the generalization "quantization makes inference faster" does not survive, and the pages have been amended rather than one being overwritten.

Cross-spoke adjacency

Index — LLM Inference Wiki

Catalog of all pages, grouped by @type. The spine: synthesis (thesis), log.md (history), this file (catalog).

DefinedTerm (concepts / mechanisms)

SoftwareApplication (engines)

Person (entities)

Organization

ScholarlyArticle (source summaries, source: true)

Collection (source summaries, source: true)

TechArticle (source summaries, source: true)

Notes

Related pages

Links to

Linked from