Spokes.wiki Search About

Log — LLM Inference Wiki

Append-only history. Each entry starts with ## [YYYY-MM-DD] <op> | <title> where <op> is ingest, query, lint, or split, so grep "^## \[" log.md | tail -5 works.

[2026-07-19] ingest | Designing for Cerebras (Cerebras docs, URL-only, T2)

Routed by the hub from Telegram. First provider page in the spoke and the first source whose binding story is raw per-token latency rather than memory or batching. cerebras-inference — a hosted inference service on a wafer-scale processor (seconds→ms, vendor claim) — is the datacenter regime pushed to the latency extreme; typed SoftwareApplication, distinguished from installable engines (vllm/llama-cpp) as a provider you call over an API. The source designing-for-cerebras reasons from serving up into the application: job queues, always-on streaming, backgrounded agent loops and voice-filler UI are latency compensation to be deleted once inference is fast. Generalized that as the provider-agnostic concept fast-inference-architecture (diagnostic: “was this built because LLM calls are slow?”). Entity: cerebras-systems (Organization, source owner). Synthesis gains a “speed as its own axis — the application side” section; the vendor speed figures extend the standing which-lever-bought-what gap onto a hardware lever (no independent benchmark yet). Pages: 2 Thing + 1 idea + 1 org + 1 source = 5. Runner-up spoke: llm-providers-wiki (Cerebras-as-provider / speed benchmarks).

[2026-06-01] split | llm-inference-wiki created from _inbox cluster (3 sources)

Spun out by the hub router when three MachineLearningMastery walk-throughs arrived in one Telegram burst, all on LLM inference mechanics — too specific for any existing spoke (research-wiki is tools-for-thought / agentic products, not inference internals). Scaffolded from CLAUDE.template.md; domain = the mechanics of LLM inference and serving. Ingested all three (URL-only, source: true + url:):

[2026-06-09] ingest | +3 “beyond the basics” (vLLM/PagedAttention, FlashAttention, speculative decoding) — all-spokes cron test

Filled the synthesis “beyond the basics” open question with three authoritative sources: vllm (SoftwareApplication, src — PagedAttention = OS-paging the kv-cache, near-zero fragmentation + KV sharing; continuous-batching engine), flash-attention (DefinedTerm, src — IO-aware exact attention, tiling/fusion, 2–4× + linear memory), speculative-decoding (DefinedTerm, src — draft-and-verify, unchanged output distribution; the concrete sampling×serving coupling). Synthesis open question struck through (quantization still open); index gains a SoftwareApplication group. url-only. 7 → 10 pages.

[2026-06-10] ingest | Quantization + llama.cpp — all-spokes pass (the data-type lever + the edge regime)

Two new pages closing the explicitly-named “quantization still absent” gap. quantization (DefinedTerm, source, HF docs) — lower-precision weights/activations/KV (int8/int4/FP8 from fp16/bf16; weight-only vs weight+activation; post-training GPTQ/AWQ/bitsandbytes/GGUF + FP8 + extreme 1–2-bit AQLM/VPTQ). The fourth production lever after PagedAttention/flash-attention/ speculative-decoding, but orthogonal — it acts on the data type and also shrinks the kv-cache (relaxing the continuous-batching bottleneck). llama-cpp (SoftwareApplication, source, Wikipedia) — Gerganov’s dependency-free C/C++ engine, the de-facto core of Ollama/LM Studio; GGUF 2–8-bit format; runs quantized models on CPU/consumer GPU. Together they introduce two serving regimes: datacenter (vLLM + batching, keep the GPU busy) vs on-device (llama.cpp + quantization, fit the model at all) — new synthesis section. Folded into synthesis (open-Q resolved + “two serving regimes” section) + index (new DefinedTerm + SoftwareApplication rows). No contradictions. 10 → 12 pages.

[2026-06-11] ingest | vLLM GitHub repo (github.com/vllm-project/vllm)

Telegram drop, hub-routed → llm-inference-wiki (clean single match: vLLM/PagedAttention are this spoke’s founding subject). Re-seen subject: vllm already existed as a docs-URL ingest → refreshed in place, re-anchored to its primary source (the GitHub repo) and the url: switched docs→repo (updated: 2026-06-11). Folded in the repo’s fuller picture: vLLM bundles every datacenter lever in one engine — PagedAttention + continuous-batching + chunked prefill + prefix caching + speculative-decoding (n-gram/suffix/EAGLE/DFlash) + quantization (FP8/MXFP4/NVFP4/INT8/INT4/ GPTQ/AWQ/GGUF) + flash-attention kernels + tensor/pipeline/expert/context parallelism — across NVIDIA/AMD/CPU/TPU/Gaudi and 200+ model architectures (MoE, multimodal). synthesis.md: added “vLLM as the datacenter regime’s convergence point” note under the two-regimes section; index entry broadened. No new pages (idempotent refresh). Benchmark caveat preserved (no first-party numbers). Site rebuild follows.

[2026-06-15] ingest | FlashAttention primary paper (Dao et al. 2022) — T1 anchor + quantified benchmarks

Quality cycle, T1-floor raise. The flash-attention page was anchored to the implementation repo; added the primary paper as a distinct source: flash-attention-paper (ScholarlyArticle, T1) — Dao, Fu, Ermon, Rudra, Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022 (arXiv:2205.14135). Supplies the hardware/model-specific numbers the synthesis open questions wanted: 15% BERT-large / 3× GPT-2 / 2.4× long-range-arena training speedups, linear (not quadratic) attention memory, Path-X 61.4% / Path-256 63.1%, formal IO-complexity proof. Partly closes the quantified benchmarks + memory math open questions (training-side; serving-side still open). Concept/paper split mirrors optimization-wiki’s NFL pattern. Found via WebSearch; figures from the abstract (PDF body not fully extracted) — noted on the page. Linked from concept page, synthesis (2 open qs), index (new ScholarlyArticle section). 1 new page.

[2026-06-16] ingest | PagedAttention paper (Kwon et al., SOSP 2023) — quality-cycle floor-raise

Closed the spoke’s standing inference-serving benchmark gap. Added paged-attention-paper (ScholarlyArticle, source:true, T1, arXiv:2309.06180) — the peer-reviewed primary behind vllm‘s PagedAttention: OS-paging for the kv-cache, near-zero fragmentation + KV sharing, 2–4× throughput over FasterTransformer/Orca at equal latency. Refreshed vllm (Benchmark-caveat now cites the 2–4× primary; PagedAttention headline links the paper) and kv-cache (fragmentation/serving-cost grounded) in place; folded into synthesis open questions (the serving controlled-study half of “quantified benchmarks” now closed — only the full which-lever decomposition remains; “memory math” serving cost grounded). Numbers are abstract-only (PDF body not extracted); recorded on the page. Entity discovery: no new nodes — authors named inline (mirrors flash-attention-paper precedent); SOSP venue too thin to page. +1 page (→14). Highest-value action this cycle: raises the spoke’s T1 floor and answers a named gap.

[2026-06-17] ingest | How does vLLM work? (Amit Shekhar / Outcome School) — accessible secondary

Hub-routed from Telegram (outcomeschool.com/blog/how-does-vllm-work). Clean single-spoke route — vLLM inference mechanics is this spoke’s core. Dedup: subject already paged (vllm T1 repo + paged-attention-paper T1). Added how-does-vllm-work (TechArticle, source:true, T3 — a single-author educational blog post, no benchmarks/first-party numbers). Gap-relevance: does NOT advance the standing open question (which-lever-bought-what benchmark decomposition) — recorded as a pedagogical on-ramp only (OS-paging analogy, the “50-of-2000 tokens reserved-but-idle” fragmentation framing, prefix/beam-search sharing). Integrated: cross-linked into vllm (“Accessible secondary” section + Related) and noted in synthesis under the benchmark open-question (a no-numbers secondary, not movement on the gap). Entity discovery: no nodes minted — author Amit Shekhar / publisher Outcome School are low graph-signal for an inference-mechanics spoke (no existing entity-index match), recorded inline (same call as prior T3 content-site authors). Ran avoid-ai-writing over the new prose. +1 page (14 -> 15).

[2026-06-18] ingest | DefinedTerm enrichment pass (subagent)

Targeted deepening of the thinnest DefinedTerm pages with fetched primary/secondary sources. Added three source pages and the hard numbers/primaries they back:

  • speculative-decoding-paper (Leviathan et al., ICML 2023; ScholarlyArticle, T1, arxiv:2211.17192) — the peer-reviewed primary behind speculative-decoding, which previously cited only a HF blog. Supplies the speculative-sampling distribution-preservation argument (accept-or-resample, provably the target distribution) and the 2–3× on T5-XXL benchmark. Enriched speculative-decoding: the “unchanged distribution” claim is now a theorem with provenance, acceptance-rate framing of the numbers.
  • flash-attention-2-paper (Dao, 2023; ScholarlyArticle, T1, arxiv:2307.08691) — the FA-2 rung the flash-attention page named without figures. Adds the utilization story: 25–40%→50–73% of A100 peak FLOPs/s, ~2× over FA-1, up to 225 TFLOPs/s. Folded into flash-attention “Versions”.
  • continuous-batching-anyscale (Anyscale, 2023; TechArticle, T2, anyscale.com) — bridges the USENIX-only Orca (Yu et al., OSDI 2022) primary to the spoke and decomposes the win: ~4× / ~8× / ~23× (static-opt / continuous alone / continuous + PagedAttention). Enriched continuous-batching with Orca provenance, the harder numbers, the batching×KV-paging-compound reading, and a Related section.

Already-developed llm-inference umbrella: added a Related section + bumped (no padding). Blocked fetch: Orca paper itself — USENIX (usenix.org/system/files/osdi22-yu.pdf and the presentation page) returned HTTP 403 to WebFetch; routed around via the Anyscale T2 secondary that attributes and quantifies it (one alternative, per no-fabrication rule). All paper figures are abstract-level (PDF bodies not extracted); recorded on each source page. Updated index.md (+3 source pages). Ran avoid-ai-writing (edit mode) over new prose. No build/verify, no git. +3 pages (15 → 18).

[2026-06-21] ingest | LLMs-local curated list (via @DanKornas tweet) → edge-regime ecosystem

Source: 0xSojalSec/LLMs-local GitHub awesome-list, surfaced by a Dan Kornas promotional tweet (routed in by the hub from the Telegram channel). A curated map of the local-LLM world: platforms, engines, models, builder tooling, UIs, hardware, tutorials, communities.

Classification: a broad roundup touching three spokes. Dominant in-scope substance = the local inference/serving stack (Ollama, llama.cpp, vLLM, SGLang, MLX vs. LM Studio/Jan/LocalAI) → this spoke. Runner-ups (agent frameworks/MCP/RAG/coding agents → agentic-tooling-wiki; model discovery/providers → llm-providers-wiki) recorded as cross-spoke context in the source page, not fragmented or re-routed.

What it added:

  • llms-local-list (Collection, T4 — single-maintainer curated link list off a marketing tweet; no benchmarks/measurement). url-only (github repo). The wiki’s first map of the on-device regime’s ecosystem.
  • local-llm-stack (DefinedTerm) — the platform-vs-engine split the list draws: turnkey platforms (LM Studio/Jan/LocalAI) over engines (Ollama/llama.cpp/vLLM/SGLang/MLX). Names the platform tier the spoke had only implied via llama-cpp (“core of Ollama/LM Studio”). Notes vLLM/SGLang straddle both regimes.
  • Touched llama-cpp (engine-beneath-platforms cross-link) and synthesis (new para under “two serving regimes”: edge ecosystem now mapped — a map, not a measurement, so no movement on the standing benchmark question).

Fetch: WebFetch HTTP 402 on x.com → fell back to firecrawl (per HUB edge rule), clean scrape. Author/maintainer (Dan Kornas / 0xSojalSec) noted inline, not as entity nodes — how-does-vllm-work precedent (low graph signal). Ran avoid-ai-writing (detect) over new prose. +2 pages (18 → 20).

[2026-07-10] ingest | LiteRT.js — Google’s in-browser AI inference (hub-routed, Telegram)

Ingested the Google Developers Blog announcement of LiteRT.js — a JS binding of LiteRT (ex-TFLite) for running .tflite models in the browser via WASM/XNNPACK (CPU) + WebGPU/ML Drift (GPU) + WebNN (NPU); succeeds TensorFlow.js (~3× claimed, 5–60× GPU/NPU-over-CPU). New litertjs (SoftwareApplication/engine, source, T2) + new concept browser-inference (the in-browser execution regime; WASM/WebGPU/WebNN hardware- access layer). Extends the synthesis “two serving regimes” → a third: datacenter GPU → native-edge → in-browser, a ladder of hardware access (privacy/zero-server-cost objective). Updated local-llm-stack (browser surface beside native engines). Scope note: LiteRT.js is general-ML (vision/audio/embeddings), not LLM-only — enters as inference-execution mechanics on browser hardware (LLMs run on the same stack via MediaPipe); the spoke’s first deliberate step past strictly-LLM serving. T2 vendor blog; perf figures unbenchmarked (standing which-lever question). avoid-ai-writing applied (clean). +2 pages, 1 updated.

[2026-07-23] ingest | How to build your own LLM runtime from scratch (Towards Data Science)

Routed from the hub (route in ../log.md). URL-only, T2 — a first-party, measured build-log (real before/after numbers, named bugs), but one engineer’s single-model/single-GPU exercise with no independent check. New page: build-llm-runtime-from-scratch (TechArticle). Anubhab Banerjee builds a decode-only inference engine for Qwen2.5-Coder-7B on an H100 in C++/CUDA PTX: INT4 group-wise quantization (.nanoqwen format), paged kv-cache (16 tok/4 KiB page), warp-specialized attention on Hopper TMA, fused GEMV, and CUDA-graph decode capture. Greedy sampling only; no serving/batching/tokenizer — explicitly the inference-engine layer, which is dead-center this spoke. Dedup: no build-from-scratch page existed; every component it implements is an existing page (quantization, kv-cache, flash-attention, paged-attention-paper) — linked, not duplicated. Scoped exactly to llm-inference, not the serving/agent layers, so no cross-spoke contention. Gap-relevance: (1) puts kernel-level concrete reality under the mechanism pages (the paged cache as a real page-boundary sync bug, not a diagram); (2) a hard T2 number on the standing which-lever-bought-what gap. Synthesis: added a section. Its durable point is a fourth instance of the spoke’s per-step-fixed-overhead pattern — CUDA graphs (7×, ~119→17 ms/tok) amortize per-kernel launch cost the way continuous-batching amortizes per-request scheduling and speculative-decoding amortizes the per-token forward pass. Recorded the honest coda: the engine is slower than llama-cpp (60 vs 200 tok/s); payoff claimed is ownership, not speed — enters as pedagogy/legibility, not a competitive runtime. Entities: author Anubhab Banerjee not paged (byline, no distinct thesis; evidence-only). No org/event. Index fixed into the TechArticle section (initial insert landed under ScholarlyArticle). Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-03] ingest | Smaller, faster, safer: running Kimi and GLM at scale (Cloudflare)

Routed from the hub (Telegram). T2 — first-party engineering blog from the operator, product to sell (Workers AI), self-reported and unreproducible outside Cloudflare. Rated above the usual vendor post because every table keeps a baseline the vendor’s own choice loses to (FP8 cache is slower per token; INT4 prefill is slower), and accuracy is benchmarked rather than asserted. New pages: cloudflare-kimi-glm-serving (TechArticle, source), prefill-decode-disaggregation (DefinedTerm/mechanism), kv-cache-isolation (DefinedTerm/mechanism), sglang (SoftwareApplication). Updated: kv-cache (byte budget, at last: 686k tokens BF16 -> 1.37M FP8 for Kimi K2.6, and the ranking that the cache fills memory before the weights do), quantization (both benchmark tables, plus the correction that quantization is not uniformly a speedup), synthesis, index. Dedup: quantization and kv-cache already existed and were refreshed in place, not duplicated. SGLang had been named twice in passing (local-llm-stack, llms-local-list) and never described — paged now that there is evidence behind it, scoped strictly to what this source attests. Gap-relevance: closes most of the standing which-lever-bought-what question for the two quantization levers (separate experiments, same hardware, five concurrency levels each) and quantifies the memory math question that had been asserted qualitatively since 2026-06-01. Both bullets amended rather than deleted — the algorithmic levers stay bundled and unattributed. Synthesis: new section “The phases separate, and the cache acquires a second cost.” Two claims — precision is a property of a phase, not a deployment (hence disaggregated pools); and the packing optimizations produced a correctness surface, since multi-tenant efficiency and multi-tenant isolation are the same bookkeeping seen from two directions. First contradiction flagged in this spoke (previously “None yet”): a sign error, not a disagreement. Earlier sources present lower precision as a memory-and-speed win, true for weights on a bandwidth-bound decode; this source measures the two other cases and both go the other way. Both claims kept, page amended, neither overwritten. Held back deliberately: the security framing. No incident, bug or attack is described — “even a one-in-a-billion mistake would show up regularly” is a statement about request volume, and the wiki records the tag scheme as defence in depth rather than a response to known leakage. The source argues from error rates and not from a threat model, so kv-cache-isolation does not extend it into one. Entities: reused cross-wiki, none duplicated — cloudflare (cloud-wiki), glm-52 and z-ai (llm-providers-wiki). Moonshot has no hub entity node; Kimi K2.6 named in prose, not paged. Verify deferred per hub policy (content-only). avoid-ai-writing run.

Related pages

Links to