Spokes.wiki Search About

Log — LLM Inference Wiki

Append-only history. Each entry starts with ## [YYYY-MM-DD] <op> | <title> where <op> is ingest, query, lint, or split, so grep "^## \[" log.md | tail -5 works.

[2026-09-04] ingest | oMLX — Mac inference server with a tiered KV cache (GitHub, URL-only, T1)

Routed by the hub from Telegram (one of three sources arriving together). omlx is an Apache-2.0 inference server for Apple Silicon — 21,412★ / 1,827 forks / 1,272 open issues / 2,549 commits, created 2026-02-13, read 2026-09-04 — and it is the first source here that assembles the datacenter serving stack for a single laptop: paged block cache with copy-on-write and prefix sharing, continuous-batching over mlx-lm's BatchGenerator, an EnginePool with LRU eviction and per-model TTL, and both OpenAI and Anthropic API shapes.

Two things earned page edits rather than a summary. The cold cache tier is a disk: evicted KV blocks are written as safetensors and restored on a matching prefix after a server restart, so kv-cache gains a section — every other lever in this spoke makes prefill cheaper per occurrence, this one attacks the number of occurrences, and it is the first appearance of the disk as a rung on the memory ladder. And the project's one hard number is a lever's absence: fused Metal kernels give GLM-5.2 prefill 845 vs ~29 tok/s on an M3 Ultra (~30×), a plain pip install -e . does not build them, and the fallback is silent (issue #2137).

local-llm-stack corrected: its platform-over-engine split describes the llama.cpp lineage, not the category — oMLX is both rows at once, because MLX ships a generation loop and no server. New entity mlx (two sources: llms-local-list names it, oMLX is built on it); apple linked cross-wiki to ../business-messaging-wiki, not re-paged. Synthesis gains "The rungs are converging, and one of them brought a disk" under the ladder-of-access section. Every performance figure is first-party, one machine, no methodology — omlx.ai/benchmarks unread, and a cold-tier hit rate is the measurement that would settle whether the SSD tier is a lever or a feature. Release split worth re-reading: the macOS 26–27 DMG has 22,343 downloads against 1,982 for the Sequoia build. Pages: 1 source + 1 framework

  • 2 refreshed = 4. Runner-up spoke: ../foss-applications-wiki (a signed desktop app an end user installs); routed here because the substance is serving mechanics.

[2026-07-19] ingest | Designing for Cerebras (Cerebras docs, URL-only, T2)

Routed by the hub from Telegram. First provider page in the spoke and the first source whose binding story is raw per-token latency rather than memory or batching. cerebras-inference — a hosted inference service on a wafer-scale processor (seconds→ms, vendor claim) — is the datacenter regime pushed to the latency extreme; typed SoftwareApplication, distinguished from installable engines (vllm/llama-cpp) as a provider you call over an API. The source designing-for-cerebras reasons from serving up into the application: job queues, always-on streaming, backgrounded agent loops and voice-filler UI are latency compensation to be deleted once inference is fast. Generalized that as the provider-agnostic concept fast-inference-architecture (diagnostic: "was this built because LLM calls are slow?"). Entity: cerebras-systems (Organization, source owner). Synthesis gains a "speed as its own axis — the application side" section; the vendor speed figures extend the standing which-lever-bought-what gap onto a hardware lever (no independent benchmark yet). Pages: 2 Thing + 1 idea + 1 org + 1 source = 5. Runner-up spoke: ../llm-providers-wiki (Cerebras-as-provider / speed benchmarks).

[2026-06-01] split | llm-inference-wiki created from _inbox cluster (3 sources)

Spun out by the hub router when three MachineLearningMastery walk-throughs arrived in one Telegram burst, all on LLM inference mechanics — too specific for any existing spoke (../research-wiki is tools-for-thought / agentic products, not inference internals). Scaffolded from CLAUDE.template.md; domain = the mechanics of LLM inference and serving. Ingested all three (URL-only, source: true + url:):

[2026-06-09] ingest | +3 "beyond the basics" (vLLM/PagedAttention, FlashAttention, speculative decoding) — all-spokes cron test

Filled the synthesis "beyond the basics" open question with three authoritative sources: vllm (SoftwareApplication, src — PagedAttention = OS-paging the kv-cache, near-zero fragmentation + KV sharing; continuous-batching engine), flash-attention (DefinedTerm, src — IO-aware exact attention, tiling/fusion, 2–4× + linear memory), speculative-decoding (DefinedTerm, src — draft-and-verify, unchanged output distribution; the concrete sampling×serving coupling). Synthesis open question struck through (quantization still open); index gains a SoftwareApplication group. url-only. 7 → 10 pages.

[2026-06-10] ingest | Quantization + llama.cpp — all-spokes pass (the data-type lever + the edge regime)

Two new pages closing the explicitly-named "quantization still absent" gap. quantization (DefinedTerm, source, HF docs) — lower-precision weights/activations/KV (int8/int4/FP8 from fp16/bf16; weight-only vs weight+activation; post-training GPTQ/AWQ/bitsandbytes/GGUF + FP8 + extreme 1–2-bit AQLM/VPTQ). The fourth production lever after PagedAttention/flash-attention/ speculative-decoding, but orthogonal — it acts on the data type and also shrinks the kv-cache (relaxing the continuous-batching bottleneck). llama-cpp (SoftwareApplication, source, Wikipedia) — Gerganov's dependency-free C/C++ engine, the de-facto core of Ollama/LM Studio; GGUF 2–8-bit format; runs quantized models on CPU/consumer GPU. Together they introduce two serving regimes: datacenter (vLLM + batching, keep the GPU busy) vs on-device (llama.cpp + quantization, fit the model at all) — new synthesis section. Folded into synthesis (open-Q resolved + "two serving regimes" section) + index (new DefinedTerm + SoftwareApplication rows). No contradictions. 10 → 12 pages.

[2026-06-11] ingest | vLLM GitHub repo (github.com/vllm-project/vllm)

Telegram drop, hub-routed → llm-inference-wiki (clean single match: vLLM/PagedAttention are this spoke's founding subject). Re-seen subject: vllm already existed as a docs-URL ingest → refreshed in place, re-anchored to its primary source (the GitHub repo) and the url: switched docs→repo (updated: 2026-06-11). Folded in the repo's fuller picture: vLLM bundles every datacenter lever in one engine — PagedAttention + continuous-batching + chunked prefill + prefix caching + speculative-decoding (n-gram/suffix/EAGLE/DFlash) + quantization (FP8/MXFP4/NVFP4/INT8/INT4/ GPTQ/AWQ/GGUF) + flash-attention kernels + tensor/pipeline/expert/context parallelism — across NVIDIA/AMD/CPU/TPU/Gaudi and 200+ model architectures (MoE, multimodal). synthesis.md: added "vLLM as the datacenter regime's convergence point" note under the two-regimes section; index entry broadened. No new pages (idempotent refresh). Benchmark caveat preserved (no first-party numbers). Site rebuild follows.

[2026-06-15] ingest | FlashAttention primary paper (Dao et al. 2022) — T1 anchor + quantified benchmarks

Quality cycle, T1-floor raise. The flash-attention page was anchored to the implementation repo; added the primary paper as a distinct source: flash-attention-paper (ScholarlyArticle, T1) — Dao, Fu, Ermon, Rudra, Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022 (arXiv:2205.14135). Supplies the hardware/model-specific numbers the synthesis open questions wanted: 15% BERT-large / 3× GPT-2 / 2.4× long-range-arena training speedups, linear (not quadratic) attention memory, Path-X 61.4% / Path-256 63.1%, formal IO-complexity proof. Partly closes the quantified benchmarks + memory math open questions (training-side; serving-side still open). Concept/paper split mirrors optimization-wiki's NFL pattern. Found via WebSearch; figures from the abstract (PDF body not fully extracted) — noted on the page. Linked from concept page, synthesis (2 open qs), index (new ScholarlyArticle section). 1 new page.

[2026-06-16] ingest | PagedAttention paper (Kwon et al., SOSP 2023) — quality-cycle floor-raise

Closed the spoke's standing inference-serving benchmark gap. Added paged-attention-paper (ScholarlyArticle, source:true, T1, arXiv:2309.06180) — the peer-reviewed primary behind vllm's PagedAttention: OS-paging for the kv-cache, near-zero fragmentation + KV sharing, 2–4× throughput over FasterTransformer/Orca at equal latency. Refreshed vllm (Benchmark-caveat now cites the 2–4× primary; PagedAttention headline links the paper) and kv-cache (fragmentation/serving-cost grounded) in place; folded into synthesis open questions (the serving controlled-study half of "quantified benchmarks" now closed — only the full which-lever decomposition remains; "memory math" serving cost grounded). Numbers are abstract-only (PDF body not extracted); recorded on the page. Entity discovery: no new nodes — authors named inline (mirrors flash-attention-paper precedent); SOSP venue too thin to page. +1 page (→14). Highest-value action this cycle: raises the spoke's T1 floor and answers a named gap.

[2026-06-17] ingest | How does vLLM work? (Amit Shekhar / Outcome School) — accessible secondary

Hub-routed from Telegram (outcomeschool.com/blog/how-does-vllm-work). Clean single-spoke route — vLLM inference mechanics is this spoke's core. Dedup: subject already paged (vllm T1 repo + paged-attention-paper T1). Added how-does-vllm-work (TechArticle, source:true, T3 — a single-author educational blog post, no benchmarks/first-party numbers). Gap-relevance: does NOT advance the standing open question (which-lever-bought-what benchmark decomposition) — recorded as a pedagogical on-ramp only (OS-paging analogy, the "50-of-2000 tokens reserved-but-idle" fragmentation framing, prefix/beam-search sharing). Integrated: cross-linked into vllm ("Accessible secondary" section + Related) and noted in synthesis under the benchmark open-question (a no-numbers secondary, not movement on the gap). Entity discovery: no nodes minted — author Amit Shekhar / publisher Outcome School are low graph-signal for an inference-mechanics spoke (no existing entity-index match), recorded inline (same call as prior T3 content-site authors). Ran avoid-ai-writing over the new prose. +1 page (14 -> 15).

[2026-06-18] ingest | DefinedTerm enrichment pass (subagent)

Targeted deepening of the thinnest DefinedTerm pages with fetched primary/secondary sources. Added three source pages and the hard numbers/primaries they back:

  • speculative-decoding-paper (Leviathan et al., ICML 2023; ScholarlyArticle, T1, arxiv:2211.17192) — the peer-reviewed primary behind speculative-decoding, which previously cited only a HF blog. Supplies the speculative-sampling distribution-preservation argument (accept-or-resample, provably the target distribution) and the 2–3× on T5-XXL benchmark. Enriched speculative-decoding: the "unchanged distribution" claim is now a theorem with provenance, acceptance-rate framing of the numbers.
  • flash-attention-2-paper (Dao, 2023; ScholarlyArticle, T1, arxiv:2307.08691) — the FA-2 rung the flash-attention page named without figures. Adds the utilization story: 25–40%→50–73% of A100 peak FLOPs/s, ~2× over FA-1, up to 225 TFLOPs/s. Folded into flash-attention "Versions".
  • continuous-batching-anyscale (Anyscale, 2023; TechArticle, T2, anyscale.com) — bridges the USENIX-only Orca (Yu et al., OSDI 2022) primary to the spoke and decomposes the win: ~4× / ~8× / ~23× (static-opt / continuous alone / continuous + PagedAttention). Enriched continuous-batching with Orca provenance, the harder numbers, the batching×KV-paging-compound reading, and a Related section.

Already-developed llm-inference umbrella: added a Related section + bumped (no padding). Blocked fetch: Orca paper itself — USENIX (usenix.org/system/files/osdi22-yu.pdf and the presentation page) returned HTTP 403 to WebFetch; routed around via the Anyscale T2 secondary that attributes and quantifies it (one alternative, per no-fabrication rule). All paper figures are abstract-level (PDF bodies not extracted); recorded on each source page. Updated index.md (+3 source pages). Ran avoid-ai-writing (edit mode) over new prose. No build/verify, no git. +3 pages (15 → 18).

[2026-06-21] ingest | LLMs-local curated list (via @DanKornas tweet) → edge-regime ecosystem

Source: 0xSojalSec/LLMs-local GitHub awesome-list, surfaced by a Dan Kornas promotional tweet (routed in by the hub from the Telegram channel). A curated map of the local-LLM world: platforms, engines, models, builder tooling, UIs, hardware, tutorials, communities.

Classification: a broad roundup touching three spokes. Dominant in-scope substance = the local inference/serving stack (Ollama, llama.cpp, vLLM, SGLang, MLX vs. LM Studio/Jan/LocalAI) → this spoke. Runner-ups (agent frameworks/MCP/RAG/coding agents → ../agentic-tooling-wiki; model discovery/providers → ../llm-providers-wiki) recorded as cross-spoke context in the source page, not fragmented or re-routed.

What it added:

  • llms-local-list (Collection, T4 — single-maintainer curated link list off a marketing tweet; no benchmarks/measurement). url-only (github repo). The wiki's first map of the on-device regime's ecosystem.
  • local-llm-stack (DefinedTerm) — the platform-vs-engine split the list draws: turnkey platforms (LM Studio/Jan/LocalAI) over engines (Ollama/llama.cpp/vLLM/SGLang/MLX). Names the platform tier the spoke had only implied via llama-cpp ("core of Ollama/LM Studio"). Notes vLLM/SGLang straddle both regimes.
  • Touched llama-cpp (engine-beneath-platforms cross-link) and synthesis (new para under "two serving regimes": edge ecosystem now mapped — a map, not a measurement, so no movement on the standing benchmark question).

Fetch: WebFetch HTTP 402 on x.com → fell back to firecrawl (per HUB edge rule), clean scrape. Author/maintainer (Dan Kornas / 0xSojalSec) noted inline, not as entity nodes — how-does-vllm-work precedent (low graph signal). Ran avoid-ai-writing (detect) over new prose. +2 pages (18 → 20).

[2026-07-10] ingest | LiteRT.js — Google's in-browser AI inference (hub-routed, Telegram)

Ingested the Google Developers Blog announcement of LiteRT.js — a JS binding of LiteRT (ex-TFLite) for running .tflite models in the browser via WASM/XNNPACK (CPU) + WebGPU/ML Drift (GPU) + WebNN (NPU); succeeds TensorFlow.js (~3× claimed, 5–60× GPU/NPU-over-CPU). New litertjs (SoftwareApplication/engine, source, T2) + new concept browser-inference (the in-browser execution regime; WASM/WebGPU/WebNN hardware- access layer). Extends the synthesis "two serving regimes" → a third: datacenter GPU → native-edge → in-browser, a ladder of hardware access (privacy/zero-server-cost objective). Updated local-llm-stack (browser surface beside native engines). Scope note: LiteRT.js is general-ML (vision/audio/embeddings), not LLM-only — enters as inference-execution mechanics on browser hardware (LLMs run on the same stack via MediaPipe); the spoke's first deliberate step past strictly-LLM serving. T2 vendor blog; perf figures unbenchmarked (standing which-lever question). avoid-ai-writing applied (clean). +2 pages, 1 updated.

[2026-07-23] ingest | How to build your own LLM runtime from scratch (Towards Data Science)

Routed from the hub (route in ../log.md). URL-only, T2 — a first-party, measured build-log (real before/after numbers, named bugs), but one engineer's single-model/single-GPU exercise with no independent check. New page: build-llm-runtime-from-scratch (TechArticle). Anubhab Banerjee builds a decode-only inference engine for Qwen2.5-Coder-7B on an H100 in C++/CUDA PTX: INT4 group-wise quantization (.nanoqwen format), paged kv-cache (16 tok/4 KiB page), warp-specialized attention on Hopper TMA, fused GEMV, and CUDA-graph decode capture. Greedy sampling only; no serving/batching/tokenizer — explicitly the inference-engine layer, which is dead-center this spoke. Dedup: no build-from-scratch page existed; every component it implements is an existing page (quantization, kv-cache, flash-attention, paged-attention-paper) — linked, not duplicated. Scoped exactly to llm-inference, not the serving/agent layers, so no cross-spoke contention. Gap-relevance: (1) puts kernel-level concrete reality under the mechanism pages (the paged cache as a real page-boundary sync bug, not a diagram); (2) a hard T2 number on the standing which-lever-bought-what gap. Synthesis: added a section. Its durable point is a fourth instance of the spoke's per-step-fixed-overhead pattern — CUDA graphs (7×, ~119→17 ms/tok) amortize per-kernel launch cost the way continuous-batching amortizes per-request scheduling and speculative-decoding amortizes the per-token forward pass. Recorded the honest coda: the engine is slower than llama-cpp (60 vs 200 tok/s); payoff claimed is ownership, not speed — enters as pedagogy/legibility, not a competitive runtime. Entities: author Anubhab Banerjee not paged (byline, no distinct thesis; evidence-only). No org/event. Index fixed into the TechArticle section (initial insert landed under ScholarlyArticle). Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-03] ingest | Smaller, faster, safer: running Kimi and GLM at scale (Cloudflare)

Routed from the hub (Telegram). T2 — first-party engineering blog from the operator, product to sell (Workers AI), self-reported and unreproducible outside Cloudflare. Rated above the usual vendor post because every table keeps a baseline the vendor's own choice loses to (FP8 cache is slower per token; INT4 prefill is slower), and accuracy is benchmarked rather than asserted. New pages: cloudflare-kimi-glm-serving (TechArticle, source), prefill-decode-disaggregation (DefinedTerm/mechanism), kv-cache-isolation (DefinedTerm/mechanism), sglang (SoftwareApplication). Updated: kv-cache (byte budget, at last: 686k tokens BF16 -> 1.37M FP8 for Kimi K2.6, and the ranking that the cache fills memory before the weights do), quantization (both benchmark tables, plus the correction that quantization is not uniformly a speedup), synthesis, index. Dedup: quantization and kv-cache already existed and were refreshed in place, not duplicated. SGLang had been named twice in passing (local-llm-stack, llms-local-list) and never described — paged now that there is evidence behind it, scoped strictly to what this source attests. Gap-relevance: closes most of the standing which-lever-bought-what question for the two quantization levers (separate experiments, same hardware, five concurrency levels each) and quantifies the memory math question that had been asserted qualitatively since 2026-06-01. Both bullets amended rather than deleted — the algorithmic levers stay bundled and unattributed. Synthesis: new section "The phases separate, and the cache acquires a second cost." Two claims — precision is a property of a phase, not a deployment (hence disaggregated pools); and the packing optimizations produced a correctness surface, since multi-tenant efficiency and multi-tenant isolation are the same bookkeeping seen from two directions. First contradiction flagged in this spoke (previously "None yet"): a sign error, not a disagreement. Earlier sources present lower precision as a memory-and-speed win, true for weights on a bandwidth-bound decode; this source measures the two other cases and both go the other way. Both claims kept, page amended, neither overwritten. Held back deliberately: the security framing. No incident, bug or attack is described — "even a one-in-a-billion mistake would show up regularly" is a statement about request volume, and the wiki records the tag scheme as defence in depth rather than a response to known leakage. The source argues from error rates and not from a threat model, so kv-cache-isolation does not extend it into one. Entities: reused cross-wiki, none duplicated — cloudflare (../cloud-wiki), glm-52 and z-ai (../llm-providers-wiki). Moonshot has no hub entity node; Kimi K2.6 named in prose, not paged. Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-14] ingest | vLLM (re-seen)

https://github.com/vllm-project/vllm arrived via Telegram; already the founding vllm source (created 2026-06-09). Refreshed in place per the re-seen-source rule: description unchanged ("A high-throughput and memory-efficient inference and serving engine for LLMs"), still actively pushed (2026-08-14), stars 89,054 / forks 20,688 — no star figure was cited on the page before, so none needed correcting. No content change; updated bumped.

[2026-08-15] ingest | How Claude's Text Watermarking Works (Anthropic)

https://www.anthropic.com/news/claude-text-watermark arrived via Telegram; routed here by the hub (runner-up ../ai-governance-wiki). Anthropic's 2026-08-14 announcement that future Claude models emit watermarked text. Routed on dominant substance: the mechanism is a change to the sampling step — the sampler's randomness is replaced by a keyed function of the preceding tokens — which is this spoke's subject. The EU AI Act framing that motivates it is recorded as cross-spoke context on the source page rather than fragmented into a second spoke. Tier T3 — first-party vendor announcement, written partly to demonstrate compliance; the only authoritative account of what shipped, and every claim in it is Anthropic's or Google DeepMind's own. New pages: claude-text-watermark (source), text-watermarking (mechanism). token-sampling amended with a "what else rides on this step" section; index + synthesis updated. Dedup: grepped every spoke for watermark/SynthID before writing — the only prior hit is speech-audio-wiki/audio-deepfake (audio modality), cross-linked, not duplicated. Gap-relevance: lands on coverage edge 6 (constrained decoding) from the other side — one mechanism needs the freedom the other removes — and presses the standing sampling × serving question, since this is the first serve-time mechanism that depends on per-request entropy. Synthesis: new section "The sampling step is being asked to carry things that are not speed" — watermarking joins kv-cache-isolation as a non-performance intervention, and the wiki now reads the pipeline as levers plus obligations. New growth edge 9 (the SynthID-Text Nature paper, T1), new cross-spoke adjacency to ../ai-governance-wiki. Recorded as absent, deliberately: no detection statistics at all — no true/false-positive rate, no minimum passage length — and no statement of whether the watermark covers API output or can be turned off. The page says what the source says and names the holes rather than filling them. Held back: the image/file C2PA content credential is signed metadata beside the artifact, not a watermark inside the tokens; noted on the source page as the contrast, not paged as a mechanism here. Entities: none created. Publisher anthropic reused cross-wiki (../research-wiki). Google DeepMind and Scott Aaronson appear as the subject's lineage, not the source's agents, so the "subject is not a discovered entity" rule applies — named in prose. Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-15] ingest | OKF token handoff — skipping the downstream tokenizer (TDS, Banerjee)

https://towardsdatascience.com/how-to-utilize-okf-efficiently-to-enable-knowledge-exchange-among-llms/ arrived via Telegram; routed here by the hub (runner-up ../research-wiki, which owns research-wiki/open-knowledge-format). Anubhab Banerjee again — the author of build-llm-runtime-from-scratch — with the same build-and-measure method. Tier T2: a hand-run benchmark with a stated protocol (median of 7 trials, greedy, 64 new tokens, blocks of 1,948–3,437 tokens), independent, nothing being sold. Routed on dominant substance. The measured contribution is a tokenizer-skip across agents: one 7B agent tokenizes, writes .npy to /dev/shm, and two smaller agents in the same family call generate(input_ids=…) on the array. OKF supplies the markdown envelope and one non-spec frontmatter field (token_pointer); the author states his bundles are not spec-valid. So the format is the wrapper and the tokenizer is the subject. New pages: okf-token-handoff (source), tokenization (mechanism — the spoke's first page on the step before prefill). Gap-relevance: coverage edge 8 (tokenization) partly opens, and deliberately only partly — there is still no source on how a tokenizer works, only on what it costs and where it stops being portable. Edge 7 (prefix caching) gains its upstream sibling: this reuses the integers, prefix caching reuses the cache computed from them. Synthesis: new subsection "Measured in percentages — the denominator does the talking" under the levers-once-measured thread. The reading the article does not print: 28.0% and 37.8% are 19.4 ms and 18.7 ms, the same constant, and against the 41.3 s pipeline it is under a tenth of a percent. The lever is real and the framing outruns it — third instance in that thread, after the hand-built runtime and the Cloudflare sign error. Also recorded: the vocabulary-equality guard (full ~151,936-entry get_vocab() comparison) makes the tokenizer a model-family boundary for any system passing tokens instead of text, and a mismatch fails silently with fluent wrong output rather than an error. Cross-spoke: added a "first sighting outside the spec" section to research-wiki/open-knowledge-format, whose standing caveat says adoption is unproven — one page touch with provenance, not a second ingest. MCP/A2A appear only as the agents' subject matter (a 5G Service-Based Interface), so ../agentic-tooling-wiki gets no claim here. Entities: none created. Author Banerjee still not paged (byline, consistent with the 2026-07-23 decision). qwen reused cross-wiki (../llm-providers-wiki). Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-17] ingest | Understanding Transformers and Attention Mechanisms (Serret, arXiv:2604.00965)

Arrived via Telegram as a PDF link, routed by the hub (runner-ups: ../machine-learning-wiki for the architecture/training half, ../mathematics-wiki for the audience). Downloaded and extracted locally with pypdf — 13 pages, not the 5 the file header reports.

serret-attention-primer, T2: Michel Fabrice Serret (Paul Scherrer Institute), math.NA, 1 Apr 2026, written for an IPAM randomized-numerical-linear-algebra workshop. Expository — complete derivations, no new result and no measurement, unreviewed — so not T1 despite being a preprint by a domain researcher with nothing to sell.

It lands on growth edge 5, which has asked since 2026-08-08 for the attention variants that shrink the cache. Three new pages: attention-mechanism (the spoke's first, after two and a half months of documenting attention's consequences — the KV cache, FlashAttention's IO cost, PagedAttention's memory management — with no page on the operation), grouped-query-attention and latent-attention.

The framing worth keeping is that softmax is a kernel choice, not the definition: attention is a database lookup with the equality relaxed to a similarity κ, and standard attention is κ(v,w)=exp(⟨v,w⟩/√d_QK). Read that way the efficiency variants change what is stored, never the operation. kv-cache gains the arithmetic it had been describing in words — 2·N_L·N_h·N_KV·d floats, O(N³d)→O(N²d) — and the observation that every factor in that formula is a lever somebody pulled, three of which the spoke now holds and the fourth (sliding-window, attacking N_tokens) it does not.

Sharpest single finding, recorded on latent-attention and in synthesis.md: MLA and RoPE do not compose. Latent attention is exactly equivalent to GQA/MHA without positional embeddings; RoPE inserts a position-dependent matrix between the two factors MLA needs to pre-merge, so keeping it in the usual place would add overhead rather than save it. DeepSeek appends a non-latent part to key and query and applies RoPE only there — compute win kept, exact equivalence given up.

Edge 5 marked two-thirds closed, and deliberately not closed: the mechanism is now held, the evidence is not. No cache saving, throughput or quality ablation for either lever exists in this corpus. Still wanted: Ainslie et al. (GQA), the DeepSeek-V2 report (MLA), anything on sliding-window attention, and TransMLA (arXiv:2502.07864) for converting trained GQA/MHA weights to the latent form. index.md updated; author not paged as an entity (one-off, per the deferred-creator convention).

[2026-08-22] ingest | GLM-5 serving parameter tuning (via research pass)

glm5-serving-parameter-tuning — Hua, Wang, Yang, Wang & Lian (China Unicom), arXiv:2607.02518, 27 April 2026. T2: China Unicom built neither sglang nor GLM-5, so the load-bearing claim is a measurement of somebody else's software on hardware they operate — the same reading that puts cloudflare-kimi-glm-serving at T2. Found hunting growth edge 1, whose 14-day cooldown expired today.

Setup, which is why it counts: sixteen H100s across two nodes, SGLang v0.5.9 with flashinfer, GLM-5 FP8, agent traffic at ~28–30k input and ~500 output tokens. Ten configurations. The fixed server flags are listed and exactly three knobs vary — chunked-prefill size, the tp/pp split, and max-running-requests.

Half-close, and the residue is the interesting part. Edge 1 asks for the feature levers separated — paging against batching against quantization against speculative decoding. Every feature is on in all ten rows here. What is separated is configuration, and one of those configuration levers is larger than anything the edge names: tp4/pp4 at 9,993 tok/s against tp8/pp2 at 6,906, a 45% throughput difference on identical hardware and identical software, with TTFT 6.69 s against 9.48 s. Chunk size across its whole range moves ~15%; concurrency 16→24 moves 8%. The parallelism decomposition has no page in this wiki at all, which is now new edge 1a.

Two further readings folded into synthesis.md:

  • The knobs peak in the interior. 3072 beats 2048 and 4096, and 4096 is the worst tested chunk on every axis. Concurrency 24 beats 16, 32 and 48. "Raise the batch, raise the chunk" is wrong in both directions on this workload, and the paper says so.
  • Throughput and tail latency moved together. 16 → 24 concurrency improved tok/s and TTFT and P90 at once. Not the trade the serving literature describes — a configuration below its knee, where both curves still improve. The knee is a property of 28–30k-token requests, not a constant.

Limits recorded on the page: FP8 only, and the authors explicitly refuse to generalize to BF16 / INT4 / W4A8. No resource telemetry, so the pipeline-bubble and collective-overhead explanations are inference from throughput. No archived command lines — irreproducible outside China Unicom, the same defect cloudflare-kimi-glm-serving carries. It is also the wiki's second SGLang deployment and the first to publish a parameter surface for it; sglang updated accordingly.

Dedup: nothing in any spoke held this paper. "OpenClaw" hits a dozen agentic-tooling pages, none about serving.

[2026-09-04] ingest | LLMs-local list re-sent — refresh, no new pages

The curator sent https://github.com/0xSojalSec/LLMs-local again over Telegram. It is a re-seen source: llms-local-list has held it since 2026-06-21. Refreshed the page in place rather than creating anything.

What the GitHub API says today: 1,138★ / 130 forks, not archived, no license, last pushed 2026-06-10 — the same commit the 2026-08-21 staleness check found. The repo has gained stars and forks and no content for 86 days.

No claim on the page changed and no Thing page was touched, so synthesis and index stand as they are. Tier stays T3 (2026-08-25 regrade); the older "T4 unchanged" note now says explicitly that the regrade supersedes it, which it did not before.

[2026-09-10] ingest | Measuring AI Model Performance: Tokens per Second (OpenMetal)

Routed in from the hub (Telegram). WebFetch returned nothing usable — the page is ~2 MB of markup — so it was fetched with curl and stripped to text locally.

Source: openmetal-tokens-per-second, T3 — an authored, dated vendor explainer that ends in GPU-server pricing, a consultation booking and a $50k credit offer. First-party numbers on hardware the publisher sells. published: 2025-03-28 from the page's own metadata (modified 2025-05-20); freshness: 60d, since the hardware and tool claims perish.

New: tokens-per-second — the metric page the spoke never had, though every claim here is denominated in it. Three quantities under one name (prompt / eval / total), the per-request vs aggregate ambiguity, what moves the number, and the requirement side. openmetal as a vendor Organization node (only its own site verified; no Wikipedia or Wikidata claimed).

Updated: prefill-decode-kv-cache (the asymmetry as a measurement hazard), quantization (dynamic vs static — a split the mechanism page did not carry, and INT8 named as the production default), tokenization (~1.3–2 tokens per English word; the denominator is model-specific), index, synthesis (new section, plus a hazard note on growth edge 1).

The claim worth holding: prompt TPS measured up to 10× eval TPS, so a total figure mostly reports the benchmarker's prompt-to-output ratio. The spoke has held that asymmetry as mechanism since 2026-06-01; this is the first source to state it as a way benchmarks go wrong.

The claim held loosely: the CPU-vs-GPU table (Llama 3.2 3B ~52 CPU / ~171 A100; 3.1 8B ~33 / ~122). No batch size, concurrency, quantization level, context length or engine is given. Recorded with that gap stated on the page, twice.

Author not paged. Rafael Ramos is a byline on a vendor blog with no biography, no other work here and nothing verifiable to link; a node would have carried the name and nothing else. Named in the source page instead. Revisit if a second OpenMetal source arrives.

[2026-09-14] ingest | vLLM Semantic Router (vllm-sr.ai)

Routed in from the hub (Telegram). Read the project site, the GitHub README and the arXiv abstract of the founding paper; the /docs/intro page added nothing concrete.

Source: vllm-semantic-router, T1 — first-party project site, Apache-2.0 repo under vllm-project, 5.8k★, v0.3 dated 2026-06-05. No published: — the site carries no date. The paper's numbers (+10.2 MMLU-Pro, −47.1% latency, −48.5% tokens) are recorded with their arXiv id; the site's LiveCodeBench 92.6-vs-92.0 claim is recorded as unmethoded.

New: model-routing — routing as a fifth serving lever, one that runs before prefill. Open questions: classifier cost, misroute cost, whether the cache survives a cascade.

Spine: synthesis section A lever that runs before the engine; adjacency to ../agentic-tooling-wiki (inferoa / xunzhuo — the router's second author) added. Entities deferred: the seven paper authors and the maintainer companies are one-source names.

[2026-09-23] quality | research pass — edge 1, which lever bought what (half close)

Hunted first at the curator's request. batchllm (T1, PDF read): per-lever deltas for chunked prefill and prefix caching on vLLM, on two GPUs, by authors who built neither; request sorting outweighs both. Half close; residue edge 1b. Also read arXiv 2607.05876 ("Floor First"): analytic, measured sweep not yet published — noted in the edge, not paged. Linked from the pre-engine-lever section of synthesis.

[2026-09-23] quality | absence-claim drain — oMLX corrected

omlx's "No independent benchmark" struck: mlx-chronos (T2, new page) has seven oMLX runs. oMLX decodes fastest on a shared M2 test but gets no cached-TTFT gain where mlx-lm and Ollama do; short prompts only, so the SSD-tier claim stays unmeasured.

[2026-09-24] ingest | vLLM launch post — "Easy, Fast, and Cheap LLM Serving with PagedAttention" (vllm.ai)

Arrived from the hub while testing the new Haiku source-classifier agent (routed here, confidence: high, no runner-up). The hub session fetched and ingested it; the run was cut off before the log entries, which the curator asked to keep and complete.

Source: vllm-pagedattention-blog, T3 — the project's own launch post, published: 2023-06-20. Headline: up to 24x throughput over HuggingFace Transformers and up to 3.5x over TGI (LLaMA-7B on A10G, LLaMA-13B on A100, ShareGPT lengths).

Touched: vllm, paged-attention-paper, continuous-batching-anyscale, index.md. Synthesis: a naming trap — three close vLLM multipliers (2-4x vs FasterTransformer/Orca, 24x vs HF Transformers, ~23x vs static batching) measure different baselines and must not be swapped.

[2026-09-28] ingest | Introducing Lev (Dmitri Sotnikov, yogthos.net)

Arrived by Telegram (sender: Lucsw), pre-routed by the hub (runner-up: ../agentic-tooling-wiki). Fetched and read the post directly.

Source: introducing-lev, T3 — the author's own launch post for his own open-source project, self-run benchmarks on sets he built himself, published: 2026-09-24.

New: lev — an open, offline decision engine (Jolt/Clojure) pitched against TypeSafe's hosted Jev: a laya encoder answers most typed questions in ~95 ms, and only the ones it scores below a per-question-type confidence threshold escalate to a local GGUF LLM (default Qwen3.5-4B) scored by log-probability, not generation. On the post's own 144-case adversarial set, gating at 0.5 raised accuracy 61.1%→92.4% at roughly double the per-case latency. Also new: a temperature-scaling calibration refit (0.466→0.081 mean error on laya's published numbers), an option-order debias mode, a cross-question constraints decoder, and static jolt binary packaging with llama.cpp built in. dmitri-sotnikov — the author (yogthos), also known for the Luminus Clojure framework; same-as verified against his GitHub (github.com/yogthos) and blog.

Entity reuse, no duplicates: typesafe-ai / system-one-models-and-jev (../llm-providers-wiki), laya / open-jev-9b / typed-decision (../machine-learning-wiki), Jolt (../programming-languages-wiki), Qwen (../llm-providers-wiki) — all linked, none re-paged. Deferred (recurrence rule): GitHub user thecodacus (named-once fork author, no biography) and benchmarkheaven.com (a linked site, not a bounded entity). Entity budget: 2 of 8 used.

Touched: model-routing (new section — confidence-gated escalation after a cheap pass, as opposed to routing before one; answers part of its open question on classifier cost), llama-cpp (new "statically linked into other tools' binaries" note), index.md (new Person section — the spoke's first).

Spine: synthesis — new section Confidence, not just cost, decides which forward pass runs; cross-spoke adjacency bullet added for ../llm-providers-wiki and ../machine-learning-wiki (the rest of the Jev/System-One ecosystem this post responds to). Runner-up ../agentic-tooling-wiki recorded on introducing-lev and in this entry, per the hub's routing note — not written into that spoke.

[2026-10-07] ingest | Retrofitting language models to operate over bytes (Minixhofer et al., Nature)

Source: byteification, T1 (source), via Telegram PDF (pypdf extraction, poppler unavailable for page rendering). Retrofits an existing subword LLM into a byte-level one ("byteification") at under 1% of a pretraining budget — Bolmo/Bwen/Blama from Olmo 3/OLMo 2/Qwen3/Llama 3. Matches subword baselines generally (non-significant gaps in several cases), clearly beats them on character-level tasks, mixed on code (pass@1 down, pass@16 up), vastly outperforms prior from-scratch byte-level models. Inference-speed claim stated qualitatively; no number survived text extraction (likely figure-only). Answers coverage edge 8's underlying question a different way than asked — not how a tokenizer is built, but how to avoid having one. New entity page allen-institute-for-ai (primary institutional affiliation, 5 of 9 authors, publisher of the OLMo family the paper retrofits). tokenization and synthesis updated.

Related pages

Links to