Quantization (for inference)

Which lens this is. The mechanism: how weights and activations are reduced in precision and what that does to memory, throughput and quality at serve time. The market-lever treatment — quantization as what makes a model cheap enough to offer at a given price — is llm-providers-wiki/quantization. Deliberate dual-lens (HUB.md); this page is the canonical target for a bare [[quantization]].

The explicitly-named gap the synthesis flagged as “still absent”: storing a model’s weights (and sometimes activations) in lower precision — int8, int4, even 1–2 bit — instead of fp32/fp16/bf16, to shrink memory and speed inference while preserving as much accuracy as possible. It is the fourth production lever alongside PagedAttention, flash-attention, and speculative-decoding — but it acts on the data type rather than the algorithm.

Why it matters for the inference pipeline

Memory footprint. Weights at int4 are ~¼ the size of fp16 — the difference between a model fitting on one GPU (or a laptop) or not.
The kv-cache connection. Quantization also applies to the KV cache itself; since cache size caps batch size and context length (continuous-batching), a smaller-precision cache directly relaxes the serving bottleneck the wiki’s thesis centers on.
Bandwidth/throughput. Decode is memory-bandwidth-bound (moving weights + cache per token); fewer bytes per parameter means more tokens/sec, especially on consumer hardware.

How it’s done

Weight-only (most common for inference): quantize weights, compute in higher precision — big memory win, minimal quality loss.
Weight + activation (e.g. FP8, int8): quantize both for more speed, harder to keep accurate.
Post-training methods (no retraining): GPTQ (2/3/4/8-bit, calibration-based), AWQ (activation-aware, 4-bit), bitsandbytes (on-the-fly 4/8-bit), GGUF/GGML (llama-cpp‘s 2–8-bit format), plus FP8 paths (FBGEMM, torchao) and extreme 1–2-bit schemes (AQLM, VPTQ) that need calibration.

The tradeoff

Lower precision = smaller/faster but lossier: aggressive (1–2 bit) quantization needs careful calibration to avoid accuracy collapse, while 8-bit and weight-only 4-bit often run “out of the box” with little degradation. This is the knob that makes the founding pipeline’s economics (prefill/decode, KV-cache memory) tractable on real hardware — and it is exactly what llama-cpp exploits to run frontier-class models on a laptop.

Measured, at last (added 2026-08-03)

Everything above was mechanism without magnitudes. cloudflare-kimi-glm-serving runs the two kinds of quantization separately on the same H200 hardware and reports both, which is the first lever-by-lever attribution in this spoke.

Cache precision (BF16 → FP8 e4m3), Kimi K2.6. Capacity roughly doubles, 686k → 1.37M tokens. Per-token decode speed goes down at every concurrency both can run (137 → 125 tok/s at one request, about 9%), and total throughput goes up anyway because BF16 runs out of memory at 64 concurrent requests while FP8 reaches 2,192 tok/s — 41% above BF16’s peak, at roughly 30% less cost per token. Accuracy across GSM8K, ARC, MMLU and MMLU-Pro moves by −1.0 to +0.77 points.

Weight precision (FP8 → INT4), GLM 5.2. Checkpoint 705 GB → 421 GB, per-GPU 88 GB → 52 GB. Decode gets faster at every concurrency (60 → 92 tok/s at one request; 1,672 → 1,933 at 64), because decode is bandwidth-bound and fewer bytes per weight means fewer bytes streamed per token. Accuracy stays within 0.8 points on every benchmark.

Two corrections to the picture this page held before:

Quantization is not uniformly a speedup. Cache quantization costs per-token speed and pays in capacity; weight quantization pays in speed but costs prefill, which must decompress before it multiplies (10,160 → 8,660 tokens/sec). Which direction it moves you depends on which phase you are in.
So precision is a per-phase decision, not a per-deployment one — the reasoning behind prefill-decode-disaggregation.

The “little degradation at 8-bit and weight-only 4-bit” claim above now has evidence behind it at production scale, from a vendor measuring its own deployment (T2).

Boundary (cross-wiki) — deliberate dual-lens, not a duplicate

This page is the inference-mechanics lens: data-type lever, KV-cache interaction, methods (GPTQ/AWQ/bitsandbytes/GGUF), bandwidth/throughput. The market / deployability lens — quantization as a competitive footprint axis (who can run what where; QAT vs PTQ; Gemma 4’s sub-1GB open models) — is paged separately as quantization in llm-providers-wiki, which defers the mechanics here. Same technique, two spoke-specific lenses by design (the hub pages concepts per-spoke; only entities are canonicalized to one node) — cross-referenced, not merged.

kv-cache · llama-cpp · vllm · sglang · continuous-batching · prefill-decode-disaggregation · cloudflare-kimi-glm-serving · flash-attention · llm-inference