Quantization

Which lens this is. The market lever: quantization as the thing that moves a model into a cheaper tier or onto smaller hardware, and how providers use it in what they sell. The mechanism — precision formats, memory and throughput at serve time — is llm-inference-wiki/quantization, which is the canonical target for a bare [[quantization]]. Deliberate dual-lens (HUB.md).

Storing and running a model’s weights (and sometimes activations) at lower numeric precision — int8, int4, even 2-bit — instead of 16/32-bit floats, to shrink its memory footprint and speed inference. In this wiki it matters as a market lever: quantization is a primary reason capable open-weight-models now run on consumer and edge hardware, making footprint a competitive axis alongside capability, cost, and context (synthesis).

The size math

Footprint is a bytes-per-parameter story: an int8 model is ~4× smaller than FP32, and int4 halves it again vs int8 hf-quantization-concepts. The mapping itself is affine — the float range maps to the integer range (int8 = [-128, 127]) via a scale and zero-point; the symmetric variant drops the zero-point (FP 0.0 → int 0). Granularity trades simplicity for accuracy: per-tensor (one scale for the whole tensor) vs per-channel/group (separate per channel) hf-quantization-concepts.

Why int4 is the workhorse — and the FP8 alternative

int4’s payoff isn’t only storage but memory bandwidth: hardware can’t address 4-bit natively, so two int4 values are packed into one byte, and loading packed int4 is ~2× faster than int8 — which is why it speeds up memory-bound inference even without native int4 compute hf-quantization-concepts. int4 drops more accuracy than int8, so it usually leans on advanced PTQ schemes (GPTQ / AWQ). The newer alternative is FP8 (the A8W8 scheme — 8-bit activations and weights), which keeps floating-point structure in two variants, E4M3 (more precision) and E5M2 (wider range), but needs recent GPUs (H100/H200, MI300) to run efficiently hf-quantization-concepts.

QAT vs PTQ — and a third case the dichotomy misses

PTQ (post-training quantization): quantize an already-trained model, after training. Simple, but loses quality as bit-width drops hf-quantization-concepts.
QAT (quantization-aware training): insert “fake quantization” ops during training so the model learns to tolerate low precision — usually better accuracy, especially at low bit-widths hf-quantization-concepts. google‘s Gemma 4 QAT release gemma-4-qat is the wiki’s worked example: it claims QAT yields higher quality than PTQ baselines at the same bit-width.
Native-format training (added 2026-08-02): there is no higher-precision model at any point. neutrino-1 claims its ternary weights are what the model learned in — “no full-precision product model that was rounded afterward” — with rounding a trained model to the same depth said to land near chance. Both PTQ and QAT start high and end low; this starts and ends in the same place, so it is not compression at all. See ternary-weights. The claim is first-party and the method unpublished.

Levels seen in the wild

Q4_0 / int4 — the standard ~4-bit workhorse (gemma-4-qat).
Targeted 2-bit — Gemma 4’s mobile scheme quantizes token-generation layers to 2-bit while keeping reasoning layers higher-precision (mixed precision), plus channel-wise quantization and static activation scaling gemma-4-qat.
2-bit after training — the PTQ floor; AQLM’s 2-bit Llama-3-8B lands at 4.08 GB and MMLU 58.72 zero-shot, the weakest row in neutrino-1‘s comparison and a fair illustration of what PTQ costs at depth.
Ternary (~1.58 bits) — the current extreme, and a different animal: see ternary-weights.
Footprint payoff: a gemma-4 E2B text model under 1 GB — sub-gigabyte LLMs; an 8B-class model in 3.88 GB on disk neutrino-1.

Boundary (cross-wiki)

This page covers quantization as a deployability / market lever (who can run what, where). The deeper inference mechanics of low-precision execution — kernels, dequant, throughput — belong to llm-inference-wiki (llm-inference), alongside the kv-cache / MoE efficiency story. Distribution formats (GGUF, compressed tensors) ride along with it.

hf-quantization-concepts · ternary-weights · gemma-4-qat · gemma-4 · neutrino-1 · open-weight-models · google · llm-inference