Hugging Face — Quantization concepts (Transformers docs)

The Transformers library’s reference explainer for what quantization is and how the schemes differ — the stable definitional source behind this wiki’s quantization page. Vendor docs, but T1 for a mechanism definition: it’s the maintainer of the de-facto open-weight loading stack (bitsandbytes, GPTQ, AWQ, compressed-tensors all sit under its HfQuantizer API).

Facts it pins down

• The size math. Quantization represents weights/activations in a lower-precision type instead of 32-bit float (FP32). An int8 model is ~4× smaller than its FP32 counterpart; int4 halves it again vs int8 — so the bytes-per-parameter is the footprint lever.
• Affine quantization maps the float range [val_min, val_max] to the integer range (int8 is [-128, 127]) via a scale S and zero-point Z; symmetric drops the zero-point (FP 0.0 → int 0, one parameter), asymmetric keeps both.
• int4 weight packing — most hardware can’t address 4-bit natively, so two int4 values are packed into one byte. The win is memory bandwidth: loading packed int4 is ~2× faster than int8, which is why int4 speeds up memory-bound inference even without native int4 compute. int4 costs more accuracy, so GPTQ / AWQ are usually needed for good int4 quality.
• FP8 (the A8W8 scheme — 8-bit activations and weights) keeps floating-point structure in two variants: E4M3 (4 exponent / 3 mantissa bits — more precision, less range) and E5M2 (5/2 — wider range, less precision). Efficient FP8 needs newer GPUs (H100/H200/B100, MI300).
• Granularity — per-tensor (one S,Z for the whole tensor, simpler) vs per-channel/group (separate per channel, more accurate).
• PTQ vs QAT — PTQ quantizes after training (simple); QAT inserts “fake quantization” ops during training so the model adapts, usually better accuracy, especially at low bit-widths.

Why it matters here

Grounds the quantization market-lever claim in the actual mechanism: footprint reduction is a bytes-per-parameter story, int4’s payoff is memory bandwidth not just storage, and QAT’s low-bit-width edge is exactly what google‘s gemma-4-qat leans on.

Related

quantization · open-weight-models · gemma-4-qat · llm-inference