Quantization: theory and formats

The last two chapters kept arriving at the same verdict: on 16GB, weight bytes are the tyrant. They set the decode ceiling (bandwidth over bytes-per-token) and they crowd out the KV cache. Quantization is the one lever that attacks both at once. Store weights in 4 bits instead of 16 and you read a quarter of the bytes per token (faster decode) and you free three quarters of the weight footprint (more KV, more concurrency). The catch is that you are throwing away information, and the entire craft is throwing it away where the model will not miss it. This chapter builds the error theory from scratch, then explains AWQ, GPTQ, FP8, and MXFP4 both conceptually and mathematically, and finishes by measuring the perplexity cost of two schemes on a real model.

Theory

Uniform quantization and its error

The simplest quantizer maps a real value $x$ in some range onto one of $2^n$ evenly spaced levels. Pick a scale $s$ (the spacing between levels) and, optionally, a zero-point offset $z$. Symmetric uniform quantization (no offset) is:

$$q=\text{round}\left(\frac{x}{s}\right),\hat{x}=s\cdot q$$

with $q$ clamped to the representable integer range $[-2^{n-1},2^{n-1}-1]$. The reconstructed $\hat{x}$ differs from $x$ by the rounding error $e=\hat{x}-x$. Because rounding sends $x$ to the nearest multiple of $s$, the error lies in $[-s/2,+s/2]$.

That last clause is the whole game. The error is $R^2/(12\cdot 2^{2n})$. You cannot afford more bits $n$, so you must shrink the range $R$ over which any single scale applies. That is what group-wise scaling does.

It is worth pausing on the numbers to feel how brutal the 16-bit to 4-bit drop is before group-wise scaling rescues it. Going from $n=16$ to $n=4$ is a factor of $2^{2\times 12}=2^{24}\approx 1.7\times 10^7$ increase in error variance for a fixed range $R$. If you applied one scale across a whole weight matrix whose values span, say, a range of 2.0, the 4-bit step would be $2.0/16=0.125$, coarse enough to visibly wreck the model. Now shrink the range to a 128-weight group where the local span might be 0.2, and the step drops to $0.2/16=0.0125$, a tenfold improvement from range reduction alone, at the same 4 bits. That is the entire reason 4-bit inference is viable: not because 4 bits is inherently enough, but because a well-chosen local range makes 4 bits behave like many more. The refinements below (AWQ, GPTQ, MXFP4) are all further attacks on the same quantity, either shrinking the effective range for the weights that matter or repairing the error the fixed grid leaves behind.

Group-wise scales

A per-tensor scale forces one $s$ across millions of weights whose magnitudes vary wildly; a single outlier inflates $R$ and blows up the step for everyone. Group-wise quantization partitions each weight matrix into small contiguous groups (typically 64 or 128 weights) and gives each group its own scale $s_g$ (and optionally zero-point). A weight $w_i$ in group $g$ becomes:

$${\hat{w}}_i=s_g\cdot \text{round}\left(\frac{w_i}{s_g}\right),s_g=\frac{{\max }_{j\in g}|w_j|}{2^{n-1}-1}$$

Now $R$ is the dynamic range within a 128-weight neighborhood, not the whole tensor, so the effective step is far smaller and the error much lower for the same bit width. The cost is storage: each group carries a scale (usually FP16). For a group size of 128 that adds $16/128=0.125$ bits per weight, and an asymmetric zero-point adds a similar increment, so "4-bit" AWQ or GPTQ actually costs about 4.13 to 4.25 effective bits per parameter (the symmetric, scale-only floor is $4+0.125=4.125$). Footprint budgeting in the KV chapter rounds this up to a conservative ~4.25 to 4.5 bits (0.56 byte/param) to leave margin for packing and alignment overhead.

Where the error actually hurts: outliers and activations

Here is the fact that separates naive quantization from the good schemes. Transformer weight matrices are not the problem; their activations are. A handful of hidden dimensions carry systematically large activation magnitudes (the "outlier features"), and the product $Wx$ is what feeds the next layer. Quantizing $W$ uniformly treats every weight column as equally important, but a weight column that multiplies a large-magnitude activation channel contributes far more to the output error than one that multiplies a near-zero channel. Uniform quantization is blind to this; the smart schemes are not.

AWQ: activation-aware weight quantization

AWQ's insight is that not all weights matter equally, and the ones that matter are identifiable from activation statistics. If a weight column is multiplied by a large-magnitude input channel, protecting that column's precision protects the output. AWQ protects it not by giving it more bits (the format is fixed at 4) but by scaling it up before quantizing and scaling the activation down to compensate, which pushes the important weights into a range where the fixed step size represents them more faithfully.

GPTQ: second-order weight quantization

GPTQ comes at the same goal from a different mathematical direction. Instead of rescaling channels, it quantizes weights one column at a time and compensates the remaining, not-yet-quantized weights for the error just introduced, using second-order (curvature) information about the loss.

Conceptually: AWQ reshapes the weights before rounding so the fixed grid fits the important ones; GPTQ rounds greedily and repairs the damage using curvature. AWQ is cheaper, more robust to calibration-set choice, and tends to hold up better at low bit widths for the models in this repertoire; GPTQ can be marginally more accurate when its Hessian estimate is good but is fussier. Both land near 4.13 to 4.25 effective bits and both are served by vLLM through Marlin kernels.

FP8: floating point, not integer

Integer quantization spends its bits on a uniform grid. FP8 spends them on a floating-point grid: a sign, a few exponent bits, a few mantissa bits. The e4m3 format (1 sign, 4 exponent, 3 mantissa) gives non-uniform spacing, fine near zero and coarse for large magnitudes, which matches how neural network values are distributed (many small, few large). FP8 does not need group scales to handle dynamic range because the exponent handles range for free; it is the natural format for activations and the KV cache, and on Blackwell it has native tensor-core support. Weights at FP8 (8 bits) are less aggressive than 4-bit integer schemes, so on this card FP8 is where I reach for KV cache and activation precision, while INT4-group (AWQ) is where I reach for weight compression.

MXFP4: microscaling 4-bit, and gpt-oss

MXFP4 is the format gpt-oss-20b ships in, and it is a clever hybrid of the two ideas above. It is a 4-bit floating-point element format (e2m1: 1 sign, 2 exponent, 1 mantissa) combined with a shared micro-exponent scale per block of 32 elements. So each element is a tiny 4-bit float, and every group of 32 shares an 8-bit exponent scale. This is group-wise scaling (small block, shared scale) married to a floating-point element (non-uniform grid), and the block of 32 is small enough that the shared scale keeps effective range tight. The upshot is roughly 4.25 bits per parameter with better outlier tolerance than INT4 at the same budget, which is why a 21B-parameter MoE fits and serves comfortably on 16GB. The relevant point for operations is that MXFP4 is a native checkpoint format for gpt-oss (I do not quantize it myself; it arrives that way) and vLLM serves it directly on Blackwell.

What degrades, and where

Quantization error is not spread evenly across capabilities. Broad, redundant knowledge (common facts, fluent syntax) is robust; the model has many pathways to it. Narrow, precise operations degrade first: multi-step arithmetic, long-chain reasoning where a single wrong token derails the trace, exact format-following, and rare-token recall. For reasoning models this is exactly the sensitive spot, so a quantization that looks fine on perplexity can still lose points on a math eval. The rule I hold to for the thesis: perplexity is a smoke test, task accuracy is the verdict. Perplexity catches gross breakage cheaply; the eval suite in Part III catches the reasoning-specific damage that perplexity misses.

The four formats, side by side

It helps to hold the repertoire's formats in one frame. Each is a different answer to "spend the bits on what?"

The pattern reading down the table: integer formats spend a uniform grid and lean on group scales to survive; floating formats spend an exponent to handle range for free, which is why FP8 needs no group scales and why MXFP4 tolerates outliers better than INT4 at the same bit count. There is no universally best format; there is a best format for a given tensor's role and a given hardware's native kernels. On Blackwell all four have fast paths, which is what makes the whole repertoire practical on one card.

Tooling

Two toolchains cover the repertoire. For AWQ and GPTQ, llm-compressor (the maintained successor to AutoAWQ/AutoGPTQ, from the vLLM/Neural Magic side) produces vLLM-loadable checkpoints and runs the calibration forward passes. For measuring the damage, I compute perplexity on a held-out text with a short transformers loop, and (in Part III) run the real task suite. Calibration data matters: use a few hundred sequences of text that resembles the serving distribution (for a reasoning model, include chain-of-thought-style text), because AWQ's activation means and GPTQ's Hessian are only as representative as the calibration set. A tiny or off-distribution calibration set is a common cause of avoidable degradation.

Lab: quantize a small model two ways, measure the perplexity delta

The card cannot quantize a 14B comfortably for a quick lab, so I use a small model to make the mechanism visible and fast; the same commands scale to the repertoire. The artifact is ppl_report.json comparing baseline, AWQ, and GPTQ perplexity.

Set up

uv init quant-lab && cd quant-lab
uv add "llmcompressor>=0.3" "vllm>=0.6" "transformers>=4.45" datasets torch

Quantize both ways

"""Quantize a small model with AWQ and with GPTQ using the same calibration set."""
import argparse
from datasets import load_dataset
from transformers import AutoTokenizer
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.awq import AWQModifier

def run(model_id, scheme, out_dir):
    tok = AutoTokenizer.from_pretrained(model_id)
    ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
    ds = ds.filter(lambda r: len(r["text"]) > 200).select(range(256))
    # NOTE: modifier kwargs drift across llmcompressor releases; scheme="W4A16"
    # already implies 4-bit weights at group size 128, so bits/group_size are not
    # passed separately. Confirm the signature against your installed version, e.g.
    #   uv run python -c "from llmcompressor.modifiers.awq import AWQModifier; help(AWQModifier)"
    if scheme == "awq":
        recipe = AWQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
    else:
        recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
    oneshot(model=model_id, dataset=ds, recipe=recipe,
            output_dir=out_dir, max_seq_length=512, num_calibration_samples=256)
    print(f"wrote {out_dir}")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", default="Qwen/Qwen2.5-0.5B-Instruct")
    args = ap.parse_args()
    run(args.model, "awq",  "out-awq")
    run(args.model, "gptq", "out-gptq")

uv run python quantize.py --model Qwen/Qwen2.5-0.5B-Instruct

Measure perplexity for each

"""Sliding-window perplexity on wikitext-2 test, identical for every model."""
import argparse, json, math, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

@torch.no_grad()
def perplexity(model_path, stride=512, max_len=2048):
    tok = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(
        model_path, torch_dtype="auto", device_map="cuda").eval()
    text = "\n\n".join(load_dataset("wikitext", "wikitext-2-raw-v1",
                                    split="test")["text"])
    enc = tok(text, return_tensors="pt")
    ids = enc.input_ids.to("cuda")
    nll, count = 0.0, 0
    for begin in range(0, ids.size(1) - 1, stride):
        end = min(begin + max_len, ids.size(1))
        window = ids[:, begin:end]
        target = window.clone()
        target[:, :-stride] = -100  # only score the fresh stride
        out = model(window, labels=target)
        n_tok = (target != -100).sum().item()
        nll += out.loss.item() * n_tok
        count += n_tok
        if end == ids.size(1):
            break
    return math.exp(nll / count)

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--paths", nargs="+", required=True)
    args = ap.parse_args()
    report = {p: round(perplexity(p), 4) for p in args.paths}
    base = report[args.paths[0]]
    for p, v in report.items():
        report[p] = {"ppl": v, "delta_vs_base": round(v - base, 4)}
    with open("ppl_report.json", "w") as f:
        json.dump(report, f, indent=2)
    print(json.dumps(report, indent=2))

uv run python perplexity.py --paths Qwen/Qwen2.5-0.5B-Instruct out-awq out-gptq

What you should see

ppl_report.json lists three perplexities: the BF16 baseline first, then AWQ and GPTQ with their deltas. Both quantized deltas should be small and positive (perplexity rises a little when you throw bits away) and, for a well-behaved model with a decent calibration set, on the order of a few hundredths to a few tenths of a perplexity point. AWQ and GPTQ should land close to each other; which one wins by a hair depends on the calibration set and the model, and it is not worth over-reading a single run. What you must not see is a large jump (perplexity doubling, or NaN): that signals a broken calibration set, a wrong group size, or a layer that should have been left in higher precision. If the deltas look reassuringly tiny, resist the conclusion that quantization is free. Perplexity is the smoke test; the reasoning-specific damage shows up only when the Part III eval suite runs the quantized model on math and multi-step tasks, and that is where I make the real accept/reject call. Record the three perplexities, the calibration set size, the model revision, date, and driver next to the file.