Quantization

It is August 2022. A 175-billion-parameter language model stored at 16-bit precision needs roughly 350 gigabytes just to hold its weights — more memory than any single GPU on the market offers, before a single activation or KV-cache entry is counted. A team at Meta AI and the University of Washington publishes a way to run that same model’s matrix multiplications in 8-bit integers instead, with no retraining and no measured loss of accuracy. Over the following ten months, three more papers push the same idea further — to 4 bits, to activations as well as weights, and to a single afternoon of GPU time — turning “how many bits does a trained weight actually need” into one of the decade’s more consequential systems questions.

Where We Were

Every technique this book has covered so far for shrinking a model’s numeric footprint operated during training. Mixed-precision recipes ran the forward and backward pass in FP16 or BF16 while keeping an FP32 master copy of the weights for the update step (Chapter 7); FP8 pushed the same idea to 8 bits, but still inside the training loop, with gradients and optimizer state actively managed alongside the forward pass. None of that machinery addresses a different, narrower problem: a model has already finished training, its weights are frozen, and the only question left is how cheaply it can be made to answer a prompt.

That question became urgent for a specific reason — scale. GPT-3, at 175 billion parameters, established in May 2020 that language models this large were viable at all (Chapter 7, language-models), but GPT-3 itself was never released as downloadable weights; it lived behind an API. When OPT, Meta’s 175-billion-parameter open replica, and BLOOM, the BigScience collaboration’s 176-billion-parameter multilingual model, released their weights in 2022, researchers outside a handful of labs could, for the first time, hold a GPT-3-class checkpoint on their own disks (Chapter 11). Holding it and running it were different problems. At FP16, OPT-175B’s weights alone need about 350 GB — more than eight 40 GB A100s’ worth of memory before any activation memory, KV cache, or batch of requests is added. The gap between “the weights exist on my machine” and “I can serve this model” was the gap quantization set out to close.

The obvious fix — round every weight and activation to 8-bit integers, the way convolutional vision models had been quantized for years — did not transfer cleanly to transformer language models at scale. Naive int8 quantization multiplies every value by a fixed scale, rounds to the nearest of 256 integer levels, and hopes the rounding error stays small relative to the signal. That assumption breaks when a small number of values are enormously larger than the rest: a fixed scale calibrated to cover the largest value wastes almost all its 256 levels on a handful of outliers, crushing everything else into a handful of bins. Vision models rarely produce values like that. Transformer language models, it turned out, systematically do — once they cross a specific size.

One year turns quantization from a memory hack into four distinct disciplines, each attacking a different half of the same matrix multiplication. PNG ↓

The Breakthrough

Dettmers et al. traced the failure of naive int8 to a phenomenon they called emergent outlier features (Dettmers et al., 2022). Below roughly 6.7 billion parameters, different layers of a transformer disagreed about which hidden dimensions should carry unusually large activation values. Past that threshold, the disagreement resolved: a small, consistent set of hidden dimensions — as few as six or seven, out of thousands — began carrying activation magnitudes many times larger than the rest, coordinated across nearly every layer of the network. These outlier dimensions were not noise to be discarded. In their own ablation, removing the outlier features cut the model’s mean top-1 softmax probability roughly in half and raised validation perplexity by 600 to 1000%, despite touching at most seven of thousands of hidden dimensions. Any quantization scheme that treated those dimensions the same as everything else would either blow its dynamic range protecting them or destroy the model’s predictions ignoring them.

LLM.int8() resolved this with a two-part scheme rather than a single global rule. Most of the matrix multiplication used vector-wise quantization: instead of one scale for an entire weight or activation matrix, each row and column got its own scale, computed from that row or column’s own maximum magnitude, which kept the rounding error local rather than dominated by the single largest value anywhere in the tensor. The outlier feature dimensions — the small set identified as systematically large — were carved out and multiplied in 16-bit precision instead, in a separate mixed-precision decomposition step, while more than 99.9% of the multiplication’s values still ran in 8-bit. The result, reported across models up to OPT-175B and BLOOM-176B, was int8 inference matching 16-bit accuracy with no retraining, at roughly half the memory footprint — the first quantization method to make GPT-3-scale open weights loadable on a single high-memory server built from consumer GPUs rather than a multi-node A100 cluster.

Two months later, Frantar et al. attacked a related but distinct problem: not activations, but weights, and not 8 bits, but 3 or 4 (Frantar et al., 2022). GPTQ framed weight quantization as a per-layer reconstruction problem — find quantized weights that reproduce the original layer’s output on a small amount of calibration data as closely as possible — and solved it column by column using approximate second-order (Hessian) information, correcting the remaining unquantized weights after each column to compensate for the error just introduced. Run without any retraining, GPTQ quantized a 175-billion-parameter model to 3–4 bits per weight in roughly four GPU-hours, with what the paper described as negligible accuracy degradation, and reported end-to-end inference speedups of about 3.25x on an A100 and 4.5x on the more cost-effective A6000 relative to FP16. For the first time, a 175-billion-parameter model’s weights fit inside a single GPU for generative inference.

SmoothQuant, published the following month, went back to the activation problem LLM.int8() had solved with mixed precision, and asked whether the outliers could be defused mathematically instead of carved out at runtime (Xiao et al., 2022). Its authors observed that weights are comfortably easy to quantize on their own — smoothly distributed, no outliers — while activations are the hard part. SmoothQuant’s fix was an offline, mathematically equivalent transformation: divide each activation channel by a per-channel smoothing factor before quantizing it, and multiply the corresponding weight channel by the same factor, so the product the network actually computes is unchanged, but the difficulty has moved from the activation (which is now smoother) to the weight (which can absorb it, having had none of its own). Because both operands end up quantizable, SmoothQuant enabled full INT8 matrix multiplication — 8-bit weights and 8-bit activations, W8A8 — across the entire forward pass, something neither LLM.int8()‘s mixed-precision carve-out nor a weight-only method like GPTQ delivers. Tested across OPT, BLOOM, and several other model families, SmoothQuant reported up to 1.56x inference speedup and 2x memory reduction with negligible accuracy loss, and enabled serving a 530-billion-parameter model on a single compute node.

AWQ arrived seven months later with a different answer to the same outlier problem LLM.int8() had identified, aimed specifically at 4-bit weight-only quantization for on-device deployment (Lin et al., 2023). Its authors found that protecting only about 1% of a weight matrix’s channels — the ones that matter most — could substantially cut quantization error, and that the right way to identify which 1% mattered was to look at the activation distribution, not the weights themselves: a weight channel that gets multiplied by consistently large activations is more important to preserve precisely than one that doesn’t, regardless of the weight values’ own magnitude. Rather than LLM.int8()‘s solution of splitting those channels out into a separate 16-bit multiplication — hardware-inefficient, because mixed-precision kernels are slower than uniform ones — AWQ scaled the salient channels up before quantizing them, and folded the inverse scale into the corresponding activations, an equivalent transformation similar in spirit to SmoothQuant’s but computed to protect specific channels a calibration pass identifies as salient rather than to smooth the whole distribution. Because it needs no backpropagation or reconstruction against calibration data, AWQ generalized well to instruction-tuned and multimodal models the paper’s authors described as not overfitting the calibration set. Alongside the quantization method, the authors shipped TinyChat, a kernel-fused inference engine reporting more than 3x speedup over Hugging Face’s FP16 implementation and enabling a 70-billion-parameter Llama 2 model to run on a mobile GPU.

LLM.int8()'s outlier discovery is the fork every later method answers differently: carve outliers out, quantize weights only, or smooth them away entirely. PNG ↓

The Mathematics

Every method in this chapter starts from the same basic operation: mapping a continuous, high-precision value into one of a small number of integer levels. For a tensor $X$ quantized symmetrically to $b$ -bit integers:

X_q = \left\lfloor \frac{2^{b-1}-1}{\max(|X|)} \, X \right\rceil, \qquad \hat{X} = \frac{\max(|X|)}{2^{b-1}-1} \, X_q

$X$ is the original tensor of real-valued numbers — a row of activations or a slice of weights. $\max(|X|)$ is the largest absolute value in that tensor, which sets the scale: dividing by it and multiplying by the number of positive integer levels available, $2^{b-1}-1$ , maps the tensor’s full range onto the integer grid before rounding to the nearest integer, $X_q$ . $\hat{X}$ is the dequantized approximation recovered by reversing the same scale — the number actually used once quantized values are converted back for arithmetic. What this buys you: one number, $\max(|X|)$ , and one integer format, $b$ , together determine every rounding error in the tensor. Every method below is a different strategy for choosing what that scale is computed over — a whole tensor, a row, a channel — and for handling the small number of values that make a single global scale a bad choice.

Symbol	Meaning
$X$	the original tensor of real-valued weights or activations
$b$	the target bit-width of the quantized integer format
$X_q$	the rounded integer tensor after quantization
$\hat{X}$	the dequantized approximation of $X$ recovered from $X_q$

LLM.int8()‘s vector-wise scheme applies exactly that formula, but computes a separate scale per row and per column of the matrix multiplication rather than one scale for the whole matrix, and removes the outlier feature dimensions from the integer path entirely before the quantized multiplication happens, computing them instead in 16-bit precision and adding the result back in:

C \approx S_{\text{out}} \odot \big(X_{q}\,W_{q}\big) \;+\; X_{16}\,W_{16}

Symbol	Meaning
$C$	the layer’s output
$S_{\text{out}}$	combined row/column dequantization scale for the int8 product
$X_q, W_q$	row- and column-quantized int8 activation and weight matrices
$X_{16}, W_{16}$	the outlier-dimension slices multiplied directly in 16-bit

GPTQ frames quantizing a single layer’s weight matrix $W$ as a reconstruction problem against a small batch of calibration inputs $X$ , minimizing the squared difference between the original and quantized layer’s output:

\hat{W} = \arg\min_{W'} \; \lVert W X - W' X \rVert_2^2

Symbol	Meaning
$W$	the layer’s original, full-precision weight matrix
$X$	calibration activations gathered by running data through the model
$W'$	a candidate quantized weight matrix
$\hat{W}$	the quantized weights that minimize the reconstruction error

GPTQ solves that objective column by column, using the Hessian of the layer’s reconstruction loss — which for this squared-error objective is simply $\mathbf{H} = 2XX^{\top}$ — to decide, after quantizing one weight, how to adjust every weight not yet quantized to compensate for the error just introduced:

\delta = -\,\frac{w_{q} - \mathrm{quant}(w_{q})}{\left[\mathbf{H}^{-1}\right]_{qq}} \cdot \mathbf{H}^{-1}_{:,\,q}

$w_q$ is the single weight currently being quantized, and $\mathrm{quant}(w_q)$ is its rounded value — their difference is the error this step introduces. $\mathbf{H}^{-1}$ is the inverse Hessian computed once per layer from the calibration activations; $\left[\mathbf{H}^{-1}\right]_{qq}$ is its diagonal entry for weight $q$ , and $\mathbf{H}^{-1}_{:,\,q}$ is its full column for that weight. $\delta$ is the update applied to every weight not yet quantized in that row. What this buys you: rather than freezing every weight the moment it’s rounded, GPTQ nudges the remaining, still-unquantized weights to absorb the rounding error just made — the same principle earlier second-order pruning methods used for removing weights, adapted here to rounding them instead. It is what makes 3–4 bit weight-only quantization survive contact with a 175-billion-parameter model in a few GPU-hours rather than requiring a full retraining run.

Symbol	Meaning
$w_q$	the weight currently being quantized
$\mathrm{quant}(w_q)$	its rounded value
$\mathbf{H}$	Hessian of the layer’s reconstruction loss, $2XX^{\top}$
$\mathbf{H}^{-1}_{:,\,q}$	the inverse Hessian’s column for weight $q$
$\delta$	the compensating update applied to the remaining weights

SmoothQuant’s smoothing step divides each activation channel by a per-channel factor and multiplies the corresponding weight channel by the same factor, before either is quantized:

Y = \big(X\,\mathrm{diag}(s)^{-1}\big)\big(\mathrm{diag}(s)\,W\big), \qquad s_j = \frac{\max(|X_j|)^{\alpha}}{\max(|W_j|)^{1-\alpha}}

$X$ and $W$ are the activation and weight matrices for one linear layer, and $Y$ is the layer’s output — unchanged by the transformation, since dividing one factor and multiplying the other by the same $\mathrm{diag}(s)$ cancels out algebraically. $s_j$ is the smoothing factor for channel $j$ , computed offline from the largest activation and weight magnitudes seen for that channel across calibration data. $\alpha$ is a migration-strength hyperparameter, typically 0.5, that controls how much of the burden moves from activations toward weights: at $\alpha$ near 1, almost all the difficulty shifts onto the weights; near 0, almost none does. What this buys you: after smoothing, both $X\,\mathrm{diag}(s)^{-1}$ and $\mathrm{diag}(s)\,W$ are easier to quantize than the originals were, because the transformation actively flattens whichever operand had the sharper outliers — the mathematical move that makes full INT8 weight-and-activation quantization, not just weight-only, survive at LLM scale.

Symbol	Meaning
$X, W$	activation and weight matrices for one linear layer
$\mathrm{diag}(s)$	diagonal matrix of per-channel smoothing factors
$s_j$	smoothing factor for channel $j$
$\alpha$	migration-strength hyperparameter

AWQ’s channel scaling is the same equivalent-transformation idea, computed to protect specific salient channels rather than to flatten the whole distribution — scaling a salient weight channel up by $s$ before quantizing shrinks that channel’s own rounding error by roughly the same factor, since the per-tensor scale used for the rest of the quantization grows only marginally when a single channel is scaled:

Q(w \cdot s) \cdot \frac{x}{s} \approx w \cdot x, \qquad \mathrm{Err}\!\left(Q(w\cdot s)\cdot \frac{x}{s}\right) \approx \frac{\Delta'}{s}\cdot\frac{1}{\Delta}\cdot \mathrm{Err}\big(Q(w)\cdot x\big)

$Q(\cdot)$ is the round-to-nearest quantization function from the first equation in this section; $w$ is a single salient weight and $x$ its corresponding activation. Scaling $w$ up by $s$ before quantizing and dividing the activation $x$ by the same $s$ leaves the product unchanged before quantization is applied, but the quantization error on the now-larger weight shrinks relative to its own magnitude. $\Delta$ and $\Delta'$ are the per-tensor quantization step sizes before and after scaling one channel; because scaling a single salient channel barely changes the tensor’s overall maximum, $\Delta'$ stays close to $\Delta$ , so the error reduction from dividing by $s$ is not canceled out by a coarser step size. What this buys you: the same protection LLM.int8() achieved by physically routing outliers into a separate 16-bit multiplication, achieved instead with a per-channel multiply that a uniform 4-bit integer kernel can execute directly — no mixed-precision hardware path required.

Symbol	Meaning
$Q(\cdot)$	the round-to-nearest quantization function from the first equation in this section
$w, x$	a salient weight and its corresponding activation
$s$	the per-channel scaling factor applied before quantizing
$\Delta, \Delta'$	the per-tensor quantization step size before and after scaling one channel
$\mathrm{Err}(\cdot)$	the quantization error of a given weight-activation product

What It Enabled

The four methods in this chapter split along two axes: what gets quantized (weights alone, or weights and activations together) and how aggressively (8 bits, preserving nearly all of a model’s precision, or 4 bits, trading more accuracy for more compression). LLM.int8() and SmoothQuant quantize both weights and activations; GPTQ and AWQ quantize weights only, leaving activations in higher precision and betting that weight storage, not activation compute, is the inference bottleneck worth solving.

Method	Year	What’s quantized	Bit-width	Needs calibration data	Reported result
LLM.int8()	2022	Weights + activations	8-bit (16-bit for outliers)	No	175B-parameter inference with no measured accuracy loss
GPTQ	2022	Weights only	3–4-bit	Yes, small batch	175B model quantized in ~4 GPU-hours; 3.25x (A100) to 4.5x (A6000) speedup over FP16
SmoothQuant	2022	Weights + activations	8-bit (W8A8)	Yes, small batch	Up to 1.56x speedup, 2x memory reduction; 530B model on a single node
AWQ	2023	Weights only	4-bit	Yes, small batch	>3x speedup over Hugging Face FP16 via TinyChat; 70B Llama 2 on a mobile GPU

The plainest way to see what four years of work bought is to hold a fixed model size against each format. A 175-billion-parameter model’s weights alone, counted at 2 bytes per parameter in FP16, need about 350 GB; at 1 byte per parameter under LLM.int8() or SmoothQuant, about 175 GB; at roughly 0.5 bytes per parameter under GPTQ or AWQ’s 4-bit packing, about 87.5 GB — the difference between needing a multi-node A100 cluster and fitting comfortably on a single high-end GPU, for the same trained weights.

Precision	Bytes per parameter	Weight memory for a 175B-parameter model
FP16 (training-time default)	2	~350 GB
INT8 (LLM.int8, SmoothQuant)	1	~175 GB
4-bit (GPTQ, AWQ)	~0.5	~87.5 GB

None of these techniques quantize the part of inference memory that grows with every additional token generated: the KV cache. That problem — and the multi-query and grouped-query attention variants, PagedAttention, and speculative decoding built to manage it — is the subject of Chapter 17, where a serving stack combines the weight compression covered here with a separate set of techniques for the activations accumulated during generation itself. Weight quantization also became a fine-tuning tool in its own right: QLoRA, published in May 2023, combined 4-bit weight quantization with the low-rank adapters this book covers in Chapter 14, keeping a frozen base model quantized in memory while training small adapter matrices in higher precision on top of it — a direct descendant of the same bit-shrinking math this chapter derives. And the underlying pressure quantization relieves — a trained model’s parameter count outrunning any single accelerator’s memory (Chapter 4) — is the same pressure that motivated sharding a model’s weights across many devices during training in the first place; quantization is the inference-time answer to a problem Chapter 9 covers on the training side.

Key Papers

Paper	Authors	Year	Link	Why it matters
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale	Dettmers et al.	2022	arXiv:2208.07339	Identified emergent outlier features past ~6.7B parameters and solved them with vector-wise quantization plus a 16-bit mixed-precision carve-out
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers	Frantar et al.	2022	arXiv:2210.17323	One-shot, Hessian-based weight-only quantization to 3–4 bits; quantized a 175B model in about four GPU-hours
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models	Xiao et al.	2022	arXiv:2211.10438	Migrated quantization difficulty from activations to weights via an equivalent transformation, enabling full INT8 (W8A8) inference
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration	Lin et al.	2023	arXiv:2306.00978	Identified salient weight channels from activation statistics and protected them via per-channel scaling; shipped with the TinyChat inference engine

PM Lens

Quantization is the line item that turns “we’d need an 8-GPU node” into “this runs on one card,” and that difference shows up directly in a serving budget, not just a research paper. When a vendor or an open-weights release ships both a full-precision and a 4-bit checkpoint, GPTQ or AWQ is almost always underneath the second one, and the honest question to ask is what calibration set it was quantized against — both methods need a small batch of representative data, and a mismatch between that calibration data and your real traffic is where quietly degraded quality hides. The four methods here also aren’t interchangeable: SmoothQuant’s W8A8 helps when activation compute is the bottleneck (long prompts, high throughput); GPTQ and AWQ’s weight-only 4-bit helps when just fitting the model in memory is the blocker (single-GPU or on-device deployment). I’ve learned to ask which bottleneck a team is actually solving before approving which quantization scheme to standardize on, because the wrong choice looks fine in a demo and shows up as latency or quality regressions in production traffic three weeks later.

Search