Mixed Precision

It is October 2017. Every network this book has followed so far — AlexNet in 2012 (Chapter 2), the optimizers that trained it (Chapter 5) — has trained in 32-bit floating point, the width GPUs happened to compute in by default. A team at NVIDIA publishes a way to run most of that arithmetic in 16 bits instead, cutting memory roughly in half and lifting throughput on the new Tensor Core hardware, provided the lost dynamic range is actively compensated for rather than ignored. That compensation, called loss scaling, opens a seven-year argument over how few bits a number needs to carry a gradient — an argument that runs through bfloat16, then 8-bit floats, and ends, for now, with frontier models pretrained natively in 8 bits.

Where We Were

Every training run in this book up to 2017 used single-precision floating point, FP32: 32 bits per number, split into 1 sign bit, 8 exponent bits, and 23 mantissa (fraction) bits. That split gives roughly seven decimal digits of precision and a dynamic range running from about $1.18 \times 10^{-38}$ up to about $3.4 \times 10^{38}$ for normal values. Weights, activations, gradients, and the momentum and variance buffers Adam keeps per parameter (Chapter 5) all lived at that width, because early GPU deep learning had no cheaper practical option: the arithmetic units doing the actual multiply-adds computed natively in FP32.

That began to change on the hardware side before it changed in training practice. In 2016, NVIDIA’s Pascal architecture — specifically the Tesla P100 — added dedicated FP16 arithmetic that ran at roughly twice the throughput of FP32 on the same chip, by packing two FP16 values into paths originally built for one FP32 value. In May 2017, NVIDIA’s Volta architecture went further with the Tesla V100, introducing Tensor Cores: dedicated units that multiply 4×4 matrices in FP16 and accumulate the result in FP32, delivering roughly twelve times the tensor throughput of Pascal. The hardware to train in half the bits, at higher speed, now existed. What was missing was a way to use it without losing accuracy.

Plain FP16 arithmetic buys its compactness by shrinking the exponent field from 8 bits to 5 and the mantissa from 23 bits to 10. That is a much narrower dynamic range than FP32’s: a largest finite value of 65,504 and a smallest normal value around $6.1 \times 10^{-5}$ . Gradients in a deep network routinely span more orders of magnitude than that window covers, especially in later layers where activations and their gradients tend to be smaller. A gradient value that falls below FP16’s smallest representable magnitude does not raise an error — it silently rounds to zero, a failure called underflow, and the corresponding weight simply stops updating. An activation or gradient that grows too large overflows to infinity instead. Both failures are quiet: the first visible symptom is usually degraded accuracy discovered well after the fact, not a crash at the moment it happens. That was the problem sitting in front of anyone who wanted to use Volta’s new Tensor Cores for something more than storage.

Tensor Core hardware ships in May 2017; the recipe that stops FP16 gradients from silently rounding to zero ships five months later. PNG ↓

The Breakthrough

NVIDIA’s Micikevicius et al. answered the underflow-and-overflow problem with a small set of concrete rules rather than a new number format (Micikevicius et al., 2017). Their recipe kept an FP32 master copy of every weight, used only for the optimizer update; the forward and backward passes themselves ran in FP16, with weights cast down to FP16 at the start of each iteration. Because most of the memory and compute in a training step goes into activations and their gradients, not the master weights, this halved memory for the bulk of the tensors while keeping the numerically sensitive accumulation step in full precision. The paper reported the technique matching FP32 training accuracy, with no changes to the model or most hyperparameters, across convolutional networks, recurrent language models, and GANs, on models with over 100 million parameters — with roughly half the memory footprint of pure FP32 training.

The part of the recipe that made FP16 usable for the gradients themselves was loss scaling. Because a large share of a network’s gradients cluster in a numeric range below FP16’s representable floor, the paper’s fix was to multiply the loss by a scale factor before the backward pass, which — by linearity of differentiation — multiplies every gradient in the backward pass by that same factor, shifting the whole distribution up into FP16’s representable range. After the backward pass computes the scaled gradients, they are divided back down by the same factor before the FP32 master weights are updated, undoing the shift. The scale factor itself needs tuning: too small, and gradients still underflow; too large, and gradients overflow to infinity instead. NVIDIA’s original recipe used a fixed scale value; later implementations made this dynamic, adjusting the scale up when training ran cleanly for a stretch and dropping it immediately whenever an overflow was detected.

Loss scaling was a workaround for a format problem, and Google had already built a different format that mostly avoided the problem outright. Designed for the TPU line (Chapter 3), bfloat16 (BF16) also uses 16 bits, but splits them differently: 8 exponent bits — identical to FP32 — and only 7 mantissa bits, instead of FP16’s 5-exponent, 10-mantissa split. Because BF16’s exponent field matches FP32’s exactly, its dynamic range matches FP32’s too; the values that made FP16 overflow or underflow are, by construction, still representable in BF16, at the cost of coarser precision within whatever exponent range a value falls into. In May 2019, Kalamkar et al. published the first comprehensive empirical study of BF16 training, covering image classification, speech recognition, language modeling, generative networks, and industrial recommendation systems (Kalamkar et al., 2019). They reported that BF16 training matched FP32 accuracy in the same number of iterations with no hyperparameter changes — and, notably, without loss scaling, since BF16’s range already covered what FP32’s did. NVIDIA added native BF16 Tensor Core support with the A100 (Ampere), which shipped in May 2020, and BF16 became the more common default for large-model pretraining over the next several years: less numerically fragile than FP16, at the cost of the extra precision FP16’s wider mantissa carried.

The next halving needed a purpose-built format rather than a repurposed 16-bit one. In September 2022, Micikevicius and a larger group of co-authors from NVIDIA, Arm, Intel, and Qualcomm proposed two 8-bit floating-point encodings for both training and inference: E4M3, with 4 exponent bits and 3 mantissa bits, and E5M2, with 5 exponent bits and 2 mantissa bits (Micikevicius et al., 2022). The names describe the same exponent/mantissa trade-off FP16 and BF16 made at 16 bits, now compressed into 8: E4M3 trades range for precision (a maximum finite magnitude of 448, but finer steps between values), and E5M2 trades precision for range (a maximum finite magnitude of 57,344, close to FP16’s 65,504 because both share a 5-bit exponent field, but only 2 mantissa bits). The paper’s recommendation, followed by most later implementations, was to use E4M3 for weights and activations in the forward pass, where precision matters more, and E5M2 for gradients in the backward pass, where values swing across a wider range. Tested across CNNs, RNNs, and Transformers up to 175 billion parameters — GPT-3 scale — FP8 training matched the accuracy of 16-bit training, and the paper additionally showed the same formats worked for post-training quantization at inference time.

FP8 needed dedicated silicon to be worth the added bookkeeping. NVIDIA’s H100 (Hopper), announced in March 2022, shipped fourth-generation Tensor Cores with native FP8 support and a Transformer Engine that dynamically recasts tensors between FP8 and 16-bit formats layer by layer. On data types both chips support, H100’s Tensor Cores deliver roughly twice A100’s per-SM throughput; FP8, a format A100 lacks entirely, pushes that to roughly four times A100’s per-SM throughput. A year and a half later, Peng et al. showed what that silicon could do at LLM scale: their FP8-LM framework extended FP8 beyond weights and activations to gradients, optimizer states, and distributed communication, applied in three incremental levels of adoption (Peng et al., 2023). Training a 175-billion-parameter GPT model on H100 hardware, FP8-LM reported a 39% reduction in real GPU memory usage and training 75% faster than the BF16-based Megatron-LM framework — and 37% faster than NVIDIA’s own Transformer Engine.

By December 2024, FP8 pretraining had reached frontier scale in a publicly documented model: DeepSeek-V3, a 671-billion-parameter mixture-of-experts model (Chapter 13), stored weights in FP8 and ran matrix multiplications in FP8 with FP32 accumulation, while keeping activations and gradients in BF16 and reserving FP32 for a handful of numerically sensitive operations. Its authors reported a relative loss error against a BF16 baseline consistently below 0.25% — within normal training-to-training variance — while contributing to a total reported training cost of $5.576 million across 2.788 million H800 GPU-hours. The seven years from a single fixed loss-scale value in one 2017 recipe to per-tensor, mixed-granularity scaling across a model with hundreds of billions of parameters is the arc this chapter traces.

FP16 and BF16 are two different answers to the same narrowed exponent; both roads lead to the format that needed a bespoke design rather than a repurposed one. PNG ↓

The Mathematics

Every format in this chapter is an instance of the same binary floating-point representation, differing only in how the available bits are split between exponent and mantissa:

x = (-1)^{s}\,(1+m)\,2^{\,e-b}

$x$ is the number being represented — a weight, activation, or gradient value. $s$ is the sign bit: 0 for positive, 1 for negative. $m$ is the mantissa, a fraction between 0 and 1 built from the mantissa bits, giving the value’s precision. $e$ is the exponent field, an unsigned integer stored directly in the exponent bits, and $b$ is a fixed bias subtracted off to let the true exponent go negative. What this buys you: every format below — FP32, FP16, BF16, FP8 E4M3, FP8 E5M2 — is the exact same formula with a different number of bits assigned to $e$ versus $m$ . More exponent bits widen the range of magnitudes $x$ can take; more mantissa bits narrow the gap between adjacent representable values. Every design decision in this chapter is a choice about where to spend a fixed bit budget along that one trade-off.

Symbol	Meaning
$x$	the represented number — weight, activation, or gradient
$s$	sign bit
$m$	mantissa fraction, built from the mantissa bits
$e$	exponent field, an unsigned integer
$b$	fixed exponent bias for the format

That trade-off, applied to the five formats this chapter covers, produces very different working ranges from the same 8, 16, or 32 bits:

Format	Sign	Exponent bits	Mantissa bits	Max finite magnitude	Smallest normal magnitude	Needs loss scaling
FP32	1	8	23	$\approx 3.4 \times 10^{38}$	$\approx 1.18 \times 10^{-38}$	No
FP16	1	5	10	65,504	$\approx 6.1 \times 10^{-5}$	Yes
BF16	1	8	7	$\approx 3.39 \times 10^{38}$	$\approx 1.18 \times 10^{-38}$	No
FP8 E4M3	1	4	3	448	$\approx 0.0156$	Yes
FP8 E5M2	1	5	2	57,344	$\approx 6.1 \times 10^{-5}$	Yes

FP16 and FP8 both need active range management precisely because they narrow the exponent field relative to FP32; BF16 does not, because it keeps FP32’s exponent field untouched and only narrows the mantissa.

Loss scaling, the technique that makes FP16 (and FP8) training usable despite that narrow exponent, is a three-step procedure: scale the loss up before backpropagation, let the scaling propagate through the gradients by linearity, then unscale before the weight update happens in FP32:

\begin{aligned} L_s &= S \cdot L(\theta) \\ g_s &= \nabla_\theta L_s \\ \theta_{t+1} &= \theta_t - \eta \cdot \dfrac{g_s}{S} \end{aligned}

$L(\theta)$ is the ordinary training loss as a function of the model’s parameters $\theta$ . $S$ is the scale factor — a single number, often in the hundreds or thousands, chosen to lift the gradient distribution into FP16’s representable range without pushing values into overflow. $L_s$ is the scaled loss actually passed into backpropagation, and $g_s$ is the resulting scaled gradient — every gradient value computed on $L_s$ comes out exactly $S$ times larger than it would from the true loss, because differentiation is linear. $\eta$ is the learning rate, $\theta_t$ and $\theta_{t+1}$ are the FP32 master weights before and after the update, and dividing $g_s$ by $S$ before the update step restores the true, unscaled gradient magnitude. What this buys you: the backward pass — the part of training with the widest range of small values — runs entirely in FP16’s narrow window without those values collapsing to zero, while the weight update that actually accumulates change over thousands of steps happens in full FP32 precision, where rounding error would otherwise compound.

Symbol	Meaning
$L(\theta)$	true training loss, function of parameters $\theta$
$S$	scale factor applied before backpropagation
$L_s, g_s$	scaled loss and the resulting scaled gradient
$\eta$	learning rate
$\theta_t, \theta_{t+1}$	FP32 master weights before and after the update

FP8’s exponent field is narrower still than FP16’s, so a single global scale factor chosen once at the start of training tends to be too coarse — the FP8 training frameworks that followed the 2022 formats paper instead recompute a scale per tensor, from that tensor’s own recent magnitude:

S_x = \dfrac{\mathrm{FP8}_{\max}}{\max(|x|)}

$x$ here is a whole tensor of values — a weight matrix, an activation, or a gradient — about to be cast down to FP8. $\mathrm{FP8}_{\max}$ is the largest finite magnitude the target FP8 format can hold: 448 for E4M3, 57,344 for E5M2. $S_x$ is the scale applied to that specific tensor before casting, chosen so its largest current value lands exactly at the format’s ceiling instead of overflowing or leaving most of the range unused. What this buys you: loss scaling in the 2017 paper picked one number for the entire network; per-tensor scaling picks a fresh number for every tensor, recomputed as training progresses and each tensor’s typical magnitude shifts — the generalization loss scaling needed once FP8’s exponent budget got tight enough that one global number stopped being precise enough.

What It Enabled

Each format halving did not replace the one before it outright; FP16 and BF16 both remain in production use, and FP8 sits alongside both rather than eliminating them. What changed with each step was how much memory and communication bandwidth a model of a given accuracy required, and how large a model a fixed hardware budget could train at all.

Technique	Year	Precision	Reported result	Model scale tested
Mixed Precision Training	2017	FP16 + FP32 master weights	Roughly 2x memory reduction, matched FP32 accuracy	100M+ parameters (CNN, RNN, GAN)
BF16 empirical study	2019	BF16, no loss scaling	Matched FP32 accuracy, no hyperparameter changes	Five domains, including language modeling
FP8 Formats for Deep Learning	2022	FP8 (E4M3 / E5M2)	Matched 16-bit training quality	Up to 175B parameters
FP8-LM	2023	FP8 across weights, gradients, optimizer states	39% less GPU memory, 75% faster than BF16 Megatron-LM	175B-parameter GPT on H100
DeepSeek-V3	2024	Native FP8 pretraining	Relative loss error under 0.25% vs. BF16 baseline	671B-parameter MoE, $5.576M total training cost

Halving the bytes per parameter has consequences beyond a single GPU’s memory. Distributed data parallelism moves every gradient across the network once per step (Chapter 8); a model trained in FP16 or BF16 instead of FP32 roughly halves that communication volume for free, on top of whatever the memory savings do for how large a model or batch fits per device. The same halving compounds again inside ZeRO’s memory partitioning (Chapter 10), whose bytes-per-parameter accounting for weights, gradients, and optimizer states is directly a function of which precision each of those tensors is stored in. FlashAttention-3, released in July 2024, added Hopper-specific FP8 kernels to attention’s own matrix multiplications, extending this chapter’s numerics into the kernel layer covered in Chapter 12. And where this chapter’s formats change the arithmetic used during training, Chapter 15 covers a related but distinct problem: compressing an already-trained model’s weights, often to formats like int8 or 4-bit integers, purely to make inference cheaper.

The pattern of halving did not stop at 8 bits. NVIDIA’s Blackwell architecture (B200/GB200), announced in March 2024, added native FP4 support to its Tensor Cores, continuing the same exponent-and-mantissa trade-off one level further down. Whether that step follows the FP16 or the BF16 path — needing aggressive per-tensor scaling to stay usable, or finding a wider-range 4-bit split that avoids it — is a question for the hardware-and-model story told in full in Chapter 20.

Key Papers

Paper	Authors	Year	Link	Why it matters
Mixed Precision Training	Micikevicius et al.	2017	arXiv:1710.03740	Introduces FP32 master weights plus loss scaling, the recipe that made FP16 training on Tensor Core hardware usable without an accuracy loss
A Study of BFLOAT16 for Deep Learning Training	Kalamkar et al.	2019	arXiv:1905.12322	First comprehensive empirical validation that BF16’s wider exponent, narrower mantissa split matches FP32 accuracy without loss scaling
FP8 Formats for Deep Learning	Micikevicius et al.	2022	arXiv:2209.05433	Defines the E4M3 and E5M2 8-bit float formats and shows FP8 matching 16-bit training quality up to 175B parameters
FP8-LM: Training FP8 Large Language Models	Peng et al.	2023	arXiv:2310.18313	Extends FP8 to gradients, optimizer states, and distributed communication, reporting large memory and speed gains at GPT-175B scale on H100

PM Lens

Precision choice used to be a research paper’s footnote; by 2024 it was a line item in a training budget. DeepSeek-V3’s $5.576 million reported training cost is only legible next to what it would have cost in BF16 — FP8-LM’s own numbers, a 39% memory cut and a 75% speedup on a 175B-parameter model, are the kind of gain that turns into fewer GPU-hours rented at whatever an H100 costs per hour that quarter. When I scope a training or fine-tuning budget now, precision is one of the first three questions, alongside data volume and parameter count, because it multiplies both the compute bill and the largest model that fits on the hardware you already have. It also has a failure mode worth respecting: FP8 training is not free lunch, it is FP16’s underflow problem one exponent bit tighter, managed with finer-grained scaling than a single global number can provide. A team reaching for FP8 without that scaling infrastructure is choosing the 2017 problem with 2022-era hardware.

Search