The Co-evolution Thesis

Line up five processor launches against five model breakthroughs and the pairings are almost too clean: a gaming card from November 2010 trains the network that ends one era of computer vision in September 2012; a data-center GPU announced in May 2017 hosts the architecture, published one month earlier, that ends the recurrent-network era; and each accelerator generation since has arrived within months of the model class that made it necessary, or that it made possible. This chapter is the ledger that pairs them off, tests the thesis this book opened with, and asks what it implies for the silicon still on the drawing board.

Where We Were

Chapter 1 opened this book with Sara Hooker’s argument, published in September 2020: research directions do not win because they are intrinsically superior, they win because they fit the hardware and tooling available at the moment someone tries to scale them up, and ideas that do not fit that hardware can sit unrecognized for decades regardless of merit (Hooker, 2020). Chapter 1 traced that mechanism through the perceptron, the first AI winter, and the accidental fit between convolutional networks and the parallel arithmetic that GPUs happened to be good at by 2012. That is the thesis stated and demonstrated once, on one historical hinge point.

What Chapter 1 could not do, writing from the vantage of a single transition, is show whether the pattern repeats. A one-off correlation between a GPU generation and a model breakthrough could be coincidence — Kepler-generation GPUs existed for years before anyone trained AlexNet on one, and plenty of Kepler-era research had nothing to do with deep learning. A pattern that recurs across five separate hardware generations, each timed within months of the architecture class that depended on it, is harder to wave away as coincidence, and easier to treat as a description of a real feedback loop: better hardware makes a class of model economically trainable, that model class demonstrates a return on investment, and the return on investment funds the next hardware generation built specifically to remove whatever bottleneck the previous one hit.

This book’s middle chapters gave that loop its individual gears, each one adopted by chip vendors because a training or serving bottleneck made it commercially unavoidable, not because it was elegant. Chapter 4 showed that a chip’s peak FLOPs count is only half the story — bandwidth to memory sets a second ceiling, the roofline, and closing that ceiling is why every accelerator since 2017 has shipped with HBM stacked millimeters from the compute die. Chapter 7 showed how narrower number formats — FP16, then BF16, then FP8 — trade numeric range for throughput per transistor. Chapter 12 showed how a purely software change, tiling attention to avoid materializing an intermediate matrix in HBM, moved a memory-bound kernel most of the way to a chip’s compute roof without a new chip at all. Each of those chapters is a piece of the mechanism; this chapter is where the pieces get lined up against the calendar.

Five hardware generations each make one model class newly affordable; the pairings between chip and model line up almost too cleanly to be coincidence. PNG ↓

The Breakthrough

The claim this chapter tests is narrow and falsifiable: for each of five hardware generations since 2010, the model class that defined the following two to three years was one that the new hardware’s specific bottleneck-relief made newly affordable, not merely faster. The GTX 580, launched November 9, 2010 with 1.5 GB of GDDR5 memory and 512 CUDA cores, was a consumer gaming part with no deep-learning-specific features at all — it was cheap, parallel, and programmable, and that was enough. Alex Krizhevsky trained AlexNet across two of them, splitting the network’s layers between the two cards because neither card’s memory could hold the whole model alone, a constraint Chapter 2 covers directly. The GPU did not need to be designed for neural networks to enable one; it needed to be cheap enough that a graduate student could buy a pair.

By the time NVIDIA announced the Volta architecture and the V100 at its May 2017 GPU Technology Conference, the loop had closed once: AlexNet’s win had made deep learning a datacenter procurement line item, and NVIDIA answered with a chip built around 640 dedicated Tensor Cores delivering roughly 120 TFLOP/s of FP16 throughput and 900 GB/s of HBM2 bandwidth — the first GPU generation with matrix-multiply hardware purpose-built for neural network workloads rather than repurposed from graphics pipelines. One month earlier, in June 2017, Vaswani and coauthors published the transformer, an architecture whose core operation is exactly the large, regular matrix multiplication those Tensor Cores were built to accelerate (Chapter 5, language-models owns the architecture itself). The transformer did not require Volta to exist on paper, but training one at any useful scale on the recurrent-network-era hardware that preceded Volta would have cost far more per token — an early, sharp instance of Hooker’s mechanism running forward instead of backward: a model shaped to fit the hardware that had just become available, rather than hardware built to fit a model that already existed.

NVIDIA’s five-chip lineage is the clearest example of the loop because NVIDIA sells chips to many labs and cannot fully predict which model class will make a given generation valuable. Google’s TPU line, covered from its 2015 debut in Chapter 3, runs the same loop with the demand and supply sides collapsed into one company, which makes the model-to-hardware direction easier to see directly rather than infer after the fact. TPU v4, described in a 2023 retrospective architecture paper after operating in Google’s datacenters from 2020, is built around an optically reconfigurable network of 4,096 chips that lets Google’s own scheduler rewire which chips talk to which at runtime, plus hardware support for the very large embedding lookups that Google’s own recommendation and language models rely on (Jouppi et al., 2023). Neither feature is a generic accelerator improvement of the kind a merchant GPU vendor would ship to every customer; both were added because Google’s internal model workloads specifically needed them, which is the loop’s model-to-hardware arrow running with no gap between the lab that trains the models and the team that designs the chip.

A100, announced by NVIDIA in May 2020 with 312 TFLOP/s of dense FP16 throughput and 2.04 TB/s of HBM2e bandwidth, was the generation that made GPT-3’s regime practical: OpenAI’s May 2020 paper on few-shot in-context learning (Chapter 7, language-models owns the model itself) trained a 175-billion-parameter dense transformer, and doing so on the generation before A100 would have meant either a much smaller model or a much longer, more expensive training run. H100, announced March 2022, added a fourth-precision option — 8-bit floating point, Chapter 7 covers the format itself — and was followed two months later, in May 2022, by FlashAttention, refined into a Hopper-specific FP8 kernel by FlashAttention-3 in July 2024 (Chapter 12). The combination of a chip with roughly 2x the peak throughput of A100 in the same precision, an even wider FP8 mode, and a kernel that let attention actually approach that throughput rather than stall on memory, is the hardware substrate every GPT-4-class model shipped between March 2023 and the reasoning models of 2024 trained and served on. GB200, announced March 2024, is built for a workload that barely existed when H100 shipped: not a single long training run, but a serving fleet generating thousands of chain-of-thought tokens per query, the test-time-compute economics Chapter 19 covers in depth.

Chip and model rows take turns leading each other; each chip unlocks the model class beneath it, and each model class sets what the next chip has to do. PNG ↓

The loop is genuinely bidirectional, and the direction reverses depending on where in a generation you look. Early in a hardware generation’s life, the causal arrow runs hardware-to-model: a new chip removes a specific bottleneck, and researchers reach for the model class that bottleneck had been suppressing. Late in a generation’s life, once a model class has demonstrated commercial return, the arrow runs the other way: labs specify what the next chip needs to do, and vendors design toward that specification. NVIDIA’s own account of Hopper’s FP8 support and the co-design of FlashAttention-3’s Hopper-specific kernels in July 2024 is a late-generation, model-to-hardware instance of the loop; AlexNet on the GTX 580 is an early-generation, hardware-to-model instance. Both are the same feedback loop, observed at different points in its cycle.

The Mathematics

The loop above is qualitative. Making it quantitative requires separating the three things that grow across generations and can be traded off against each other: how fast a single chip’s arithmetic units run, how much of that speed a given piece of code actually achieves, and how many chips a lab points at one job. Define effective compute, the useful arithmetic a training run actually gets done per unit time, as the product of those three factors:

C_{\text{eff}}(t) = N(t) \cdot P_{\text{peak}}(t) \cdot \eta(t)

$N(t)$ is the number of chips a lab deploys on one job at time $t$ — the cluster-scale axis, the one that grew from two GTX 580s to a GB200 NVL72 rack’s 72 GPUs and beyond, across many racks, for frontier training runs. $P_{\text{peak}}(t)$ is the peak arithmetic throughput of a single chip at the precision being used, the axis vendor datasheets advertise. $\eta(t)$ is model FLOPs utilization — the fraction of that peak a real training or serving job actually achieves, bounded above by 1, and shaped by Chapter 4’s roofline, Chapter 7’s numeric formats, and kernel-level work like Chapter 12’s FlashAttention. $C_{\text{eff}}(t)$ , the product of all three, is what actually determines how large a model a lab can afford to train in a given wall-clock time and budget. What it buys you: a decomposition of “more compute” into three separately movable levers, which is why a single chip generation’s launch date is never the whole story — a lab that doubles cluster size on old, poorly utilized hardware can lose the race to a lab with half the chips and triple the utilization.

Symbol	Meaning
$C_{\text{eff}}(t)$	Effective compute actually delivered to a job at time $t$
$N(t)$	Number of chips deployed on the job
$P_{\text{peak}}(t)$	Peak arithmetic throughput of one chip, at the precision in use
$\eta(t)$	Model FLOPs utilization — achieved throughput as a fraction of peak

The scaling-laws literature this book’s Chapter 10, language-models owns treats $C_{\text{eff}}$ as a single input and asks how loss falls as it grows; this chapter is instead interested in why $C_{\text{eff}}$ itself has grown roughly four orders of magnitude since 2012, and the decomposition shows the answer is not any one term. $P_{\text{peak}}$ rose from the GTX 580’s roughly 1.6 TFLOP/s of FP32 throughput to the H100’s near-2,000 TFLOP/s of dense FP8 — already a three-order-of-magnitude jump before counting sparsity or lower precisions further — but $\eta$ moved in the same direction across the same period: naive attention implementations before FlashAttention often ran at a small fraction of a GPU’s advertised throughput on long sequences, purely from HBM traffic, and IO-aware kernels closed much of that gap without adding a single FLOP of peak hardware capability. $N$ grew fastest of all: training runs went from two consumer cards to clusters of tens of thousands of datacenter accelerators.

A second equation makes the economics of that growth explicit, and is the one that governs the export-control section below. For a lab renting or amortizing chip time at price $\rho$ per chip-hour, the cost of achieving a fixed amount of effective compute is:

\text{Cost} = \frac{C_{\text{eff}} \cdot \rho}{P_{\text{peak}} \cdot \eta}

Symbol	Meaning
$\text{Cost}$	Total spend to reach a fixed amount of effective compute
$C_{\text{eff}}$	Target effective compute, as defined above
$\rho$	Rental or amortized price per chip-hour
$P_{\text{peak}}$	Peak arithmetic throughput of one chip, at the precision in use
$\eta$	Model FLOPs utilization — achieved throughput as a fraction of peak

What It Enabled

The clearest way to see the co-evolution loop is to line up the five hardware-model pairings the introduction promised, with the specific bottleneck each hardware generation removed.

Hardware, launch	Peak throughput, native precision	Model class enabled	Bottleneck removed
GTX 580, Nov 2010	≈1.6 TFLOP/s FP32	AlexNet, 2012 (Chapter 2)	Cheap, programmable parallel arithmetic outside a national lab
V100, May 2017	≈120 TFLOP/s FP16, Tensor Cores	Transformer, 2017 (language-models/05)	Purpose-built matrix-multiply hardware for the exact op attention runs
A100, May 2020	312 TFLOP/s FP16, dense	GPT-3, 2020 (language-models/07)	Bandwidth and throughput for 175B dense parameters at feasible cost
H100, announced Mar 2022	≈1,979 TFLOP/s FP8, dense	GPT-4-class models, 2023 (language-models/18)	FP8 throughput plus IO-aware attention closing the utilization gap
GB200 NVL72, announced Mar 2024	1.44 EFLOP/s FP4, rack-scale	Reasoning models, 2024–2025 (Chapter 19, language-models/20)	Rack-scale NVLink bandwidth for long chain-of-thought serving, not just training

The table is not a claim that any of these chips was designed with foreknowledge of the model it would enable — the GTX 580 predates AlexNet by two years and was designed for Battlefield 3, not backpropagation. It is a claim that the model class that mattered most in each hardware generation’s window was the one that generation’s specific relief — cheap parallelism, matrix hardware, bandwidth, low precision, rack-scale interconnect — happened to unlock, exactly the mechanism Hooker’s paper names.

The clearest stress test of that mechanism arrived in December 2024, when DeepSeek-AI published the DeepSeek-V3 technical report (DeepSeek-AI, 2024), a 671-billion-parameter mixture-of-experts model — the routing mechanism itself is Chapter 13’s territory — trained not on H100s but on NVIDIA H800s, a China-compliant variant NVIDIA designed after the U.S. Commerce Department’s Bureau of Industry and Security issued export controls on October 7, 2022 capping advanced AI accelerators sold into China above a fixed performance-density threshold. The H800 kept H100’s compute throughput largely intact while cutting NVLink interconnect bandwidth well below H100’s, which is precisely the $P_{\text{peak}}$ -and-bandwidth ceiling the cost equation above describes. DeepSeek’s technical report states the final training run used 2.788 million H800 GPU-hours; at an assumed rental price of $2 per GPU-hour, that comes to $5.576 million for the final pretraining, context extension, and post-training stages combined — a figure the report itself frames as excluding earlier research and ablation costs, not a total project budget. The report’s engineering — an FP8 mixed-precision training recipe and a custom communication scheduler built specifically to work around H800’s narrower interconnect — is a direct, dated instance of the $\eta$ term absorbing a cut to $P_{\text{peak}}$ and interconnect bandwidth, exactly as the cost equation predicts a lab facing a hardware ceiling must do.

The market reaction in January 2025, when DeepSeek’s R1 reasoning model release triggered a sharp drop in AI-hardware-linked equities, was in large part a reaction to that engineering result: proof that a lab constrained to a lower-bandwidth chip could still reach a competitive model class by raising utilization, which read to markets as evidence that frontier capability might require less $N \cdot P_{\text{peak}}$ than the highest-end hardware suppliers had priced in. The U.S. Commerce Department tightened the rule again on October 17, 2023, closing the specific performance-density loophole the H800 and its A800 sibling had used, which is itself an instance of the model-to-hardware-policy loop: a workaround demonstrated its value, and the constraint was redrawn around it.

That efficiency story sits inside a larger cost the co-evolution loop keeps generating regardless of which lab wins a given generation: the electricity to run it. The International Energy Agency’s 2025 “Energy and AI” report estimates global datacenter electricity consumption at around 415 TWh in 2024, about 1.5% of global electricity demand, growing roughly 12% annually over the preceding five years and projected to reach around 945 TWh by 2030 — just under 3% of projected global demand — with electricity consumption in AI-accelerated servers specifically projected to grow about 30% annually and account for almost half of that net increase (International Energy Agency, 2025). Every row in the master table above bought more effective compute per dollar of hardware; none of them has yet reduced the trend in total energy demand, because the industry has consistently spent efficiency gains on doing more, not on doing the same amount for less.

Commercial return normally feeds the next bottleneck directly; export controls insert a detour through engineering instead, and the loop still closes. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
The Hardware Lottery	Hooker	2020	arXiv:2009.06489	Names the mechanism this chapter tests across five hardware generations rather than one
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings	Jouppi et al.	2023	arXiv:2304.01433	Documents the model-to-hardware direction of the loop: a chip designed around embeddings and reconfigurable topology because Google’s own model workloads demanded it
DeepSeek-V3 Technical Report	DeepSeek-AI	2024	arXiv:2412.19437	The clearest dated instance of utilization absorbing an export-control-driven hardware cut, and the trigger for the January 2025 market reassessment of hardware demand
Energy and AI	International Energy Agency	2025	iea.org/reports/energy-and-ai	The co-evolution loop’s uncounted cost: efficiency gains have consistently funded more compute, not less electricity

PM Lens

I have sat in enough hardware-roadmap reviews to notice the same tell every time: the slide always leads with a chip’s peak FLOPs number, and the roadmap decision almost never actually turns on that number. It turns on whether the model class we want to ship next fits the specific bottleneck the new chip relieves — memory bandwidth for a bigger context window, interconnect for a bigger MoE, FP8 support for a cheaper serving fleet — and whether we can get enough of that chip before a competitor does. DeepSeek’s H800 result made that argument for me better than any internal analysis could: a team with a deliberately worse interconnect still shipped a frontier-class model by spending engineering effort on utilization instead of buying more silicon, at a reported $5.576 million for the final training run. When I now evaluate an accelerator generation for a product decision, I ask the DeepSeek question first — what is our $\eta$ , not just our $N \cdot P_{\text{peak}}$ — before I ask what the datasheet promises. The datasheet number is a ceiling nobody hits on the first try.

Search