The Hardware Lottery

It is 1958, and inside a converted machine room at the Cornell Aeronautical Laboratory a device called the Mark I Perceptron is learning to tell cards marked on the left from cards marked on the right, using a bank of 400 photocells and rows of motor-driven potentiometers standing in for adjustable connection strengths. The learning rule inside that machine is one a modern engineer would recognize immediately: compare a prediction to a label, and nudge the weights toward the answer. What will change, more than once over the following decades, is not the rule but the machine underneath it — and each change decides which version of the idea gets to survive.

Where We Were

Frank Rosenblatt’s perceptron was built to be biologically suggestive and computationally exact at the same time. In The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Rosenblatt described a network of threshold units — modeled loosely on retinal neurons — whose connection strengths could be adjusted automatically from labeled examples (Rosenblatt, 1958). The Office of Naval Research funded a hardware build, not just a simulation: the Mark I Perceptron used an actual photocell array as its retina and physically moved potentiometer dials to represent updated weights. Algorithm and hardware were, for a brief moment, designed together, and the press coverage that followed — the New York Times reported the Navy expected it to eventually “walk, talk, see, write, reproduce itself and be conscious of its existence” — was wildly ahead of what a single layer of threshold units could do.

The correction arrived eleven years later. Marvin Minsky and Seymour Papert’s Perceptrons: An Introduction to Computational Geometry gave a rigorous account of what a single layer of linear threshold units could and could not compute (Minsky and Papert, 1969). The headline result was that a single-layer perceptron cannot compute the logical XOR function — a problem with no linear boundary separating its positive and negative examples. The book was read, fairly or not, as a case against the entire connectionist program, and funding for neural-network research in the United States and United Kingdom contracted sharply through the 1970s. This is the pattern the rest of the chapter turns on: a limitation that was, in this case, genuinely algorithmic — no hardware fixes a boundary that a straight line cannot draw — got generalized into a verdict on the whole approach, at exactly the moment a training method for multi-layer networks did not yet exist.

That training method arrived in 1986. David Rumelhart, Geoffrey Hinton, and Ronald Williams showed how to propagate an error signal backward through multiple layers of a network, adjusting every weight in proportion to its share of the blame for the output error (Rumelhart, Hinton, and Williams, 1986). Backpropagation removed the algorithmic obstacle Minsky and Papert had identified — a two-layer network trained this way can compute XOR — and touched off a second wave of connectionist research. It immediately ran into a different kind of obstacle. General-purpose CPUs in the mid-1980s executed the dense matrix multiplications a multi-layer network needs at a small fraction of the throughput that would later make training practical, and there was no dedicated hardware built for the purpose the way the Mark I Perceptron’s photocells had been built for Rosenblatt’s model. The algorithm was ready before the machine was.

Yann LeCun and colleagues spent the following decade demonstrating what backpropagation-trained networks could do on a task with genuine commercial stakes: reading handwritten digits on bank checks and postal envelopes. Gradient-Based Learning Applied to Document Recognition laid out the convolutional network architecture — local receptive fields, shared weights, and pooling — and the training results that made it a deployed system, not a laboratory curiosity, at check-processing volume in the United States by the end of the 1990s (LeCun, Bottou, Bengio, and Haffner, 1998). The architecture worked. It did not generalize into the dominant approach for computer vision for another decade, because scaling it to color photographs of natural scenes, rather than grayscale digits on a fixed-size grid, needed far more compute than 1990s workstations could supply at an acceptable training time, and support-vector machines and hand-engineered feature pipelines filled the gap on the hardware that was actually available.

Two more pieces of the pre-1958–2009 record complete the setup for the rest of this book. In July 2006, Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh published A Fast Learning Algorithm for Deep Belief Nets, showing that a stack of restricted Boltzmann machines could be trained one layer at a time and then fine-tuned as a single deep network — a practical recipe for training networks deeper than the two or three layers that had been standard, still running on the CPUs of the day (Hinton, Osindero, and Teh, 2006). The paper is widely credited with reopening serious interest in deep architectures after roughly two decades in which “neural network” was, in much of the field, a term associated with the earlier plateau rather than the state of the art. Three years later, Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei released ImageNet: a labeled image dataset built on top of the WordNet hierarchy, reaching 3.2 million images across more than 5,000 categories at its first publication (Deng, Dong, Socher, Li, Li, and Fei-Fei, 2009). ImageNet supplied the benchmark; it did not yet supply the hardware to train on it economically at the scale the dataset demanded. Both pieces — a deep-training recipe and a dataset large enough to need one — were in place by mid-2009, three years before a GPU-trained network would win the competition built on that same dataset (the subject of the next chapter).

The Breakthrough

No new architecture or training method is the breakthrough of this chapter. The breakthrough is a piece of vocabulary, arriving more than sixty years after Rosenblatt’s machine, for the pattern that Where We Were just walked through in five separate incidents. Richard Sutton’s essay The Bitter Lesson, posted in March 2019, argues that the biggest lesson learnable from roughly seventy years of AI research is that general methods which scale with added computation eventually outperform methods built on hand-engineered, human-derived knowledge, even though the knowledge-based methods look better in the short run (Sutton, 2019). Sutton’s own examples are chess, Go, speech recognition, and computer vision, in each of which a search- or learning-heavy method that assumed less about the problem eventually beat a method built on domain expertise, once enough compute was available to run it.

Sara Hooker’s The Hardware Lottery, published in September 2020, supplies the mechanism Sutton’s essay leaves implicit: which methods get to “scale with computation” in the first place depends on whether the surrounding hardware and software stack was built to run them efficiently, and that fit is frequently accidental (Hooker, 2020). Hooker’s central case study is exactly the LeCun convolutional network from Where We Were — an architecture that existed in a form recognizable to a 2012 reader by 1998, but that only became the dominant approach in computer vision once GPUs, developed primarily for a different market, turned out to accelerate its dense, regular convolutions far more than they accelerated the sparser, more sequential computations of many of its 1990s-era competitors. The paper’s phrase for this is a “hardware lottery”: an idea wins or loses less because it is right or wrong on its merits and more because it happens to be compatible with the tools available at the moment someone tries it seriously.

Reading Hooker’s mechanism back onto the history in Where We Were produces a loop, not a one-off event, and it is the loop — not any single hardware artifact — that this book’s later chapters trace forward through GPUs, TPUs, and the accelerators built explicitly for transformers.

The perceptron sat in the “does not fit” branch through the 1970s not because threshold units were a bad idea but because no training method existed for the deeper networks that could fix their expressiveness limits. Backpropagation moved the idea toward “fits” in principle in 1986, but 1980s CPUs kept it in the slow-iteration branch for another decade. LeCun’s convolutional networks sat in that same branch through the 1990s, waiting on hardware that had not yet been built for an unrelated purpose — real-time 3D graphics — and would only later turn out to accelerate exactly the computation deep networks needed. Chapter 2 of this book picks up the loop at the point where that hardware, and a research group willing to bet a benchmark result on it, finally meet.

The Mathematics

Rosenblatt’s perceptron makes a single binary decision from a weighted sum of its inputs:

$$\hat{y} = \mathrm{step}\left( \mathbf{w}^\top \mathbf{x} + b \right)$$

Learning happens by comparing that prediction to a label and moving the weights toward the label:

$$\mathbf{w} \leftarrow \mathbf{w} + \eta \left( y - \hat{y} \right) \mathbf{x}$$

Symbol	Meaning
$\mathbf{w}$	the weight vector, one entry per input feature
$b$	the bias term, shifting the decision boundary independent of any input
$\eta$	the learning rate, controlling the size of each update
$y$	the true label for the current example
$\hat{y}$	the perceptron’s predicted label
$\mathbf{x}$	the input feature vector for the current example

Minsky and Papert’s argument against a single layer of these units is that no choice of weights $w_1, w_2$ and bias $b$ can satisfy all four constraints the XOR function imposes at once:

$$\begin{aligned} w_1 \cdot 0 + w_2 \cdot 0 + b &\le 0 \\ w_1 \cdot 1 + w_2 \cdot 0 + b &> 0 \\ w_1 \cdot 0 + w_2 \cdot 1 + b &> 0 \\ w_1 \cdot 1 + w_2 \cdot 1 + b &\le 0 \end{aligned}$$

What It Enabled

The immediate effect of the 1958–2009 record was mostly frustration: a training method existed by 1986, a working deployed architecture existed by 1998, a practical deep-training recipe existed by 2006, and a benchmark dataset large enough to demand real capacity existed by 2009 — and none of it added up to the deep-learning era until a fifth ingredient, GPU-accelerated training, arrived and is taken up directly in GPUs Learn to Think. The durable effect was retrospective: once that ingredient arrived, Rosenblatt’s threshold units, Rumelhart, Hinton, and Williams’s backpropagation, and LeCun’s convolutions turned out to be exactly the pieces the new hardware was good at accelerating, and the field’s forward progress from 2012 onward is, in large part, those same decades-old ideas run on hardware finally built to fit them.

	1958–2009 (this chapter)	2012 onward (Chapters 2–3)
Compute substrate	general-purpose CPUs, or one-off custom hardware (Mark I Perceptron)	GPUs, later TPUs, purpose-accelerated for dense matrix math
Typical trainable depth	1–3 layers practical at scale; deeper required special tricks (DBN pretraining)	tens to hundreds of layers, trained end to end
Bottleneck	dense matrix multiplication throughput	still compute-bound, but throughput orders of magnitude higher
Field’s dominant paradigm	hand-engineered features plus shallow classifiers	learned features via deep networks trained on GPUs
Illustrative benchmark	ImageNet published with no trained model behind it (2009)	AlexNet wins ImageNet Large Scale Visual Recognition Challenge (2012)

Event	Year	Type of obstacle it hit
Mark I Perceptron	1958	none yet — hardware built to fit the idea
Minsky-Papert limits	1969	algorithmic: no training method for multi-layer nets
Backpropagation	1986	hardware: CPUs too slow for practical training
LeCun convolutional nets	1998	hardware: scaling beyond digits needed more compute than available
Deep belief nets	2006	hardware: recipe existed, throughput still CPU-bound
ImageNet dataset	2009	hardware: benchmark outran the compute to train on it

Every later chapter of this book is, in some sense, a continuation of the loop diagrammed above. TPUs and the Accelerator Cambrian covers the next turn of the wheel, in which Google built silicon specifically for the matrix multiplications neural networks need rather than inheriting hardware built for graphics. The full argument that each hardware generation and each dominant model architecture shape one another — GPUs shaping convolutional and then transformer designs, and those designs in turn shaping what chipmakers build next — is developed at length in The Co-evolution Thesis, which this chapter’s two framing papers exist to set up rather than to fully resolve.

Key Papers

Paper	Authors	Year	Link	Why it matters
The perceptron: a probabilistic model for information storage and organization in the brain	Rosenblatt	1958	doi	Introduces the trainable threshold unit and pairs it with purpose-built hardware — the book’s opening case of algorithm and machine designed together
Perceptrons: An Introduction to Computational Geometry	Minsky and Papert	1969	MIT Press	Proves the linear-separability limits of a single layer, precipitating a funding contraction later read as a verdict on connectionism generally
Learning representations by back-propagating errors	Rumelhart, Hinton, and Williams	1986	doi	Solves the algorithmic obstacle from 1969, then immediately exposes a hardware one
Gradient-Based Learning Applied to Document Recognition	LeCun, Bottou, Bengio, and Haffner	1998	paper	Deploys a convolutional network commercially, on hardware too slow to let the architecture scale beyond its original task for another decade
A Fast Learning Algorithm for Deep Belief Nets	Hinton, Osindero, and Teh	2006	doi	Practical recipe for training networks deeper than the era’s typical two or three layers, reopening interest in deep architectures
ImageNet: A Large-Scale Hierarchical Image Database	Deng, Dong, Socher, Li, Li, and Fei-Fei	2009	Princeton	Publishes the benchmark before the hardware to train on it at scale exists, setting up Chapter 2
The Bitter Lesson	Sutton	2019	essay	States the pattern in economic terms: general, compute-hungry methods beat hand-engineered ones once enough compute exists
The Hardware Lottery	Hooker	2020	arXiv	Names the mechanism behind the pattern and supplies this book’s title concept and its central case study

PM Lens