The GPT Lineage

It is June 2018, one year after the Transformer. OpenAI takes only the decoder half of the architecture, trains it to predict the next token, and over the following twenty-three months publishes three papers whose main variable is size: 117 million parameters, then 1.5 billion, then 175 billion. Along the way, the interface to a language model changes from fine-tuning weights to simply writing a prompt — and this chapter is the story of that shift.

Where We Were

The Transformer (The Transformer) arrived in June 2017 as an encoder-decoder architecture for machine translation. It solved the training-speed and long-range-dependency problems of recurrent networks, but it did not by itself change how NLP systems were built. In 2017, the field still operated one task at a time: a bespoke architecture and a labeled dataset for sentiment analysis, another pair for question answering, another for entailment. Labeled data was the bottleneck, and nothing learned on one task transferred cheaply to the next.

The pretrain-then-finetune paradigm — the subject of Pretrain, Then Finetune, which teaches this era in full — broke that bottleneck during 2018. Within it, GPT-1 (Radford et al., 2018), released in June 2018, matters to this chapter for one specific decision: it was a 12-layer, decoder-only Transformer of roughly 117 million parameters, pretrained left-to-right on BooksCorpus and then fine-tuned per task. The architecture — masked self-attention so each token sees only its predecessors, a stack of identical blocks, a next-token softmax on top — is the template every subsequent GPT scales without structural change. Fine-tuned on twelve downstream tasks, this single task-agnostic architecture improved the state of the art on nine of them, including an 8.9-point absolute gain on commonsense-reasoning story completion and 5.7 points on the RACE reading-comprehension benchmark — evidence that a generic decoder pretrained on raw text could out-compete architectures purpose-built per task.

Then, in October 2018, BERT appeared with bidirectional masked pretraining and outperformed GPT-1 across the standard benchmarks. To most of the field, the causal, left-to-right objective looked like the losing branch of the family tree: why let the model see only leftward context when a masked objective could use both directions? The trade-off between the two objectives is taught in Pretrain, Then Finetune; what matters here is that the causal objective had one property the masked objective lacked. A causal model is a generator, and a generator can be asked to continue any text — including text that describes a task. That property was worth little at 117 million parameters. The next two papers in the lineage were experiments in what it was worth at 13 times, and then 1,500 times, that size.

Three GPT papers in twenty-three months change almost nothing about the architecture. What moves is scale: 117 million parameters, then 1.5 billion, then 175 billion. PNG ↓

The Breakthrough

GPT-2 (Radford et al., 2019), announced in February 2019, reframed the question. Instead of asking how well a pretrained model could be fine-tuned, it asked whether a language model trained on enough diverse text would begin to perform tasks without any fine-tuning at all. The paper’s framing was that a general system should model not a single conditional distribution per task but one distribution conditioned on the task itself — and that natural language can express the task inside the input. A translation demand can be written as text; so can a summarization demand; the string “TL;DR:” is a task specification.

The bet required a dataset broad enough to contain implicit demonstrations of many tasks. The team built WebText by scraping the 45 million outbound links posted on Reddit that had received at least 3 karma — a cheap human-curation signal — yielding just over 8 million documents and 40 GB of text after deduplication and cleaning, with Wikipedia removed to avoid contaminating evaluation sets. The architecture stayed a GPT-1-style decoder with modest changes: layer normalization moved to the input of each sub-block (the now-standard pre-norm arrangement), the context window doubled from 512 to 1024 tokens, and the vocabulary grew to 50,257 tokens of byte-level BPE — a tokenizer choice whose mechanics are the subject of Tokenization. Four sizes were trained: 117M parameters (12 layers — equivalent to the original GPT), 345M (24 layers), 762M (36 layers), and 1,542M (48 layers, 1600-dimensional states).

The result: the 1.5B model set state-of-the-art perplexity on 7 of 8 language-modeling datasets in a zero-shot setting — no gradient step taken on any of them — and reached 63.24% accuracy on the LAMBADA long-range completion task and 70.70% on the Winograd Schema Challenge, a commonsense-reasoning benchmark, a 7-point improvement over the prior state of the art. On every summarization, translation, and question-answering probe, performance improved log-linearly with model size, and the largest model still underfit WebText. That last observation was the loaded gun: capability was climbing with scale and had not yet saturated. OpenAI judged the generation quality concerning enough to withhold the full model at announcement, releasing it in stages — a 774M checkpoint in August 2019 and the full 1.5B model on November 5, 2019.

GPT-3 (Brown et al., 2020), submitted in May 2020, changed almost nothing about the method and everything about the magnitude. The design was informed by the power-law scaling results published by the same broader group four months earlier (Kaplan et al., 2020) — the subject of Scaling Laws. GPT-3 was a 175-billion-parameter decoder: 96 layers, 12,288-dimensional states, 96 attention heads of dimension 128, a 2,048-token context window, trained with a 3.2M-token batch size at a learning rate of 0.6×10⁻⁴. The paper describes it as ten times larger than any previous non-sparse language model. Its training mix drew from a filtered Common Crawl (410B tokens), WebText2 (19B), two book corpora (12B and 55B), and English Wikipedia (3B), sampled non-uniformly for a total of about 300 billion training tokens — the curation questions this raised are taken up in The Data Question.

The headline finding was not a benchmark number but a behavior. Place a few examples of a task inside the prompt — English phrase, French translation, English phrase, French translation, English phrase — and the model completes the pattern, performing the task on the new input with no gradient updates whatsoever. The paper named the settings by the number of demonstrations: zero-shot (a natural-language instruction only), one-shot (one demonstration), and few-shot (typically 10 to 100 demonstrations, as many as fit in the context window). This is in-context learning: task adaptation that happens entirely in the forward pass, in the model’s activations, while the weights stay frozen.

The few-shot numbers made the behavior impossible to dismiss: 86.4% on LAMBADA, 71.2% on TriviaQA open-domain question answering, 100% on two-digit addition and 80.4% on three-digit addition — arithmetic no one trained it to do. On SuperGLUE, few-shot GPT-3 outperformed a fine-tuned BERT-Large on four of eight tasks (while remaining weak on others, such as 49.4% on the WiC word-sense task). And human evaluators asked to distinguish GPT-3-written news articles from human-written ones scored around 52% — barely above chance.

Every completion is this loop executed once per output token; only the width, depth and context length changed between GPT-1 and GPT-3. PNG ↓

The architecture in this loop is essentially unchanged from GPT-1 to GPT-3 — only $N$ , the widths, and the context length grow. The lineage is a controlled experiment in scale.

	GPT-1 (Jun 2018)	GPT-2 (Feb 2019)	GPT-3 (May 2020)
Parameters	~117M	1,542M	175,000M
Layers	12	48	96
Model width	768	1,600	12,288
Context window	512	1,024	2,048
Training data	BooksCorpus	WebText, 40 GB	~300B tokens, filtered web + books + Wikipedia
Task adaptation	Fine-tune weights per task	Zero-shot prompting	Few-shot in-context learning
Release	Paper + weights	Staged over 9 months	API only, no weights

The Mathematics

All three models optimize the same pretraining objective — the causal language-modeling loss, whose chain-rule roots go back to From Shannon to N-grams. In GPT-1’s notation, given an unsupervised corpus of tokens $\mathcal{U} = \{u_1, \ldots, u_n\}$ :

L_1(\mathcal{U}) = \sum_{i} \log P\left(u_i \mid u_{i-k}, \ldots, u_{i-1}; \Theta\right)

Symbol	Meaning
$\mathcal{U}$	The unlabeled corpus, a sequence of tokens $u_1, \ldots, u_n$
$u_i$	The token at position $i$
$k$	The context window — how many previous tokens the model may condition on
$\Theta$	The neural network parameters being trained
$P(\cdot \mid \cdot\,; \Theta)$	The model’s predicted probability of the next token given its context

GPT-1 followed this with a supervised fine-tuning stage per task (mechanics in Pretrain, Then Finetune). GPT-2’s conceptual move was to delete that second stage. A task-specific system estimates $p(\text{output} \mid \text{input})$ ; a general one must condition on the task too:

p(\text{output} \mid \text{input}) \;\longrightarrow\; p(\text{output} \mid \text{input}, \text{task})

GPT-3 made the conditioning concrete. In the few-shot setting, $K$ demonstration pairs are concatenated with the query into a single context, and the model’s ordinary next-token machinery produces the answer:

\hat{y} = \arg\max_{y}\; p_\theta\left(y \mid x_1, y_1, x_2, y_2, \ldots, x_K, y_K, x\right)

Symbol	Meaning
$x_j, y_j$	The $j$ -th demonstration: an example input and its correct output, written as text
$K$	Number of demonstrations in the prompt (GPT-3 used 10–100, bounded by the context window)
$x$	The new query input to be solved
$\hat{y}$	The predicted output — the highest-probability continuation
$p_\theta$	The frozen pretrained language model with parameters $\theta$

Two empirical regularities gave the equation its force. First, few-shot performance improves smoothly as parameters grow — the GPT-3 paper’s aggregate curves show the gap between zero-shot and few-shot widening with scale, which the authors read as larger models being better in-context learners, not just better memorizers. Second, none of this was targeted by the training objective: in-context learning is an emergent phenomenon of scaling the loss in the first equation. Whether such emergence is sharp or an artifact of how we measure it became a live debate later, covered in Scaling Laws.

What It Enabled

The practical consequence of the lineage is a change in who can adapt a language model. In the fine-tuning regime, adapting a model to a task meant a labeled dataset, a training pipeline, GPU time, and a new copy of the weights to serve. In the prompting regime, it means writing text. The cost of trying a new task fell from an ML-engineering project to a few minutes of typing.

	Fine-tuning era (2018–2019)	Prompting era (2020–)
Task specification	Labeled dataset, thousands of examples	Instruction plus 0–100 examples in the prompt
Who adapts the model	ML engineers with training infrastructure	Anyone who can write
Artifacts produced	One checkpoint per task	One shared model, many prompts
Iteration cycle	Hours to days per experiment	Seconds per experiment
Serving	Deploy and version each fine-tuned copy	One deployment serves every task
Failure mode	Overfits small task data	Sensitive to prompt wording and example choice

GPT-1's branch gets more expensive with every new task; GPT-2 and GPT-3's branch gets cheaper, which is why the industry settled on prompting. PNG ↓

The distribution model changed with the interface. OpenAI never released GPT-3’s weights; access opened through a commercial API announced on June 11, 2020, and in September 2020 Microsoft took an exclusive license to the underlying model while everyone else consumed it through the API. The language model became a metered utility, and “prompt engineering” became a job description.

The lineage also set the agenda for the rest of this book. The observation that GPT-2 underfit WebText and that GPT-3’s gains tracked smooth power laws made scaling a quantitative discipline — Scaling Laws covers Kaplan and the Chinchilla correction. Raw in-context learning proved awkward to steer, which motivated Instruction Tuning and RLHF — the pipeline that turned GPT-3’s base capability into the ChatGPT product of November 2022 (ChatGPT and the Open Explosion). And the discovery that what you put in the context changes what the model can do is the seed of chain-of-thought prompting and everything in Prompting Becomes Reasoning.

Key Papers

Paper	Authors	Year	Link	Why it matters
Improving Language Understanding by Generative Pre-Training	Radford et al.	2018	PDF	GPT-1: fixed the decoder-only architecture and the generative-pretraining recipe the whole lineage scales
Language Models are Unsupervised Multitask Learners	Radford et al.	2019	PDF	GPT-2: zero-shot task transfer from web-scale pretraining; showed capability still climbing at 1.5B
Language Models are Few-Shot Learners	Brown et al.	2020	arXiv	GPT-3: 175B parameters; named and demonstrated in-context learning as an emergent ability of scale
Scaling Laws for Neural Language Models	Kaplan et al.	2020	arXiv	The power-law results that justified betting a 100x scale-up on the same recipe

PM Lens

The product lesson I take from GPT-3 is that the benchmark tables were not the breakthrough — the interface was. When adaptation cost dropped from “hire an ML team and label data” to “write a paragraph,” the addressable builder population expanded by orders of magnitude overnight. That is why a closed API with no weights still spawned an ecosystem: copywriting products like Jasper and Copy.ai were essentially prompt libraries wrapped around someone else’s model, and they found paying customers before most enterprises had finished evaluating BERT fine-tuning vendors. The second lesson is distribution strategy: OpenAI metering GPT-3 as an API in June 2020, plus Microsoft’s exclusive license that September, established the model-as-utility business shape the whole industry now assumes. When I evaluate an AI capability today, I ask the GPT-3 question first: does this change what the model can do, or does it change who can make the model do it? The second kind is where markets move.

Search