Pretrain, Then Fine-tune

It is January 2018. The Transformer is seven months old, yet nearly every natural-language system in production is still trained from scratch, one labeled dataset at a time. Over the next ten months, four papers — ULMFiT, ELMo, GPT-1, and BERT — converge on a single recipe that ends this regime: pretrain a language model once on unlabeled text, then fine-tune it cheaply for each task.

Where We Were

Through 2017, transfer learning in NLP effectively stopped at the first layer of the network. Pretrained word vectors — word2vec (2013) and GloVe (2014), covered in chapter 2 — gave each word a single fixed vector, learned once from a large corpus and pasted into the bottom of whatever task-specific model came next. Everything above that embedding layer — the LSTM encoders, the attention mechanisms, the classifier heads — was initialized randomly and trained only on the task’s own labeled data.

This arrangement had two structural problems. First, a static vector cannot represent polysemy: “bank” received the same embedding in “river bank” and “bank deposit,” no matter how much context surrounded it. The embedding was learned from co-occurrence statistics across every sense of the word at once, so the representation handed to the task model was an average of meanings rather than the meaning in context. Second, labeled data was the binding constraint. Datasets like the Stanford Sentiment Treebank or SQuAD contained tens or hundreds of thousands of examples — enough to train a small task model, nowhere near enough to learn language itself. Computer vision had already escaped the analogous trap: since 2012, vision practitioners routinely initialized from ImageNet-pretrained networks and fine-tuned. NLP had no equivalent of ImageNet pretraining, and researchers knew it.

The ingredients, however, were accumulating. Recurrent sequence models had shown that neural networks could model text; the Transformer (June 2017) had removed recurrence as a bottleneck; and in April 2018 the GLUE benchmark (Wang et al., 2018) arrived — nine natural-language-understanding tasks under one leaderboard — giving the field a shared yardstick just in time to measure what happened next.

The Breakthrough

The breakthrough was not a single model but a shared recipe, discovered nearly simultaneously by four groups: use language modeling — predicting words from other words — as an unsupervised pretraining objective on a large corpus, then transfer the entire network, not just its first layer, to downstream tasks.

ULMFiT (Howard & Ruder, January 2018) made the transfer-learning claim explicit in its title: Universal Language Model Fine-tuning (Howard & Ruder, 2018). The base model was deliberately unglamorous — an AWD-LSTM, a regular LSTM with tuned dropout — pretrained on WikiText-103 (28,595 Wikipedia articles, 103 million words). The contribution was a three-stage procedure: general-domain language-model pretraining, target-task language-model fine-tuning, then classifier fine-tuning, stabilized by three techniques (discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing) designed to prevent the pretrained knowledge from being catastrophically overwritten. On six text-classification tasks, ULMFiT reduced error by 18–24% over the prior state of the art, and with only 100 labeled examples it matched the performance of training from scratch on 100 times more data.

ELMo (Peters et al., February 2018) attacked the polysemy problem directly (Peters et al., 2018). Instead of one vector per word type, ELMo — Embeddings from Language Models — produced a vector per word occurrence, computed by a deep bidirectional language model (biLM): two LSTMs, one reading left-to-right and one right-to-left, pretrained on a large corpus. Crucially, ELMo exposed the network’s internals: each downstream task learned its own weighted combination of the biLM’s layers, since lower layers capture syntax and higher layers capture context-dependent meaning. Dropped into existing task architectures as features (the biLM itself stayed frozen), ELMo improved the state of the art across six benchmark NLP problems, including question answering, textual entailment, and sentiment analysis.

GPT-1 (Radford et al., June 2018) fused the two ideas with the Transformer (Radford et al., 2018). OpenAI’s “Improving Language Understanding by Generative Pre-Training” trained a 12-layer decoder-only Transformer (768-dimensional states, 12 attention heads, roughly 117 million parameters) as a standard left-to-right language model on BooksCorpus — over 7,000 unpublished books whose long contiguous passages let the model learn long-range dependencies. Like ULMFiT and unlike ELMo, GPT-1 fine-tuned the whole network per task, using task-aware input transformations — reformatting entailment pairs, similarity pairs, and multiple-choice questions into token sequences — so the architecture barely changed. It improved the state of the art on 9 of the 12 tasks studied, including absolute gains of 8.9% on commonsense reasoning (Story Cloze) and 5.7% on question answering (RACE).

BERT (Devlin et al., October 2018) then asked what GPT-1’s left-to-right objective was leaving on the table (Devlin et al., 2018). A causal model sees only leftward context; ELMo’s two LSTMs saw both directions but were trained separately and only shallowly concatenated. A naive deep bidirectional language model is degenerate — with full self-attention, each position could simply look at itself. BERT’s answer was the masked language model (MLM): corrupt the input by hiding 15% of tokens, and train a bidirectional Transformer encoder to reconstruct them from both sides. Of the selected tokens, 80% become a special mask token, 10% become a random token, and 10% stay unchanged — hedging against the mismatch that the mask token never appears at fine-tuning time. A second objective, next sentence prediction (NSP), classified whether sentence B actually followed sentence A. Pretrained on BooksCorpus (800M words) plus English Wikipedia (2,500M words) over a 30,000-token WordPiece vocabulary, BERT shipped in two sizes — Base (12 layers, 110M parameters) and Large (24 layers, 340M parameters) — and set new state of the art on eleven NLP tasks: GLUE to 80.5% (+7.7 absolute), MultiNLI to 86.7%, SQuAD v1.1 F1 to 93.2, SQuAD v2.0 F1 to 83.1.

Fine-tuning itself was deliberately minimal in all of these systems. BERT’s downstream recipe added a single task-specific output layer on top of the pretrained encoder and then updated all parameters end to end for a few epochs on the labeled data; GPT-1 did the same on its decoder, with an auxiliary language-modeling loss to keep the representations from drifting. The pretrained weights did nearly all of the work — which is precisely what made the approach economically different from everything before it. A pretraining run was expensive and happened once; a fine-tuning run was cheap and happened per task, per team, per product.

The fork BERT crystallized — masked versus causal pretraining — became the field’s central architectural decision, and this chapter is where the two objectives part ways.

The Mathematics

All four systems minimize a negative log-likelihood; they differ in what is predicted from what. The causal (autoregressive) objective, used by ULMFiT, ELMo’s two halves, and GPT-1, scores each token given everything to its left — the same chain-rule factorization that underlies all language modeling since Shannon:

$$\mathcal{L}_{\mathrm{CLM}}(\theta) = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})$$

Symbol	Meaning
$\mathcal{L}_{\mathrm{CLM}}$	causal language-modeling loss
$\theta$	all model parameters
$T$	number of tokens in the sequence
$x_t$	the token at position $t$
$x_{<t}$	all tokens before position $t$
$p_\theta(\cdot \mid \cdot)$	the model’s predicted probability

BERT’s masked language model replaces “predict the next token” with “reconstruct hidden tokens from both sides.” A random subset $M$ of positions (15% in BERT) is corrupted, and only those positions are scored:

$$\mathcal{L}_{\mathrm{MLM}}(\theta) = -\,\mathbb{E}_{M}\!\left[\, \sum_{t \in M} \log p_\theta(x_t \mid \tilde{x}) \,\right]$$

Symbol	Meaning
$\mathcal{L}_{\mathrm{MLM}}$	masked language-modeling loss
$M$	the randomly chosen set of masked positions
$\tilde{x}$	the corrupted input sequence
$x_t$	the original token at masked position $t$
$\mathbb{E}_{M}$	average over random maskings
$p_\theta$	the model’s predicted probability

ELMo’s contribution is a formula for using a pretrained language model without fine-tuning it. For token $k$, the downstream task learns a weighted average of the frozen biLM’s layer representations:

$$\mathrm{ELMo}_k = \gamma \sum_{j=0}^{L} s_j \, \mathbf{h}_{k,j}$$

Symbol	Meaning
$\mathrm{ELMo}_k$	the contextual embedding for token $k$
$L$	number of biLM layers
$\mathbf{h}_{k,j}$	layer $j$‘s hidden representation of token $k$
$s_j$	learned task-specific softmax weight for layer $j$
$\gamma$	learned task-specific scale factor

Finally, ULMFiT contributed the discipline of how to fine-tune without destroying pretrained knowledge. Its discriminative fine-tuning assigns each layer its own learning rate, decaying geometrically from top to bottom:

$$\eta^{(l-1)} = \frac{\eta^{(l)}}{2.6}$$

What It Enabled

The recipe rewrote the economics of building NLP systems. Before 2018, each task meant designing an architecture and training it from zero on that task’s labeled data; after, it meant downloading pretrained weights and fine-tuning for a few hours.

	Before (through 2017)	After (2018 onward)
What transfers	First-layer word vectors only	The entire network
Word representation	One static vector per word	Contextual vector per occurrence
Per-task architecture	Custom, hand-designed	Same backbone + small task head
Labeled-data need	Tens of thousands of examples minimum	Hundreds can suffice (ULMFiT: 100 examples ≈ 100× data from scratch)
Unlabeled text	Used only for word vectors	The primary training resource
Evaluation	Fragmented, per-task	Shared multi-task suites (GLUE)

The four 2018 systems themselves spanned a small design space whose axes — backbone, objective, transfer method — defined the next several years of research:

Model	Date	Backbone	Objective	Transfer method	Scale notes
ULMFiT	Jan 2018	AWD-LSTM	Causal LM	Full fine-tuning, staged	WikiText-103, 103M words
ELMo	Feb 2018	2-layer biLSTM ×2	Forward + backward LM	Frozen features, learned layer mix	biLM stays frozen
GPT-1	Jun 2018	12-layer Transformer decoder	Causal LM	Full fine-tuning, input transformations	~117M params, BooksCorpus
BERT	Oct 2018	Transformer encoder	Masked LM + NSP	Full fine-tuning	Base 110M / Large 340M
RoBERTa	Jul 2019	Transformer encoder	Masked LM (NSP dropped)	Full fine-tuning	160GB text, 8K batch

The immediate aftermath ran in two directions. On the encoder side, RoBERTa (Liu et al., July 2019) showed that BERT had been significantly undertrained (Liu et al., 2019): keeping the architecture fixed and changing only the training — dynamic masking instead of static, next-sentence prediction removed, batch size raised from 256 to 8,000 sequences, and pretraining data grown from 16GB to roughly 160GB (adding CC-News, OpenWebText, and Stories) — produced state-of-the-art results on GLUE, RACE, and SQuAD, including 90.2 on MNLI and 89.4 F1 on SQuAD v2.0 (dev). The lesson generalized far beyond BERT: the recipe and the data mattered as much as the architecture, an early empirical hint of the scaling laws formalized in 2020. Encoder models also spawned an efficiency lineage — DistilBERT (October 2019) compressed BERT via knowledge distillation for production serving.

On the decoder side, OpenAI concluded that the causal objective plus scale was the more general bet: GPT-2 (February 2019) and GPT-3 (May 2020) abandoned per-task fine-tuning altogether in favor of prompting and in-context learning — the subject of the next chapter. A third family asked why choose at all: T5 and BART (both October 2019) recast every task as text-to-text with encoder-decoder denoising objectives, covered in chapter 9.

Key Papers

Paper	Authors	Year	Link	Why it matters
Universal Language Model Fine-tuning for Text Classification	Howard & Ruder	2018	arXiv:1801.06146	First explicit ImageNet-style transfer recipe for NLP; fine-tuning techniques that prevent catastrophic forgetting
Deep contextualized word representations	Peters et al.	2018	arXiv:1802.05365	ELMo: context-dependent embeddings from a pretrained biLM; solved polysemy that static vectors could not
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding	Wang et al.	2018	arXiv:1804.07461	The shared yardstick on which the pretraining race was scored
Improving Language Understanding by Generative Pre-Training	Radford et al.	2018	OpenAI report	GPT-1: causal Transformer pretraining + full fine-tuning; 9 of 12 tasks improved; origin of the GPT lineage
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	Devlin et al.	2018	arXiv:1810.04805	Masked language modeling enables deep bidirectionality; new state of the art on eleven tasks
RoBERTa: A Robustly Optimized BERT Pretraining Approach	Liu et al.	2019	arXiv:1907.11692	Showed BERT was undertrained; data and recipe rival architecture — a precursor to scaling-law thinking