Instruction Tuning

It is September 2021. GPT-3 has shown that a large enough language model can perform new tasks from a handful of examples pasted into its prompt, but ask it a question directly — zero examples, plain English — and it often continues your text instead of answering it. A Google paper called FLAN proposes a fix that costs less than 2% of pretraining compute: fine-tune the model on dozens of existing NLP datasets, each rewritten as a natural-language instruction, and it learns to follow instructions it has never seen.

Where We Were

The field had two paradigms for getting work out of a pretrained model, and both had a gap in the middle. The first was pretrain-then-finetune: take BERT or T5, fine-tune on a labeled dataset, and get one specialist model per task. It worked, but every new task needed its own training run and its own labeled data. The second was in-context learning, introduced with GPT-3 in May 2020: no gradient updates at all, just demonstrations in the prompt. It generalized, but few-shot prompts consumed context, required careful engineering, and zero-shot performance — the setting closest to how a person actually asks for things — lagged well behind.

The gap had a simple diagnosis. A pretrained language model is trained to continue text, not to satisfy requests. “Summarize this article:” is, to a next-token predictor, just a string that plausibly precedes more strings — sometimes a summary, sometimes another instruction, sometimes a list of similar homework problems. T5 had already unified every task into text-to-text form in October 2019, but its task prefixes were terse codes like “translate English to German:”, learned per task, not natural language a user would type.

An early measurement of the middle path came in April 2021. Natural Instructions (Mishra et al., 2021) collected 61 distinct tasks with their human-authored crowdsourcing instructions and 193k task instances, and showed that models fine-tuned with instructions generalized to unseen tasks 19% better than models trained without them — evidence that instructions themselves, not just task data, carried transferable signal. What was missing was scale, on both axes: bigger models and broader task mixtures.

The Breakthrough

FLAN: instructions as the training signal. “Finetuned Language Models Are Zero-Shot Learners” (Wei et al., 2021), submitted in September 2021, named the technique instruction tuning and gave it its canonical experiment. The team took LaMDA-PT, a dense decoder-only transformer of 137 billion parameters, and aggregated 62 publicly available text datasets into 12 task clusters — translation, sentiment, natural language inference, summarization, and so on. For each dataset they manually wrote ten unique templates that described the task in natural language (“Is this movie review positive or negative?”), plus up to three inverted variants that turned the task around (from sentiment classification to review generation, say) to keep the model from overfitting to a single format.

The evaluation protocol is what made the claim clean. To test zero-shot performance on natural language inference, they instruction-tuned on every cluster except natural language inference — the model had never seen the task type, only the habit of following instructions. The resulting model, FLAN, beat zero-shot GPT-3 175B on 20 of 25 evaluated datasets, and beat even few-shot GPT-3 on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. The cost was almost an accounting error: 30k gradient steps with a batch of 8,192 tokens, about 60 hours on a 128-core TPUv3 — under 2% of the pretraining step count.

The ablations mattered as much as the headline. Performance improved as more task clusters were added. Instructions themselves were load-bearing — fine-tuning on the same data without the natural-language templates generalized far worse. And the benefit was scale-gated: on models of 8B parameters and smaller, instruction tuning actually hurt held-out performance. The paper’s explanation was capacity: “for small-scale models, learning the ∼40 tasks used during instruction tuning fills the entire model capacity, causing these models to perform worse on new tasks.” Only around the 100B scale did the technique reliably pay off — one of the clearest early examples of a scale-dependent behavior of the kind chapter 10 catalogued.

T0: the same idea, 16x smaller. One month later, “Multitask Prompted Training Enables Zero-Shot Task Generalization” (Sanh et al., 2021) — a BigScience collaboration submitted in October 2021 — reached the same destination from a different direction and undercut the scale requirement. T0 was an 11-billion-parameter encoder-decoder initialized from T5+LM, fine-tuned on a mixture built from P3, the Public Pool of Prompts: 2,073 community-written prompt templates covering 177 datasets, contributed by 36 people across 24 institutions. Four task types were held out entirely — natural language inference, coreference resolution, sentence completion, and word sense disambiguation. On these fully unseen tasks T0 matched or exceeded GPT-3, “often outperforming models up to 16x its size.” Where FLAN found instruction tuning only worked at ~100B on a decoder-only model, T0 showed that with an encoder-decoder base and heavy prompt diversity, 11B sufficed. The disagreement was productive: it established that the recipe’s ingredients — base objective, template variety, task count, model scale — could be traded against each other.

Scaling the recipe. The next year turned the recipe into an axis to be scaled like any other. Super-NaturalInstructions (Wang et al., 2022), submitted in April 2022, grew the benchmark to 1,616 tasks across 76 task types with expert-written instructions, and its Tk-Instruct model outperformed InstructGPT by over 9% on the benchmark despite being an order of magnitude smaller. “Scaling Instruction-Finetuned Language Models” (Chung et al., 2022), submitted in October 2022, pushed both axes at once — 1.8K tasks, models up to 540B — and folded in chain-of-thought training data. Flan-PaLM 540B improved on base PaLM by 9.4% on average across its evaluation suite and reached 75.2% on five-shot MMLU; the released Flan-T5 checkpoints made strong instruction-following available at sizes that ran on a single GPU.

Self-Instruct: the model writes its own curriculum. By late 2022 the binding constraint was no longer method but data. Every mixture above was built from academic NLP datasets — classification, extraction, inference — which cover a narrow slice of what people actually ask a model to do, and writing new instruction data by hand is slow and expensive. “Self-Instruct: Aligning Language Models with Self-Generated Instructions” (Wang et al., 2022), submitted in December 2022, closed the loop: use the pretrained model itself to generate the instruction data. Starting from just 175 human-written seed tasks, the pipeline prompts the model with 8 in-context examples (6 human-written, 2 model-generated) to write new instructions, classifies each as a classification or open-ended task, generates matching input-output instances, filters near-duplicates, and recycles survivors into the pool. The run produced roughly 52K instructions with 82K instances. Fine-tuning vanilla GPT-3 on this self-generated data yielded a 33% absolute improvement on Super-NaturalInstructions — approaching the performance of InstructGPT-001, which was trained with private human-written data.

The Mathematics

Instruction tuning changes no architecture and no loss function — it changes what the familiar next-token objective is computed over. Each training example is a source instance $x$ with target $y$, drawn from task $t$ and rendered through an instruction template $I_t$; the model is trained to produce the target given the instruction-wrapped input:

$$\mathcal{L}(\theta) = -\,\mathbb{E}_{t \sim p(t)}\; \mathbb{E}_{(x,y) \sim D_t} \sum_{k=1}^{|y|} \log p_\theta\!\left(y_k \mid I_t(x),\, y_{<k}\right)$$

Symbol	Meaning
$\theta$	Model parameters being fine-tuned
$t,\ p(t)$	A task and the mixture distribution over tasks
$D_t$	The labeled dataset for task $t$
$I_t(x)$	Input $x$ rendered through one of task $t$‘s instruction templates
$y_k,\ y_{<k}$	The $k$-th target token and the target tokens before it

The mixture $p(t)$ is where the engineering lives. Datasets range from hundreds of examples to millions, and sampling proportionally would let the giants drown the rest. FLAN adopted T5’s examples-proportional mixing with a hard cap — it limited training examples per dataset to 30k and used a mixing-rate maximum of 3,000, so that “a dataset does not receive additional sampling weight for examples in excess of 3,000”:

$$p(t) = \frac{\min\left(|D_t|,\, K\right)}{\sum_{t'} \min\left(|D_{t'}|,\, K\right)}$$

Symbol	Meaning
$p(t)$	Probability of drawing the next example from task $t$
$\lvert D_t \rvert$	Number of training examples in task $t$‘s dataset
$K$	Mixing-rate cap, 3,000 in FLAN
$t'$	Index running over all tasks in the mixture

Self-Instruct adds one more piece of machinery: a diversity filter on the generated pool. A candidate instruction $I_{\text{new}}$ joins the task pool $\mathcal{P}$ only if it is sufficiently unlike everything already there:

$$\max_{I \in \mathcal{P}} \ \mathrm{ROUGEL}\left(I_{\text{new}},\, I\right) < 0.7$$

What It Enabled

The immediate effect was a new column in every model comparison: not “which tasks was it fine-tuned on?” but “does it follow instructions?” The before/after is stark when laid out:

	Pretrain-then-finetune (2018–2020)	Prompted GPT-3 (2020–2021)	Instruction-tuned (2021–)
Models per task	One specialist each	One generalist	One generalist
New task cost	Labeled data + training run	Prompt engineering + demonstrations	A plain-English request
Zero-shot usability	None — wrong output space	Weak, format-sensitive	Strong on unseen tasks
Extra training compute	Per task, repeatedly	Zero	Under 2% of pretraining, once
Data requirement	Thousands of labels per task	Handful of examples per prompt	One shared task mixture

The lineage of instruction-tuned models through 2023 traces the recipe’s three scaling knobs — tasks, model size, and data source:

Model	Date	Base	Params	Instruction data	Headline result
FLAN (Wei et al., 2021)	Sep 2021	LaMDA-PT	137B	62 datasets, 12 clusters, hand templates	Beats zero-shot GPT-3 on 20 of 25 tasks
T0 (Sanh et al., 2021)	Oct 2021	T5+LM	11B	P3: 2,073 prompts, 177 datasets	Outperforms models up to 16x its size
Tk-Instruct (Wang et al., 2022)	Apr 2022	T5	11B	1,616 tasks, expert instructions	Beats InstructGPT by 9%+ on Super-NI
Flan-PaLM (Chung et al., 2022)	Oct 2022	PaLM	540B	1.8K tasks incl. chain-of-thought	75.2% five-shot MMLU
Self-Instruct GPT-3 (Wang et al., 2022)	Dec 2022	GPT-3	175B	52K self-generated instructions	+33 points absolute on Super-NI
Alpaca (Taori et al., 2023)	Mar 2023	LLaMA	7B	52K demos from text-davinci-003	Rough parity with its teacher, under 600 dollars

Three downstream threads matter most. First, instruction tuning became the first stage of the modern alignment pipeline. InstructGPT, submitted in March 2022, begins with exactly this kind of supervised fine-tuning on instruction-response demonstrations before adding a reward model and reinforcement learning — the subject of chapter 13 on RLHF — and that pipeline is what shipped as ChatGPT in November 2022 (chapter 14). In retrospect, FLAN and T0 built the layer that makes a language model usable; RLHF built the layer that makes it preferable.

Second, Self-Instruct made instruction data cheap, which made capable chat models reproducible. When Meta released LLaMA in February 2023, Stanford’s Alpaca (March 2023) fine-tuned the 7B model on 52K instruction-following demonstrations “generated in the style of self-instruct using text-davinci-003” — under 500 dollars of API calls for the data, about 3 hours on 8 A100s for the fine-tune, under 600 dollars total, though restricted to non-commercial use. In a blind pairwise evaluation Alpaca won 90 comparisons to its teacher’s 89. A capability that had been a frontier-lab exclusive in early 2022 became a weekend project, igniting the open-model explosion covered in chapter 14. The same move — training on a stronger model’s outputs — is a form of knowledge distillation, and synthetic instruction data became a load-bearing part of the answer to the data question.

Third, the task-mixture mindset persisted even after chat data replaced academic datasets: every modern post-training run is still a weighted mixture of capabilities — reasoning, coding, safety, multilinguality — governed by exactly the capped-sampling arithmetic above.

Key Papers

Paper	Authors	Year	Link	Why it matters
Cross-Task Generalization via Natural Language Crowdsourcing Instructions	Mishra et al.	2021	arXiv:2104.08773	Natural Instructions: 61 tasks, first measured evidence that instructions transfer across tasks
Finetuned Language Models Are Zero-Shot Learners	Wei et al.	2021	arXiv:2109.01652	FLAN: named instruction tuning; 137B model beats zero-shot GPT-3 on 20 of 25 tasks
Multitask Prompted Training Enables Zero-Shot Task Generalization	Sanh et al.	2021	arXiv:2110.08207	T0: 11B encoder-decoder with 2,073 community prompts matches models 16x larger
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks	Wang et al.	2022	arXiv:2204.07705	Scaled the benchmark to 1,616 tasks; Tk-Instruct beat InstructGPT at a tenth the size
Scaling Instruction-Finetuned Language Models	Chung et al.	2022	arXiv:2210.11416	Flan-PaLM: 1.8K tasks plus chain-of-thought data; 75.2% MMLU; released Flan-T5
Self-Instruct: Aligning Language Models with Self-Generated Instructions	Wang et al.	2022	arXiv:2212.10560	Bootstrapped 52K instructions from 175 seeds; made instruction data nearly free
Alpaca: A Strong, Replicable Instruction-Following Model	Taori et al.	2023	Stanford CRFM	Self-Instruct-style data on LLaMA 7B for under 600 dollars; opened the floodgates

PM Lens