One Model, Many Tasks

It is October 2019, and two papers arrive on arXiv six days apart. T5 (Raffel et al., 2019), submitted on October 23, proposes that every text task — translation, summarization, classification, even regression — be cast as the same problem: text in, text out. BART (Lewis et al., 2019) follows on October 29 with a complementary idea: pretrain one sequence-to-sequence model to reconstruct deliberately corrupted text, then point it at any task. Together they close out the pretrain-then-finetune era’s central question — what should one general-purpose model look like? — just before scale rewrites the question entirely.

Where We Were

By late 2019 the transfer-learning recipe was firmly established: pretrain a Transformer on unlabeled text, then fine-tune it on each downstream task. Chapter 6 covers how BERT’s masked bidirectional pretraining (October 2018) made that recipe dominant, and it owns the masked-versus-causal distinction this chapter builds on.

But the recipe had an awkward seam. BERT was an encoder: it produced contextual representations, not text. To use it, practitioners bolted a different task-specific head onto the model for every job — a classification layer for sentiment, a span-prediction layer for extractive question answering, a pairwise scorer for entailment. Ten tasks meant ten architectures, ten output formats, ten fine-tuning pipelines. And because BERT had no decoder, it could not naturally generate — summarization and translation, the tasks that most obviously require producing text, sat outside its comfort zone. On the other side of the divide, GPT-2 (February 2019) generated fluently but read strictly left to right, giving up the bidirectional context that made BERT strong on understanding tasks; Chapter 7 tells that story.

Spring 2019 produced two notable bridge attempts. MASS (Song et al., 2019), submitted in May 2019, pretrained an encoder-decoder by masking a consecutive fragment of the input sentence and training the decoder to predict exactly that fragment — representation learning and generation trained jointly. UniLM (Dong et al., 2019), submitted the same month, kept a single shared Transformer and switched between unidirectional, bidirectional, and sequence-to-sequence behavior purely by changing self-attention masks. Both showed that the understanding/generation divide was a design choice, not a law. What neither delivered was a systematic answer to the field’s accumulated confusion: a rapidly growing list of pretraining objectives, architectures, corpora, and fine-tuning schemes, most evaluated under incomparable conditions.

The Breakthrough

T5 — the Text-to-Text Transfer Transformer — made two contributions that are worth keeping separate. The first was a framing. Every task the model would ever see was expressed as a text string in and a text string out. Translation became the input translate English to German: That is good. with the German sentence as the target. Natural-language inference became mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity. with the target being the literal string naming the label. No task heads, no per-task output layers — one encoder-decoder Transformer, one maximum-likelihood objective, one decoding procedure, for everything. The task itself was specified by a short text prefix, a small ancestor of what would later be called prompting.

A detail that is easy to miss: in this framing even the labels are text. A sentiment classifier does not emit a probability over two classes; it literally writes the word naming the class, drawn from the same subword vocabulary as everything else — prefixes, sentinels, and answers alike. (How such vocabularies are built is Chapter 8’s territory.) This looks wasteful — the model could in principle output an invalid string — but in practice it rarely does, and the uniformity is what makes the whole scheme composable.

The second contribution was the study. Holding compute and data comparable, Raffel et al. systematically varied pretraining objectives, architectures (encoder-decoder, decoder-only language model, prefix language model), unlabeled corpora, and fine-tuning strategies across dozens of benchmarks. Each baseline configuration was pretrained on a deliberately modest budget of $2^{35} \approx 34$ billion tokens — less than BERT’s 137 billion or RoBERTa’s 2.2 trillion — so that many variants could be compared under identical conditions before anything was scaled. To feed the study they built C4, the Colossal Clean Crawled Corpus — about 750 GB of filtered text derived from an April 2019 Common Crawl snapshot. The headline findings: an encoder-decoder architecture paired with a denoising objective beat decoder-only language modeling across their tested tasks, and among denoising variants, dropping random spans (with a corruption rate of 15 percent and an average span length of 3 tokens) worked as well as more elaborate schemes while producing shorter, cheaper training targets. In span corruption, each dropped span is replaced in the input by a unique sentinel token, and the target consists of the sentinels followed by the recovered spans — the model never re-generates the uncorrupted text.

Insights in hand, they scaled. The final T5 models came in five sizes — from the roughly 220-million-parameter Base configuration up to 11 billion parameters — each pretrained on 1 trillion tokens of C4. T5-11B reached 88.9 on the SuperGLUE benchmark, close to the human baseline of 89.8, along with state-of-the-art results across summarization, question answering, and classification (Raffel et al., 2019).

BART approached the same destination from the denoising-autoencoder direction. Its architecture is the plain sequence-to-sequence Transformer — a bidirectional encoder feeding a left-to-right decoder, which the authors describe as generalizing both BERT and GPT — with roughly 10 percent more parameters than an equivalently sized BERT. The idea is to corrupt a document with an arbitrary noise function and train the model to reconstruct the original. Because the noise function is unconstrained, BART could test what kind of damage makes the best teacher:

Transformation	What it does	In the final model?
Token masking	Random tokens replaced with a mask element	No
Token deletion	Random tokens removed; model must find missing positions	No
Text infilling	Sampled spans replaced by a single mask token; span length must be inferred	Yes
Sentence permutation	Sentences shuffled into random order	Yes
Document rotation	Document rotated to begin at a random token	No

Text infilling combined with sentence permutation won. Trained at scale — 160 GB of news, books, stories, and web text (the same data as RoBERTa), batch size 8,000 for 500,000 steps — BART matched RoBERTa on GLUE and SQuAD while setting new state-of-the-art results on generation: 44.16 ROUGE-1 on CNN/DailyMail summarization, 45.14 ROUGE-1 on the more abstractive XSum (about 6 points over prior work), and 37.96 BLEU on WMT Romanian-English translation, a 1.1 BLEU gain over a back-translation baseline (Lewis et al., 2019).

The two papers differ in emphasis — T5 unified the task interface, BART generalized the pretraining corruption — but they land on the same object: a single pretrained encoder-decoder that reads with full bidirectional context and writes autoregressively, fine-tuned identically for any task.

The Mathematics

Both models train and fine-tune with one loss: standard maximum likelihood on the target sequence, decomposed autoregressively. (The factorization itself is covered in Chapter 1; the encoder-decoder attention that conditions each step on the input is Chapter 5’s subject.)

$$\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log p_\theta\left(y_t \mid y_{<t},\, x\right)$$

Symbol	Meaning
$x$	Input token sequence, including any task prefix
$y_t$	The $t$-th token of the target sequence
$y_{<t}$	All target tokens before position $t$
$T$	Length of the target sequence
$\theta$	All model parameters
$p_\theta$	The model’s predicted next-token distribution

What differs between pretraining schemes is only which $x$ and $y$ the corpus manufactures. BART states the general recipe: sample a document, corrupt it with a noise function, and ask the model to reconstruct the original.

$$\mathcal{L}_{\text{denoise}}(\theta) = -\,\mathbb{E}_{x \sim \mathcal{D}}\left[\log p_\theta\left(x \mid c(x)\right)\right]$$

Symbol	Meaning
$x$	An uncorrupted document from the corpus
$\mathcal{D}$	The pretraining corpus
$c(x)$	The corrupted version of $x$ produced by the noise function
$\theta$, $p_\theta$	Model parameters and predicted distribution, as above

T5’s span corruption is a compute-motivated instance of the same idea. With corruption rate $r$ and average span length $\mu$, the number of corrupted spans in a length-$L$ input is

$$n_{\text{spans}} = \frac{r \cdot L}{\mu}$$

Symbol	Meaning
$n_{\text{spans}}$	Number of corrupted spans per training example
$r$	Fraction of input tokens corrupted (T5 uses 0.15)
$L$	Length of the input sequence in tokens
$\mu$	Average span length (T5 uses 3)

What It Enabled

The immediate effect was practical: for the first time, one pretrained checkpoint with one output interface could be fine-tuned on essentially any text task without architectural surgery.

Model	Architecture	Pretraining objective	Task interface	Can generate text?
BERT (2018)	Encoder only	Masked token prediction	New head per task	No
GPT-2 (2019)	Decoder only	Next-token prediction	Prompt continuation	Yes
T5 (2019)	Encoder-decoder	Span corruption	Text in, text out with task prefix	Yes
BART (2019)	Encoder-decoder	Document denoising	Text in, text out	Yes

The lineage runs in two directions. Directly: mT5 (Xue et al., 2020) extended the recipe to a Common Crawl-derived corpus covering 101 languages in October 2020 (Xue et al., 2020), and BART became the workhorse fine-tuning base for abstractive summarization systems. C4 itself outlived the model that created it, becoming a standard pretraining corpus and a case study in Chapter 11’s data question. T5’s encoder found a second life outside language entirely: Imagen (May 2022) used a frozen T5 text encoder to condition an image-generation model, a story told in The Great Image Generation Book, Chapter 14. More consequentially, the text-to-text framing is the substrate on which instruction tuning was built: FLAN (September 2021) and its successors, covered in Chapter 12, are precisely “many tasks, one text interface” with natural-language instructions replacing T5’s terse prefixes — and the strongest early instruction-tuned open checkpoints, the Flan-T5 family (Chung et al., 2022), were T5 models, competitive in few-shot evaluation with far larger models such as PaLM 62B (Chung et al., 2022).

And yet the architecture itself lost. T5’s own controlled study said encoder-decoder beat decoder-only — so why is nearly every frontier model since 2020 a decoder-only Transformer?

Seven months after T5, GPT-3 (Brown et al., 2020) — a 175-billion-parameter decoder-only model, ten times larger than any previous dense language model — showed that at sufficient scale, tasks could be specified in-context, with no gradient updates at all (Brown et al., 2020); Chapter 7 owns that story, and Chapter 10 covers the scaling laws that made betting on scale rational. The comparison was later run carefully: Wang et al. (2022) found that causal decoder-only models trained on plain autoregressive language modeling show the strongest zero-shot generalization after purely unsupervised pretraining, while non-causal masked-objective models pull ahead only after multitask fine-tuning (Wang et al., 2022). Engineering sealed it: a decoder-only model is one uniform stack — simpler to scale, and every pretraining token contributes a prediction loss — and at inference time a growing conversation can reuse its cached computation incrementally rather than re-encoding the input, the serving economics covered in The Engine Room, Chapter 17. The denoising objective survived in spirit; the encoder-decoder body became a niche.

Key Papers

Paper	Authors	Year	Link	Why it matters
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	Raffel et al.	2019	arXiv:1910.10683	The text-to-text framing, the C4 corpus, and the definitive controlled study of the pretrain-finetune design space
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	Lewis et al.	2019	arXiv:1910.13461	Generalized pretraining as document denoising; long the standard fine-tuning base for summarization
MASS: Masked Sequence to Sequence Pre-training for Language Generation	Song et al.	2019	arXiv:1905.02450	Earliest masked encoder-decoder pretraining; direct precursor to T5 and BART
Unified Language Model Pre-training for Natural Language Understanding and Generation	Dong et al.	2019	arXiv:1905.03197	Unified understanding and generation in one Transformer via attention masks alone
Language Models are Few-Shot Learners	Brown et al.	2020	arXiv:2005.14165	GPT-3; moved the task interface from fine-tuning to in-context prompting, deciding the architecture race
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer	Xue et al.	2020	arXiv:2010.11934	Text-to-text recipe extended to 101 languages
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?	Wang et al.	2022	arXiv:2204.05832	The controlled post-mortem: causal decoder-only wins zero-shot; masked non-causal wins after multitask finetuning
Scaling Instruction-Finetuned Language Models	Chung et al.	2022	arXiv:2210.11416	Flan-T5: instruction tuning built directly on the text-to-text substrate

PM Lens