The Data Question

It is January 2021. GPT-3 has shown what 175 billion parameters can do, and a volunteer research collective called EleutherAI publishes an 825 GiB text corpus with full documentation of every component — the first time a pretraining dataset arrives as a designed artifact rather than an appendix. Over the next four years, the question “what should a language model read?” moves from footnote to the most closely guarded secret in the industry.

Where We Were

Through 2019 and 2020, pretraining data was the least examined ingredient of the recipe. Papers reported architectures in diagrams and hyperparameters in tables, then dispatched the corpus in a paragraph. GPT-2 (February 2019) trained on web pages selected using social-media engagement as a quality proxy — a heuristic, not a methodology — and the GPT lineage treated data as something you gathered once and scaled past. The most systematic effort of the era came from T5 (Raffel et al., 2019), whose Colossal Clean Crawled Corpus (C4) — introduced in October 2019 alongside the text-to-text framework — applied a stack of hand-written cleaning heuristics to Common Crawl, the nonprofit web archive that would become the raw ore for nearly every corpus that followed. GPT-3 (Brown et al., 2020), submitted in May 2020, trained its 175-billion-parameter model on a weighted mix dominated by filtered Common Crawl, with the filtering itself described only briefly.

Two forces ended this casualness. The first was arithmetic. The Kaplan scaling laws of January 2020 had told labs to grow parameters much faster than data, which made corpora feel abundant; the Chinchilla correction (Hoffmann et al., 2022) of March 2022 — the subject of chapter 10 — reversed the prescription, requiring tokens to double every time parameters did. Hoffmann et al. projected that a compute-optimal one-trillion-parameter model would need roughly 21.2 trillion training tokens and noted that this was “far beyond what is currently used to train large models.” If the law held, the binding constraint on the next generation of models was not chips but text.

The second force was evidence that the text already in use was in poor condition: duplicated, contaminated with test sets, and stuffed with boilerplate. Nobody had looked closely, because looking closely at hundreds of gigabytes of text is an engineering project in its own right. Between 2021 and 2024, a handful of papers did the looking — and data curation became a discipline with its own measurements, its own algorithms, and its own scaling laws.

The Breakthrough

The breakthrough of this chapter is not a single method but a change of stance, executed in four movements: treat the corpus as a designed object, clean it with algorithms rather than heuristics, publish the pipeline so it can be ablated like an architecture, and finally ask whether the right data can substitute for scale itself.

Movement one: curation. The Pile (Gao et al., 2021), released by EleutherAI in January 2021, assembled 825 GiB of English text from 22 diverse subsets — many drawn from academic and professional sources rather than the open web — on the explicit thesis that “increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability.” Models trained on the Pile outperformed those trained on raw Common Crawl and CC-100 across its component evaluations. The deeper contribution was procedural: every subset was named, sized, licensed, and justified in the paper. A corpus was, for the first time, something you could review.

Movement two: hygiene. “Deduplicating Training Data Makes Language Models Better” (Lee et al., 2021), submitted in July 2021 by a Google team, examined the standard corpora and found them saturated with repetition — including a single 61-word English sentence repeated more than 60,000 times inside C4. The consequences were measurable on the model side: over 1% of the unprompted output of language models trained on these datasets was copied verbatim from training data, and after deduplication — combining exact substring matching with approximate near-duplicate detection — models emitted memorized text ten times less frequently while needing fewer training steps to reach the same or better accuracy. The paper also quantified a quieter problem: train–test overlap affected more than 4% of the validation examples in standard benchmarks, meaning some measured “generalization” was recall. That finding seeded the contamination anxiety that now shadows every leaderboard, taken up in measuring minds.

Movement three: the open pipeline. For two years the strongest corpora stayed proprietary. RefinedWeb (Penedo et al., 2023), submitted in June 2023 by the Falcon LLM team, made a pointed claim: “properly filtered and deduplicated web data alone can lead to powerful models,” even surpassing models trained on curated collections like the Pile. The team extracted five trillion tokens from Common Crawl through aggressive filtering and deduplication, and publicly released 600 billion of them. A year later the same first author, now at Hugging Face, went further. FineWeb (Penedo et al., 2024), submitted in June 2024, processed 96 Common Crawl snapshots into a 15-trillion-token corpus and — the real contribution — documented and ablated every design choice, including “in-depth investigations of deduplication and filtering strategies,” releasing the curation codebase and every ablation model. The motivating complaint was stated plainly: the pretraining datasets for state-of-the-art open models “are not publicly available and very little is known about how they were created.” Alongside it came FineWeb-Edu, a 1.3-trillion-token collection of educational text filtered from FineWeb; models pretrained on it “exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC” — a superset beaten by a subset more than ten times smaller.

Movement four: quality as a substitute for scale. “Textbooks Are All You Need” (Gunasekar et al., 2023), submitted by Microsoft Research in June 2023, pushed the logic to its limit. Phi-1 is a 1.3-billion-parameter code model trained for four days on eight A100s using just 7 billion tokens: 6 billion of “textbook quality” data filtered from the web, plus roughly 1 billion tokens of textbooks and exercises synthesized by GPT-3.5. It reached 50.6% pass@1 on HumanEval and 55.5% on MBPP — competitive with far larger code models — and a 350-million-parameter sibling trained on the same pipeline still achieved 45% on HumanEval. Two ideas graduated at once: model-based quality filtering (use a model to decide what a model should read) and deliberate synthetic data (use a strong model to write the curriculum for a small one), a lineage that connects forward to bootstrapped instruction data and to distillation.

By 2024 the pipeline these movements produced had become the industry’s standard shape:

The Mathematics

The workhorse algorithm of corpus hygiene is near-duplicate detection at web scale. Comparing every document against every other is quadratic and impossible at billions of documents, so pipelines like those of Lee et al. and FineWeb rely on MinHash, which turns similarity into a probability. Represent each document as its set of n-grams (shingles); the similarity of two documents $A$ and $B$ is their Jaccard index:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

MinHash replaces each document’s shingle set with a short signature. Apply a random permutation $\pi$ to the universe of shingles and record only the minimum value it assigns within each document. The classic identity is that a single such hash collides exactly as often as the documents are similar:

$$\Pr\left[\min \pi(A) = \min \pi(B)\right] = J(A, B)$$

Symbol	Meaning
$A, B$	The two documents, represented as sets of n-gram shingles
$\pi$	A random permutation of the shingle universe, acting as one hash function
$\min \pi(\cdot)$	The smallest hash value the permutation assigns within a document
$J(A,B)$	The Jaccard similarity of the two shingle sets

Signatures still need pairing up, so pipelines add locality-sensitive hashing: split each signature into $b$ bands of $r$ hashes and mark two documents as candidate duplicates if any band matches exactly. For a pair with true similarity $s$, the probability of becoming a candidate is:

$$P(s) = 1 - \left(1 - s^{r}\right)^{b}$$

Symbol	Meaning
$s$	True Jaccard similarity of a document pair
$r$	Number of hash values per band
$b$	Number of bands in the signature
$P(s)$	Probability the pair is flagged as a candidate duplicate

The second mathematical question of the era was what happens when clean data runs out. “Scaling Data-Constrained Language Models” (Muennighoff et al., 2023), submitted in May 2023, extended the Chinchilla framework to repeated data and found that “up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.” Beyond that, repetition decays in value, captured by an effective-data term:

$$D' = U_D + U_D \, R_D^{*} \left(1 - e^{-R_D / R_D^{*}}\right)$$

Symbol	Meaning
$D'$	Effective dataset size in fresh-token equivalents
$U_D$	Number of unique tokens available
$R_D$	Number of repetitions beyond the first pass
$R_D^{*}$	Fitted decay constant, approximately 15.4

Token counts throughout this chapter are counts of subword units, whose construction is the subject of chapter 8.

What It Enabled

The most visible product of the era is the corpora themselves, whose evolution tells the story in one table:

Corpus	Date	Scale	Source	Curation strategy
C4 (Raffel et al., 2019)	Oct 2019	—	Common Crawl	Hand-written cleaning heuristics
The Pile (Gao et al., 2021)	Jan 2021	825 GiB	22 curated subsets	Diversity through deliberate curation
RefinedWeb (Penedo et al., 2023)	Jun 2023	5T tokens extracted, 600B released	Common Crawl only	Strict filtering plus deduplication
phi-1 corpus (Gunasekar et al., 2023)	Jun 2023	7B tokens	Filtered web plus GPT-3.5 output	Textbook quality, partly synthetic
FineWeb (Penedo et al., 2024)	Jun 2024	15T tokens	96 Common Crawl snapshots	Fully ablated, fully published pipeline
FineWeb-Edu (Penedo et al., 2024)	Jun 2024	1.3T tokens	Filtered from FineWeb	Educational-quality filtering

Three consequences followed. First, an open-model ecosystem became possible. The Pile fed the early open training runs, and FineWeb-scale corpora put frontier-scale token counts within reach of anyone with the compute — the open releases chronicled in ChatGPT and the open explosion and the engineering of such runs in training runs in practice both stand on this substrate. Second, and in tension with the first, the frontier labs stopped talking. FineWeb’s authors noted that the pretraining data of leading models remained undisclosed; as architectures converged on the transformer, the corpus — its sources, its filters, its licensing deals — became the residual differentiator. The chapter’s title question got its answer: data is the moat precisely because it is the one input that cannot be read off a config file.

Third, the field split its response to the coming data wall. One camp repeats and stretches: the Muennighoff result licenses a few epochs of reuse, buying roughly a four-fold extension of any corpus. Another camp synthesizes: the phi-1 recipe — strong model writes, small model reads — grew into a standard tool, and its close cousin, generating instruction-following data from a model’s own outputs, is covered in instruction tuning. A third camp routes around pretraining entirely, retrieving knowledge at inference time instead of memorizing it at training time — the retrieval systems of chapter 17. And hanging over all of it is the contamination problem: once benchmarks leak into web-scale corpora, evaluation itself needs auditing, the concern that opens chapter 22.

Key Papers

Paper	Authors	Year	Link	Why it matters
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	Raffel et al.	2019	arXiv:1910.10683	Introduced C4, the heuristically cleaned Common Crawl corpus the field then studied and fixed
Language Models are Few-Shot Learners	Brown et al.	2020	arXiv:2005.14165	GPT-3’s filtered-Common-Crawl mix set the pre-curation-era baseline
The Pile: An 800GB Dataset of Diverse Text for Language Modeling	Gao et al.	2021	arXiv:2101.00027	First pretraining corpus built and documented as a designed artifact; 22 subsets, 825 GiB
Deduplicating Training Data Makes Language Models Better	Lee et al.	2021	arXiv:2107.06499	Measured duplication, memorization, and train–test contamination; made dedup standard practice
Training Compute-Optimal Large Language Models	Hoffmann et al.	2022	arXiv:2203.15556	Chinchilla’s token requirement turned data into the binding constraint
Scaling Data-Constrained Language Models	Muennighoff et al.	2023	arXiv:2305.16264	Scaling laws for repeated data; about four epochs of reuse come nearly free
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only	Penedo et al.	2023	arXiv:2306.01116	Showed filtered, deduplicated web data alone can beat curated collections
Textbooks Are All You Need	Gunasekar et al.	2023	arXiv:2306.11644	Phi-1: 1.3B parameters on 7B textbook-quality tokens rivaling much larger code models
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale	Penedo et al.	2024	arXiv:2406.17557	15T-token open corpus with fully ablated, published pipeline; FineWeb-Edu proved quality filtering wins

PM Lens