ChatGPT and the Open Explosion

It is November 30, 2022. OpenAI ships a free research preview of a chatbot wrapped around an InstructGPT-style model and calls it ChatGPT. The interface is nothing new — a text box — but the adoption curve is: a million users in five days, a hundred million within two months. The ten months that follow turn that single product launch into a race that Meta, a leak on an anonymous imageboard, a Stanford lab, and a Paris startup all run at once, and by the end of it a capable chat model is no longer something only one company can ship.

Where We Were

By late 2022 the ingredients for a capable chat model already existed, just not assembled into a product. OpenAI’s InstructGPT line — supervised fine-tuning on demonstrations followed by reinforcement learning from human feedback — had been described in a March 2022 paper and was quietly available through the API as text-davinci-003. GPT-3 had proven that a large enough transformer could do new tasks from a prompt. Instruction tuning had proven that a small amount of supervised fine-tuning made a base model follow requests instead of merely continuing them. None of it had a front door. Access to a frontier model meant an API key, a waitlist, and a per-token bill; there was no product built for a person who had never written a prompt.

Open alternatives existed but did not close the gap. Meta’s OPT and the BigScience collaboration’s BLOOM, both released in 2022 and covered in training runs in practice, were released with weights, but neither was instruction-tuned for dialogue, and both were trained on token budgets that chapter 10 would later show were badly mismatched to their parameter counts. A hobbyist could download the weights; nothing they downloaded talked back the way OpenAI’s private models did.

Eleven months separate the ChatGPT launch from Mistral 7B, an unrestricted model outperforming one nearly twice its size. PNG ↓

The Breakthrough

The product moment. ChatGPT was not a new model — it was InstructGPT’s recipe, applied to a model in the GPT-3.5 series, given a conversational interface and made free to try. The numbers made the point better than any technical claim could: a million users in five days, a hundred million within two months, by most accounts the fastest-adopted consumer software application to that date. On February 1, 2023, OpenAI introduced ChatGPT Plus, a $20-a-month tier for priority access during peak load. The effect on the rest of the field was immediate and structural: every lab with a language model now had a live demonstration of what a chat product was worth, and none of them had one to ship.

The open answer. Meta’s response arrived three months later. “LLaMA: Open and Efficient Foundation Language Models” (Touvron et al., 2023), released in February 2023, offered a collection of dense transformers from 7B to 65B parameters, trained exclusively on publicly available data — no proprietary corpora, unlike GPT-3 or PaLM. The paper’s central claim inverted the usual pitch: LLaMA-13B, trained on a trillion tokens, outperformed GPT-3’s 175B parameters on most benchmarks, and LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B. On MMLU, LLaMA-65B scored 63.4% against GPT-3’s 43.9%; LLaMA-13B alone reached 46.9%, ahead of a model thirteen times its size. The framing explicitly followed the logic chapter 10 covers in full: given that a model is trained once but served indefinitely, the right target is not the training-compute optimum but the model that is fastest at inference for a given level of capability — smaller, trained on more data than Kaplan-era scaling would have prescribed.

Training the 65B model took 21 days on 2,048 NVIDIA A100 GPUs over 1.4 trillion tokens drawn mostly from English CommonCrawl (67%), C4, GitHub, Wikipedia, books, arXiv, and Stack Exchange — a mixture the data question revisits from the angle of what “publicly available” training data actually contains. Access was gated: researchers had to apply, and Meta retained the weights.

The gate did not hold. On March 3, 2023, an application to Meta’s access form leaked, and within days a torrent of the LLaMA weights was posted to 4chan and mirrored across HuggingFace and GitHub. Meta filed takedown notices; the files were already everywhere. Two things happened almost simultaneously in the weeks that followed. Stanford’s Alpaca — covered in chapter 12 — fine-tuned the leaked 7B checkpoint on 52,000 instruction demonstrations for under $600, turning a base model into something that behaved like a rough ChatGPT for the cost of a used laptop. And on March 10, 2023, developer Georgi Gerganov released llama.cpp, a pure C/C++ reimplementation of LLaMA inference with no Python or GPU dependency, built specifically to test whether the 7B model could run on a MacBook’s CPU. It could. Together, a leaked checkpoint, a $600 fine-tune, and a laptop-friendly inference engine did in a few weeks what no single company had done: put a talking language model within reach of anyone with a personal computer.

Meta's gate held for weeks, not months; a $600 fine-tune and a laptop-friendly inference engine built the ecosystem the leak made possible before any official commercial release existed. PNG ↓

Meta legitimizes it. Having watched an unofficial leak spawn an ecosystem, Meta released “Llama 2: Open Foundation and Fine-Tuned Chat Models” (Touvron et al., 2023) officially on July 18, 2023 — 7B, 13B, and 70B models plus chat variants, this time with weights and a license permitting commercial use (a 34B variant was trained but withheld for lack of safety red-teaming time). Llama 2 doubled context length to 4,096 tokens, trained on 2 trillion tokens, and layered rejection sampling and PPO on top of supervised fine-tuning, adding a technique called Ghost Attention (GAtt) that kept the model consistent with a system prompt across many turns of dialogue — refinements to the same RLHF pipeline chapter 13 covers, and a preview of the preference-optimization alternatives chapter 15 takes up. In blind human evaluation across roughly 4,000 prompts, Llama 2-Chat 70B won 36% of comparisons against ChatGPT with a 31.5% tie rate — not a win, but close enough that “open” and “roughly as good as the thing that started this” stopped being contradictory. The license carried one notable restriction: any company with more than 700 million monthly active users needed Meta’s separate permission to use it, a clause aimed squarely at the largest platforms.

The efficient answer. Mistral AI, a Paris startup founded earlier that year, released Mistral 7B on September 27, 2023 (Jiang et al., 2023) under an unrestricted Apache 2.0 license — no user-count clause, no non-commercial carve-out. At 7.3 billion parameters, it outperformed Llama 2 13B on every benchmark the paper evaluated and approached Llama 1 34B on reasoning, mathematics, and code, at roughly a fifth the parameter count. The efficiency came from two architectural choices layered onto the standard transformer: grouped-query attention, which shares key and value projections across groups of query heads to cut inference memory — the mechanism chapter 17 of the engine room covers in full — and sliding window attention, which bounds how many previous tokens each layer attends to directly. The next section works out what that bound buys, and costs, in practice.

The Mathematics

The training-compute cost of a dense transformer follows a widely used approximation: roughly six floating-point operations per parameter per training token, accounting for one forward and one backward pass.

C \approx 6\,N\,D

Symbol	Meaning
$C$	Approximate total training compute, in FLOPs
$N$	Model parameter count
$D$	Training dataset size, in tokens

Llama 2’s paper discloses exactly the kind of hardware accounting that lets this formula be checked against reality: 184,320 A100-80GB GPU-hours for the 7B model and 1,720,320 GPU-hours for the 70B model, both trained on the same 2-trillion-token corpus. Converting GPU-hours and a GPU’s rated peak throughput into a compute budget, and comparing it against $C \approx 6ND$ , yields an estimate of model FLOPs utilization — how much of the hardware’s theoretical peak the training run actually captured:

\mathrm{MFU} = \frac{C}{H \cdot P_{\mathrm{peak}} \cdot 3600}

Symbol	Meaning
$\mathrm{MFU}$	Model FLOPs utilization, the fraction of peak hardware throughput achieved
$C$	Training compute required, from $C \approx 6ND$
$H$	Total GPU-hours consumed by the run
$P_{\mathrm{peak}}$	A single GPU’s peak throughput, in FLOPs per second

Plugging in Llama 2 70B’s disclosed numbers — $N = 70 \times 10^9$ , $D = 2 \times 10^{12}$ tokens, $H = 1{,}720{,}320$ GPU-hours, and an A100’s rated 312 TFLOP/s in BF16 — gives $C \approx 8.4 \times 10^{23}$ FLOPs against a hardware capacity of roughly $1.93 \times 10^{24}$ FLOPs, an implied utilization near 43%. Running the same arithmetic on the 7B model’s 184,320 GPU-hours lands near 41%. Both figures sit inside the 30–50% range labs typically report for large training runs, which is a reasonable confirmation that the $6ND$ approximation and Meta’s disclosed hardware numbers agree with each other.

Mistral 7B’s sliding window attention bounds direct attention to a fixed span at each layer, but stacking layers lets information travel further than any single layer’s window:

R(l) = \min\!\left(L,\ l \cdot W\right)

Symbol	Meaning
$R(l)$	Effective receptive field at layer $l$ , in tokens
$L$	Actual sequence length
$l$	Layer index, counting from the input
$W$	Sliding window size, 4,096 tokens in Mistral 7B

With 32 decoder layers and a 4,096-token window, Mistral 7B’s theoretical receptive field reaches $32 \times 4{,}096 = 131{,}072$ tokens — far beyond the sequence lengths it was actually trained on — while each layer’s attention computation stays bounded to a 4,096-token span, which is what makes long sequences affordable to serve at all.

What It Enabled

The eleven months between ChatGPT and Mistral 7B changed what “having a language model” meant for anyone who was not OpenAI, Google, or Meta:

	Before ChatGPT, Oct 2022	After Mistral 7B, Sep 2023
Best available model	API-only, waitlisted	Open weights, downloadable
Cost to experiment	Per-token API billing	Free after download, self-hosted
Customization	None — prompting only	Full fine-tuning on your own data
Data privacy	Prompts leave your infrastructure	Runs entirely on hardware you control
Commercial license	N/A, vendor terms of service	Apache 2.0 (Mistral) to 700M-MAU-capped (Llama 2)
Runs on	Datacenter accelerators only	A single consumer GPU, or a laptop CPU

The model lineage from that stretch of 2023 is the direct ancestor of most open-weights models still in use:

Model	Date	Params	License	What it established
ChatGPT	Nov 2022	Undisclosed	Product, API-only	The chat interface as the default way to use an LLM
LLaMA (Touvron et al., 2023)	Feb 2023	7B–65B	Research, gated	Public-data-only training competitive with GPT-3
Alpaca	Mar 2023	7B	Non-commercial	Instruction-tuning a leaked base model for under $600
Llama 2 (Touvron et al., 2023)	Jul 2023	7B–70B	Commercial, 700M-MAU cap	RLHF-tuned chat models with an official commercial license
Mistral 7B (Jiang et al., 2023)	Sep 2023	7B	Apache 2.0	Small-model efficiency via GQA and sliding-window attention

Three threads run forward from here. First, the licensing spectrum this chapter opened — gated research release, MAU-capped commercial license, unrestricted Apache 2.0 — became a permanent axis of competition among open-weights labs, alongside benchmark scores. Second, the efficiency techniques Mistral 7B popularized did not stay static: Mistral AI’s own follow-up, Mixtral 8x7B, replaced the dense feed-forward layers with a sparse mixture of experts in January 2024, and grouped-query attention became close to a default choice, covered alongside the rest of the KV-cache toolbox in inference engines. Third, the RLHF pipeline that Llama 2 refined with rejection sampling and Ghost Attention was itself about to be challenged: within a year, direct preference optimization and related methods, covered in chapter 15, let open developers fine-tune chat behavior without training a separate reward model at all — and the same open-weights base models this chapter describes became the standard testbed for those methods, as they later did for extending context length well past the 4,096 tokens Llama 2 shipped with. Meanwhile the closed frontier that ChatGPT opened kept moving too, toward the multimodal models covered in chapter 18.

None of these four threads stayed where this chapter left them; each becomes its own chapter or its own model family within a year. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Introducing ChatGPT	OpenAI	2022	openai.com	The product launch that made chat the default interface and triggered the race this chapter covers
LLaMA: Open and Efficient Foundation Language Models	Touvron et al.	2023	arXiv:2302.13971	Public-data-only open models competitive with GPT-3; leaked weeks after release and set off the open explosion
Llama 2: Open Foundation and Fine-Tuned Chat Models	Touvron et al.	2023	arXiv:2307.09288	Meta’s official commercial-license release; RLHF with rejection sampling, PPO, and Ghost Attention
Mistral 7B	Jiang et al.	2023	arXiv:2310.06825	Grouped-query and sliding-window attention beat a model nearly twice its size, under an unrestricted license

PM Lens

This is the stretch where “which model” stopped being one question and became three: capability, license, and where it runs. I now build a small comparison table before any build decision, the same way this chapter’s “before/after” table does — Apache 2.0 Mistral-family models go into anything I might want to resell without asking permission; Llama-family models are fine until a partner’s user count gets uncomfortably close to 700 million, which has actually come up in a partnership conversation; ChatGPT Plus at $20 a month remains the fastest way to prototype a prompt before committing to any of the above. The llama.cpp moment is the one I point to when someone claims a capability requires a cloud contract — for a meaningful set of tasks, a quantized 7B model on a laptop is enough, and it is free. What changed permanently in this window is not that open models caught up to closed ones on every benchmark; several still lag. It is that “self-host it” became a real line item next to “call the API,” and every build-versus-buy conversation I run now starts by asking which of those two economics actually fits the product.

Search