RLHF

It is March 2022. Instruction tuning has taught language models to recognize a request as a request, but recognizing the shape of a good answer is not the same as producing one — labeled demonstrations show a model one acceptable response, not which of several plausible responses a person would actually prefer. A paper out of OpenAI folds a five-year-old reinforcement-learning technique from Atari and robotics into the fine-tuning pipeline, and the resulting model — 1.3 billion parameters, 100 times smaller than GPT-3 — is preferred by human raters over its giant, un-tuned ancestor.

Where We Were

Supervised fine-tuning on instructions, as chapter 12 covered, solved a real problem: a pretrained model predicts plausible continuations of text, not answers to requests, and demonstration data teaches it the difference. But demonstrations have a structural limit. For a task like “translate this sentence” there is close to one correct target, and imitating it works. For a task like “write a helpful, honest response to this question” there is a whole distribution of acceptable answers that differ in tone, length, caution, and framing, and a single labeler-written demonstration captures only one point in that space. Worse, a model trained purely to imitate demonstrations has no signal about how much better one plausible answer is than another, or about the many small ways an answer can be subtly unhelpful, evasive, or wrong without being flatly incorrect.

Reinforcement learning is the standard machine-learning tool for optimizing a scalar objective that isn’t a differentiable per-example label — but it requires a reward function, and nobody had a formula for “how aligned with human intent is this sentence.” The reward had to come from people, and people are slow, expensive, and inconsistent at giving numbers. What they are comparatively good at is comparisons: given two candidate outputs, saying which one is better is faster and more reliable than scoring either one in isolation on an absolute scale. The problem that had to be solved before any of this reached language models was a control problem — could an RL agent be trained at all from a trickle of pairwise human judgments instead of a hand-written reward function?

That question was answered outside language modeling entirely, in classic reinforcement-learning testbeds — Atari games and simulated robot locomotion — three years before it reached text.

The Breakthrough

Christiano 2017: preferences as the reward signal. “Deep Reinforcement Learning from Human Preferences” (Christiano et al., 2017), submitted in June 2017, showed that an RL agent could learn complex behaviors from a human comparing short video clips of its own trajectories, with no access to the environment’s true reward function at all. The loop had three parts: the current policy generates pairs of short trajectory segments, a human watches both and says which one better exhibits the desired behavior (a MuJoCo hopper doing a backflip, an Atari agent making progress at a level), and a separate reward-predictor network is trained to output higher scores for the preferred segment — effectively learning a numeric surrogate for “goodness” from ordinal judgments alone. The policy is then updated with a standard RL algorithm against this learned reward, and the loop repeats, with new trajectories queried for fresh comparisons as the policy improves. The result that mattered for everything downstream was data efficiency: the method solved tasks using feedback on less than 1% of the agent’s interactions with the environment, and produced novel behaviors — including the backflip, which no one had specified as a goal — from about an hour of human comparison time.

PPO: the optimizer everyone standardized on. One month later, “Proximal Policy Optimization Algorithms” (Schulman et al., 2017), submitted in July 2017, was not framed around human feedback at all — it was a general-purpose policy-gradient algorithm, positioned as a simpler, easier-to-tune alternative to trust region policy optimization (TRPO) that kept TRPO’s stability with plain first-order optimization. Its central device, a clipped surrogate objective, discourages any single update from moving the policy too far from where it collected data, which keeps training stable across many small gradient steps on the same batch. PPO said nothing about where the reward came from; that turned out to be exactly why it became the default RL algorithm to pair with a learned, preference-trained reward model. Reward from a neural network trained on human comparisons is noisier and less stationary than a hand-coded game score, and PPO’s conservative, clip-bounded updates tolerate that noise better than more aggressive policy-gradient methods.

Learning to Summarize: the recipe meets language. For three years the preference-learning loop stayed in control tasks. “Learning to Summarize from Human Feedback” (Stiennon et al., 2020), submitted in September 2020, ported it to open-ended text generation, using a curated version of Reddit’s TL;DR dataset as the task. The pipeline: start from a model already fine-tuned on demonstrations (supervised summaries), have it produce pairs of summaries for the same post, collect human judgments of which summary is better, train a reward model on 64,832 such comparisons, and fine-tune the summarizer against that reward model with PPO, adding a penalty for straying too far — in KL divergence — from the starting policy. Results at 1.3B and 6.7B parameters showed the value of the approach starkly: their 1.3B human-feedback-trained model was preferred over the dataset’s own human-written reference summaries 61% of the time, while a 13B model trained with supervised learning alone — ten times the parameters, no human-feedback stage — was preferred only 43% of the time. Feedback-trained summaries also transferred to CNN/DailyMail news articles without any domain-specific fine-tuning, approaching reference-summary quality on a task the model had never been tuned for. The paper established, for the first time in language generation, that a smaller model optimized against human preferences could beat a larger model optimized against a fixed dataset.

InstructGPT: the three-stage pipeline at scale. “Training Language Models to Follow Instructions with Human Feedback” (Ouyang et al., 2022), submitted in March 2022, generalized the summarization recipe from one task to arbitrary instructions, and turned it into the pipeline the field now calls RLHF. About 40 contractors, hired through Upwork and ScaleAI, produced two kinds of data on real API prompts: roughly 13,000 demonstrations of good responses (for a supervised fine-tuning stage) and roughly 33,000 prompts on which labelers ranked 4 to 9 model outputs from best to worst (for a reward model trained on all $\binom{K}{2}$ pairwise comparisons implied by each ranking). The reward model was fixed at 6 billion parameters even for the 175B policy — the paper reports that training a 175B reward model was unstable and not worth the compute. A further 31,000 prompts, drawn only from the API with no human labels, were used for the RL stage itself: the SFT model was fine-tuned with PPO against the reward model, with a per-token KL penalty against the SFT policy to keep the RL model from drifting into reward-model blind spots, and — in a variant the paper calls PPO-ptx — an added pretraining-data loss term to fix an “alignment tax,” the tendency of RLHF fine-tuning to regress performance on standard NLP benchmarks relative to the base model.

The headline comparison was the one that made the paper impossible to ignore: in human evaluations, outputs from the 1.3B-parameter InstructGPT model were preferred over outputs from the 175B GPT-3, despite having 100 times fewer parameters, and at matched 175B scale InstructGPT’s outputs beat GPT-3’s roughly 85±3% of the time. The gains were not limited to a vague notion of “preference”: InstructGPT generated truthful, informative answers on TruthfulQA about twice as often as GPT-3, produced roughly 25% fewer toxic outputs when prompted to be respectful, and hallucinated on closed-domain summarization tasks 21% of the time versus 41% for GPT-3. Size stopped being the only lever that mattered; what the model had been optimized against mattered as much as how large it was.

The Mathematics

The pipeline has three components with three different learning objectives layered on top of one another: a reward model learned from comparisons, a policy-gradient optimizer that uses that reward, and a penalty that keeps the optimized policy tethered to where it started.

The reward model is trained on pairs, not absolute scores. Given a prompt $x$ and two candidate responses, where $y_w$ is the one a human preferred and $y_l$ is the one they didn’t, the model learns a scalar function $r_\theta(x,y)$ such that higher-scoring responses are the ones people pick more often:

$$\mathcal{L}_{\text{RM}}(\theta) = -\frac{1}{\binom{K}{2}}\; \mathbb{E}_{(x,\,y_w,\,y_l)\sim D}\Big[\log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)\Big]$$

Symbol	Meaning
$\theta$	Reward model parameters
$x$	The prompt shared by both candidate responses
$y_w,\ y_l$	The response humans preferred, and the one they didn’t
$r_\theta(x,y)$	The reward model’s scalar score for response $y$ to prompt $x$
$\sigma$	The logistic sigmoid function
$K$	Number of responses ranked per prompt; $\binom{K}{2}$ pairwise comparisons come from one ranking

Once a reward model exists, the policy is optimized against it with PPO. The core device is a clipped surrogate objective that bounds how far one gradient step is allowed to move the policy relative to where it was when the training data was collected:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(r_t(\theta)\,\hat{A}_t,\ \ \mathrm{clip}\big(r_t(\theta),\,1-\epsilon,\,1+\epsilon\big)\,\hat{A}_t\big)\Big]$$

Symbol	Meaning
$\theta$	Policy parameters being optimized
$r_t(\theta)$	Ratio of new to old policy probability for the action taken at step $t$
$\hat{A}_t$	Estimated advantage of the action taken at step $t$
$\epsilon$	Clip range hyperparameter, 0.2 in the original PPO experiments
$\mathrm{clip}(\cdot)$	Truncates its argument to the interval $[1-\epsilon,\,1+\epsilon]$

InstructGPT combines this optimizer with a reward that is not just the reward model’s score but that score minus a penalty for drifting from the supervised policy, plus — in the PPO-ptx variant — a term that keeps pretraining-style language modeling alive during RL fine-tuning:

$$\mathrm{objective}(\phi) = \mathbb{E}_{(x,y)\sim D_{\pi^{\mathrm{RL}}_\phi}}\Big[r_\theta(x,y) - \beta \log\frac{\pi^{\mathrm{RL}}_\phi(y\mid x)}{\pi^{\mathrm{SFT}}(y\mid x)}\Big] \;+\; \gamma\, \mathbb{E}_{x\sim D_{\text{pretrain}}}\big[\log \pi^{\mathrm{RL}}_\phi(x)\big]$$

Symbol	Meaning
$\phi$	RL policy parameters
$\pi^{\mathrm{RL}}_\phi,\ \pi^{\mathrm{SFT}}$	The policy being trained, and the fixed supervised-fine-tuned policy it started from
$r_\theta(x,y)$	Reward model score for response $y$
$\beta$	KL-penalty coefficient
$\gamma$	Pretraining-loss mixing coefficient, zero for plain PPO, nonzero for PPO-ptx
$D_{\text{pretrain}}$	The original language-model pretraining distribution

What It Enabled

RLHF changed what “improving a language model” meant. Before it, progress mostly meant more data, more parameters, or better instruction-tuning mixtures — all upstream of what the model would actually say. After it, a model’s behavior could be shaped directly against a signal built from the exact axis a product team cared about, whatever that axis was: helpfulness, harmlessness, brevity, refusal behavior, or house style.

	Supervised demonstrations only (pre-2020)	RLHF pipeline (2020–2022)
Training signal	Imitate one written example per prompt	Optimize against a learned model of relative preference
Captures “better, not just correct”	No — one demonstration per prompt	Yes — comparisons directly encode degree
Smaller model can beat a larger one	Rarely	Yes — 1.3B beat 175B GPT-3 in preference
Handles no-single-right-answer tasks	Poorly (dialogue, open-ended writing)	Directly, by design
Side effects to manage	Overfitting to demonstration style	Reward hacking, alignment tax, “blind spot” exploitation
Compute/memory footprint	One model	Policy, reference policy, reward model, and value function concurrently

The lineage from Christiano’s control-task preference learning to a shipped chat product took five years and four papers:

Model / paper	Date	Scale	Headline result
Preference RL (Christiano et al., 2017)	Jun 2017	Atari, MuJoCo	Novel behaviors from under 1% feedback coverage, ~1 hour of human time
PPO (Schulman et al., 2017)	Jul 2017	General RL	Clipped objective becomes the default optimizer for learned-reward RL
Summarization RLHF (Stiennon et al., 2020)	Sep 2020	1.3B–6.7B	1.3B model preferred over human references 61% vs. 43% for a 13B supervised model
InstructGPT (Ouyang et al., 2022)	Mar 2022	1.3B–175B	1.3B RLHF model preferred over 175B GPT-3; ~85% win rate at matched scale

InstructGPT’s pipeline shipped, largely intact, as ChatGPT in November 2022 (chapter 14) — the moment RLHF stopped being a research result and became the default expectation for what a “finished” language model behaves like. But the pipeline this chapter describes is also the expensive version: a reward model trained once, then a policy optimized against it through a live RL loop that keeps four models — the policy, a frozen reference copy, the reward model, and a value function — resident at the same time, with all the instability that on-policy RL brings (reward hacking, sensitivity to the KL coefficient, extra hyperparameters). Those costs are exactly what motivated the next wave of preference-optimization methods — Constitutional AI’s RLAIF, DPO, and GRPO — which keep the comparison-based training signal but remove the online RL loop; that redesign is chapter 15. The reward-model-plus-RL structure also generalized past human preference entirely: reasoning models trained with reinforcement learning against verifiable rewards — a unit test passing, a math answer matching — rather than a learned preference model, are chapter 20, and the resulting shift of compute from training-time to inference-time is priced out in engine-room’s test-time-compute economics.

Key Papers

Paper	Authors	Year	Link	Why it matters
Deep Reinforcement Learning from Human Preferences	Christiano et al.	2017	arXiv:1706.03741	Proved an RL policy could be trained from pairwise human comparisons alone, on control tasks
Proximal Policy Optimization Algorithms	Schulman et al.	2017	arXiv:1707.06347	Clipped surrogate objective became the default optimizer for learned-reward RL, including RLHF
Learning to Summarize from Human Feedback	Stiennon et al.	2020	arXiv:2009.01325	First RLHF result in language generation; 1.3B model beat a 13B supervised model in preference
Training Language Models to Follow Instructions with Human Feedback	Ouyang et al.	2022	arXiv:2203.02155	InstructGPT: named the three-stage RLHF pipeline; 1.3B beat 175B GPT-3 in human preference
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons	Bradley and Terry	1952	DOI	The pairwise-comparison model whose loss function underlies every RLHF reward model

PM Lens