Preference Optimization Beyond PPO

It is May 2023. InstructGPT’s recipe — a reward model plus PPO — has shipped inside ChatGPT for six months and works, but every lab running it fights the same four-model memory bill and the same brittle RL training loop. Three papers spanning December 2022 to February 2024 attack the problem from different angles: one removes the human from the feedback loop, one removes the reinforcement-learning step entirely, and one removes the critic network that RL still needed.

Where We Were

The RLHF pipeline that produced InstructGPT and ChatGPT is covered in full in RLHF: collect human preference pairs, fit a Bradley-Terry reward model on them, and optimize a policy against that reward model with PPO under a KL penalty toward a reference model, as described in ChatGPT and the Open Explosion. By early 2023 that pipeline had two acknowledged costs.

The first was infrastructure. A PPO step for RLHF needs four copies of the model resident at once — the policy being trained, the frozen reference policy for the KL term, the reward model scoring generations, and a value network (critic) estimating expected future reward so the advantage can be computed at each token. Four models of the same scale as the base LLM meant four times the memory footprint before a single gradient was applied, and PPO’s on-policy sampling-then-update loop was notoriously sensitive to hyperparameters and prone to instability. Getting it to converge reliably took reward normalization, advantage whitening, and small learning rates — craft that varied from lab to lab and rarely made it into a published methods section. Fitting all of this on the accelerators of the day, and keeping the training loop stable, was its own specialty.

The second cost was human labeling. Every preference pair used to train the reward model required a person to read two model outputs and pick the better one — for helpfulness that scales adequately, but for harmlessness it does not: to teach a model to refuse or redirect a harmful request safely, someone has to first write or approve the harmful request, and do it at the volume RLHF requires. Anthropic’s own earlier human-feedback work on helpfulness and harmlessness, part of the lineage covered in RLHF, had also shown that human-labeled harmlessness data tended to make models evasive: the easiest way to score well on harmlessness labels was to refuse everything, which made models less helpful without making them more genuinely safe.

Three separable problems — where do preference labels come from, do you need reinforcement learning at all, and do you need a critic — got three separable answers, in that order, across fourteen months.

The Breakthrough

Constitutional AI (Bai et al., 2022), posted in December 2022, attacked the labeling problem first, and specifically for harmlessness. The method runs in two stages. In the supervised stage (SL-CAI), the model samples a response to a prompt, then critiques its own response against a short written list of principles — the “constitution” — and revises it accordingly; the model is then fine-tuned on the revised responses, bootstrapping a better starting policy without any human harm-labeling. In the reinforcement stage (RL-CAI), the model itself generates the preference labels: it samples pairs of responses and judges which one better satisfies the constitution, and those AI-generated preference pairs — not human ones — train the reward model that RL then optimizes against. This second stage is what the paper calls reinforcement learning from AI feedback (RLAIF): everywhere Bradley-Terry preference data had come from a human contractor, it now comes from a language model applying a written rubric. The result the paper highlights is behavioral, not just a benchmark number: Constitutional AI models were harmless and non-evasive — they engaged with a harmful-seeming query by explaining an objection to it rather than refusing outright, which plain RLHF-on-human-harm-labels models tended not to do.

Direct Preference Optimization (Rafailov et al., 2023), submitted in May 2023, attacked the second problem — does preference tuning need reinforcement learning at all — and answered no. The paper’s move is algebraic rather than architectural: under the same KL-regularized reward-maximization objective RLHF already optimizes, the optimal policy has a closed-form relationship to the reward function and the reference policy. Inverting that relationship lets you substitute policy ratios everywhere a reward model’s score would have appeared in the Bradley-Terry preference likelihood. The consequence is that you can maximize the likelihood of the observed human preferences directly as a classification loss on the policy’s own log-probabilities — no reward model is ever fit as a separate network, no PPO rollouts are sampled, no critic is trained. DPO reduces the four-model RLHF pipeline to two: the policy being trained and a frozen reference copy for the same KL anchor RLHF always used.

Group Relative Policy Optimization (Shao et al., 2024), introduced inside the DeepSeekMath paper in February 2024, attacked the third problem from the opposite direction: keep the reinforcement-learning step — useful when reward comes from an automatic verifier (a math answer checker, a unit test) rather than a fixed preference dataset — but remove the critic. GRPO samples a group of $G$ outputs per prompt from the current policy, scores each with the reward function, and estimates each output’s advantage relative to the mean and spread of its own group’s rewards, rather than from a learned value network. The group itself supplies the baseline that PPO’s critic used to supply.

The Mathematics

DPO’s derivation starts from the same preference model RLHF uses, the Bradley-Terry model over a pair of completions $y_1, y_2$ for a prompt $x$, with a latent reward $r^*$:

$$p^*(y_1 \succ y_2 \mid x) = \frac{\exp\big(r^*(x, y_1)\big)}{\exp\big(r^*(x, y_1)\big) + \exp\big(r^*(x, y_2)\big)}$$

Symbol	Meaning
$x$	The prompt
$y_1, y_2$	The two completions being compared
$r^*(x, y)$	The true, unobserved reward of completion $y$ given prompt $x$
$p^*(y_1 \succ y_2 \mid x)$	The true probability that a human prefers $y_1$ over $y_2$

The same KL-constrained objective RLHF maximizes — expected reward under the policy $\pi_\theta$, penalized for KL divergence from a reference policy $\pi_{\mathrm{ref}}$ by strength $\beta$ — has a known closed-form optimum:

$$\pi_r(y \mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y \mid x)\, \exp\!\left(\frac{1}{\beta} r(x, y)\right)$$

Inverting that relationship for $r$ and substituting into the Bradley-Terry likelihood cancels the intractable $Z(x)$ term (it appears in both the numerator and denominator and divides out), leaving a loss written purely in terms of policy log-probabilities on the preferred completion $y_w$ and the dispreferred one $y_l$:

$$\mathcal{L}_{\mathrm{DPO}}(\pi_\theta; \pi_{\mathrm{ref}}) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right)\right]$$

Symbol	Meaning
$x$	The prompt
$y_w, y_l$	The preferred (“winning”) and dispreferred (“losing”) completions in a human-labeled pair
$\pi_\theta$	The policy being trained
$\pi_{\mathrm{ref}}$	The frozen reference policy, typically the supervised fine-tuned model DPO starts from
$\beta$	The strength of the KL penalty toward the reference policy — the same role it plays in PPO-based RLHF
$\sigma$	The logistic sigmoid function
$Z(x)$	The reward model’s normalizing partition function — present in the derivation, absent from the final loss

The loss admits one more reading worth making explicit. The quantity $\beta \log\big(\pi_\theta(y \mid x) / \pi_{\mathrm{ref}}(y \mid x)\big)$ inside it is exactly the reward the closed-form solution above assigns to a completion, once the arbitrary normalizing constant is fixed by convention — which is the sense in which, as the paper’s title puts it, the language model is secretly a reward model, with no separate network needed to read that reward back out. The loss’s gradient with respect to $\theta$ weights each preference pair by how confidently the current policy still gets the comparison wrong: pairs it already ranks correctly by a wide margin contribute little to the update, while pairs it still ranks incorrectly, or only barely correctly, dominate it. That is the same “attend more to your current mistakes” behavior a Bradley-Terry reward model exhibits during ordinary RLHF training — DPO reproduces it without training a Bradley-Terry model at all.

The DPO paper’s headline empirical result was that this loss, with no RL loop, exceeded PPO’s win rate on TL;DR summarization (approximately 61% versus 57% at temperature 0) and matched a 128-sample best-of-N baseline on Anthropic’s HH dialogue task at a fraction of the inference cost.

GRPO keeps the RL loop but removes the critic. For a prompt $q$, it samples a group of $G$ outputs $\{o_1, \ldots, o_G\}$ from the old policy $\pi_{\theta_{\mathrm{old}}}$, scores each with a reward function to get $\{r_1, \ldots, r_G\}$, and computes each output’s advantage by normalizing within the group:

$$\hat{A}_{i} = \frac{r_i - \mathrm{mean}(r_1, \ldots, r_G)}{\mathrm{std}(r_1, \ldots, r_G)}$$

That per-token advantage then plugs into a PPO-shaped clipped objective with an explicit KL penalty term:

$$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \min\!\left(\frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{\mathrm{old}}}(o_i \mid q)}\hat{A}_i,\ \mathrm{clip}\!\left(\frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{\mathrm{old}}}(o_i \mid q)}, 1-\epsilon, 1+\epsilon\right)\hat{A}_i\right) - \beta\, \mathbb{D}_{\mathrm{KL}}\!\left[\pi_\theta \,\|\, \pi_{\mathrm{ref}}\right]\right]$$

Symbol	Meaning
$q$	The prompt, sampled from the training distribution
$G$	The group size — DeepSeekMath sampled 64 outputs per question
$o_i$	The $i$-th sampled output in the group
$\pi_{\theta_{\mathrm{old}}}$	The policy version used to generate the group, before this update
$\hat{A}_i$	The group-relative advantage of output $o_i$
$\epsilon$	The PPO clipping range, bounding how far a single update can move the policy ratio
$\beta$	The KL penalty strength toward the reference policy

What It Enabled

The DeepSeekMath paper trained a 7B model, DeepSeekMath-RL, with GRPO on top of math-focused pretraining and reported 88.2% on GSM8K and 51.7% on MATH under chain-of-thought evaluation — the strongest results among open 7B-scale models at the time, achieved on a model far smaller than the closed frontier systems it was compared against.

	PPO-based RLHF	DPO	GRPO
Models resident during training	4 (policy, reference, reward, critic)	2 (policy, reference)	3 (policy, reference, reward)
Reward source	Learned reward model from human pairs	Implicit — derived from policy log-probs, no network	Rule-based verifier or learned reward model
Needs online sampling/rollouts	Yes	No — trains on a static preference dataset	Yes — samples a group per prompt each step
Needs a critic/value network	Yes	No	No
Typical use case	General preference alignment	General preference alignment, offline	Verifiable-reward domains: math, code

Each method trades away a different resource, and none is a free lunch. DPO’s static dataset cannot react to a policy that has drifted since the preferences were collected — there is no step in training where the current policy generates a fresh completion to be judged, the way an RL loop provides. GRPO’s group sampling means every optimizer step costs $G$ times the inference compute of a single completion, which pays for itself only when the reward is cheap to compute at scale — a rule-based verifier is nearly free per call, a learned reward model considerably less so. And Constitutional AI’s RLAIF pattern trades human labeling cost for a dependency on the judging model’s own calibration: an AI judge inherits the blind spots of whatever produced it, so the two-stage bootstrap helps only to the extent that the self-critique step catches errors the base model would otherwise repeat uncorrected.

GRPO’s group-relative baseline made it a natural fit wherever reward can be computed automatically rather than learned from human labels — a math answer is either right or wrong, a unit test either passes or fails. That property is why GRPO became the RL algorithm behind DeepSeek-R1 (2025), covered in Reasoning Models: once reward comes from a verifier instead of a preference model, the whole pipeline this chapter describes — critic-free RL with a group baseline — is the training loop underneath “reasoning” models that spend more tokens at inference time.

DPO’s simplicity — a supervised-style loss on a static dataset — made it the default for open-weight preference tuning through 2023 and 2024; Constitutional AI’s RLAIF pattern — using a model, guided by a written rubric, to generate the preference labels another model trains on — generalized past harmlessness into a broader habit of substituting model judgment for human labeling wherever labeling a preference is easier than solving the underlying task, a pattern this book returns to whenever a chapter discusses model-as-judge evaluation.

Key Papers

Paper	Authors	Year	Link	Why it matters
Constitutional AI: Harmlessness from AI Feedback	Bai et al.	2022	arXiv	Introduced RLAIF — AI-generated preference labels replacing human harm-labeling, plus self-critique and revision
Direct Preference Optimization: Your Language Model is Secretly a Reward Model	Rafailov et al.	2023	arXiv	Closed-form reward-to-policy substitution; removes the RL loop and the separate reward model from preference tuning
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models	Shao et al.	2024	arXiv	Introduced GRPO — a critic-free, group-baseline PPO variant that became the RL algorithm behind DeepSeek-R1

PM Lens