skip to content

Search
← All concepts

#rlhf

Preference-based alignment (RLHF, DPO, GRPO).

2 chapters, oldest era first: one idea traced across the books.

  1. The Language Machines 2017–2022 Ch. 13: RLHF Bradley–Terry reward models and the PPO objective with a KL penalty — InstructGPT.
  2. The Language Machines 2023–2024 Ch. 15: Preference Optimization Beyond PPO DPO's closed-form trick, Constitutional AI and RLAIF, and GRPO.