Left — RLHF: the frontier-lab standard
- The original alignment recipe, built on reinforcement learning.
- A reward model is trained on human preference rankings.
- PPO then optimizes the policy against that reward signal.
- Complex, expensive, multi-stage pipeline.
- ONLINE — the model can explore new responses while training.
- More powerful in theory: it can discover behaviors your data never showed.
Right — DPO: the loop, deleted
- Skips the reinforcement-learning loop entirely.
- Trains directly on preference pairs: chosen vs rejected.
- A contrastive loss does the work — no reward model, no PPO.
- Simpler and cheaper, with comparable results on most benchmarks.
- OFFLINE — the dataset is fixed, so it cannot discover new behaviors.
- What DPO deletes: the reward model, the PPO loop, and online exploration — at the cost of novelty.
Start here, then test
- Most teams should start with DPO: roughly 90% of the way with a fraction of the effort.
- Reach for RLHF only when the task needs behaviors the preference data cannot supply.
The honest test
- Benchmark on your own evals, not on paper results — comparable on paper, different on your task.
- If DPO lags you, ask where: the gap is usually unseen behaviors, not optimization quality.
- If closing that gap needs the model to explore beyond the dataset, the loop you deleted is the reason — only then does RLHF pay for itself.
Note — image backend outage. The primary image generator
(OpenRouter qwen-image-3) returned an insufficient-credits (402) error, and the
automatic self-hosted fallback produced a dense-text render not safe to ship, so this
card ships text-only to keep the teaching legible. The visual will be re-issued on the
next run once the image backend has credits.