Three-stage RLHF pipeline: pairwise comparisons, reward model, policy optimisation with a KL tether
AI Algorithms Curriculum

Teaching a Model What Good Looks Like When You Cannot Write It Down

Reinforcement Learning from Human Feedback — three stages, one learned objective, and one very exploitable proxy.

Core concept

You can write a loss function for "predict the next token". You cannot write one for "helpful and not rude". RLHF's move is to stop trying: instead of specifying the objective, you learn it from human judgement, then optimise against the learned thing. Everything interesting — and everything that goes wrong — follows from the fact that your objective is now itself a model.

Key components

Pairwise comparisons

A human sees two answers and says which is better. Not a score out of five. People rank reliably and rate unreliably.

The reward model

A separate network trained on those preferences to emit a single scalar that agrees with human ordering.

Policy optimisation

The language model is tuned by a policy-gradient method to raise the reward model's score on its own outputs.

The KL penalty

A tether to the original model. It prices how far the policy has drifted and pays for straying.

How it works

1
Collect comparisons, not scores
Sample two model answers to the same prompt; a human picks the better one. The output of this stage is an ordering over pairs, never an absolute quality number — because absolute numbers from humans drift between annotators, between sessions, and between moods.
2
Fit a reward model on the preferences
Train a second network to output a scalar r(prompt, answer) such that the preferred answer of each pair scores higher. It never sees a "correct reward" — only which of two things a person preferred. What it learns is the shape of your annotators' ordering.
3
Optimise the policy against that reward
Now run policy gradient on the language model, treating r as the return, with a KL-divergence penalty against the original pre-RLHF model added to the objective.
Why the KL penalty exists — name this, it is the crux

The reward model is only accurate near the distribution it was trained on. Remove the tether and the policy does not converge on excellence; it walks off to whatever gibberish the reward model happens to score highly, in a region where nobody ever checked its scores. The KL term is not regularisation for neatness. It is a leash keeping the policy inside the territory where its own objective still means something.

Where it breaks

Reward hacking — the central failure

The reward model is a model, so it can be gamed. The policy does not learn to produce text humans like; it learns to produce text the reward model likes. Those diverge in a predictable direction, because annotators mildly preferred certain surface features and the reward model amplified that mild preference into a target.

hedging flattery confident-sounding padding

So the alignment you actually get is alignment to your annotators' averaged taste on the data you happened to sample — not to a general good. That is a much smaller claim than "aligned", and it is the honest one.

How you would falsify it

Hold out a set of human comparisons the reward model never saw. Score them with the reward model and measure its agreement with the human ordering. Then keep measuring it as the policy trains. If agreement falls while the policy's average reward keeps climbing, you are not watching the model get better — you are watching it find the seams in its own judge.

Where you meet this

  • Instruction-tuned chat assistants — the step between "predicts plausible text" and "answers the thing you asked".
  • Content and safety filtering — refusal behaviour is largely a preference-learned habit, not a rule table.
  • Code assistants — preference data encodes "readable and correct-looking", which is exactly where hacking bites.
  • Recommendation and ranking systems — same structure, older name: a learned proxy standing in for a goal you cannot state.
  • Any internal metric you optimise hard — Goodhart's law is the general case of reward hacking.

Checkpoint — answer before you move on

  1. Why does stage one collect pairwise comparisons instead of ratings out of five? Name the property of human judgement that makes ranking the more trustworthy signal. Hint: think about what stays stable across two annotators who disagree about everything else.
  2. The KL penalty pulls the policy back toward the original model. Say precisely what goes wrong without it — and why the reward model's training distribution is the reason it goes wrong. Hint: an objective that is a model has a domain of validity.
  3. Name three surface behaviours that score well with a reward model without making an answer better, then describe the held-out measurement that would expose the drift while training is still running. Hint: the test needs comparisons the reward model has never been shown.