Chapter 20Adaptation and Alignment

A capable model is not yet a useful one

A pretrained language model does exactly one thing, which is to continue text plausibly, because that is what Equation (18.1) asked of it. Ask a base model “What is the capital of France?” and a perfectly good continuation is:

What is the capital of Germany? What is the capital of Italy?

Nothing has gone wrong, because that is a very likely continuation of a list of questions and the model was trained to produce likely continuations. The model knows the answer, so nothing is missing from its capability. What is missing is that nobody ever told it a question should be followed by an answer rather than by another question. Closing that gap is what this chapter is about, and it happens in stages.

Three stages, three different things

Pretraining is enormously expensive and happens once, while the other two are moderate and both are repeated. The stages are not alternatives, because each assumes the one before it and each teaches something the previous one could not.

Instruction tuning

Continue training on pairs of instructions and good responses, where the data is what matters. A few tens of thousands of well-written pairs turn a text continuer into something that answers questions, follows formats and stops when it is done.

Note what has not happened. The capital of France was already there, so what changed is the model’s default behaviour when it sees a question rather than what it knows. A large instruction set can teach facts of its own in passing, and that is a side effect rather than the point of the stage. That is why the stage is cheap relative to pretraining, and why data quality dominates data quantity here.

Preference tuning

Instruction tuning teaches the shape of a response and cannot easily teach which of two well-shaped responses is better. Nobody can write a loss function for helpfulness, and yet people can reliably say which of two answers they prefer, and that comparison is enough.

Reinforcement learning from human feedback (Ouyang et al. 2022) turns those comparisons into a training signal in three steps. Collect pairs of model responses and have people rank them, train a reward model to predict those rankings, then optimise the language model against the reward model with a penalty for drifting too far from where it started.

Turning a comparison into a loss

The reward model gives a response a score $r(x, y)$, and no human ever supplies such a score. What a human supplies is an ordering, and the Bradley-Terry model is what converts one into the other. If a rater preferred $y_w$ over $y_l$, the model’s probability of agreeing is

$$\begin{equation} P(y_w \succ y_l \mid x) \;=\; \sigma\big(r(x, y_w) - r(x, y_l)\big), \label{eq:bradley-terry} \quad\text{(20.1)} \end{equation}$$

so only the difference of the two rewards matters and the scale is free. Training minimises the negative log of Equation (20.1) over the collected pairs. Four pairs make the behaviour concrete, with the rewards a partly trained model might give:

The third row is the one that teaches. There the model prefers the bluff, the pair costs $1.1711$, more than three times the first row, and training pushes $r_w$ up and $r_l$ down until the ordering flips. The fourth row is worth remembering for later, because equal rewards give exactly $0.5$ and a loss of $\log 2 = 0.6931$, which is the most a single comparison can cost when the model has no opinion at all.

Why the penalty is not an afterthought

The policy is then optimised against the reward model rather than against people, and those are different things. The objective keeps it on a leash:

$$\begin{equation} \max_{\pi_\theta} \;\; \mathbb{E}_{y \sim \pi_\theta}\big[r(x, y)\big] \;-\; \beta \, \mathrm{KL}\!\left(\pi_\theta \,\|\, \pi_{\text{ref}}\right), \label{eq:rlhf-objective} \quad\text{(20.2)} \end{equation}$$

where $\pi_{\text{ref}}$ is the instruction-tuned model the policy started from. Take two candidate behaviours, an honest answer that the reward model scores $4.00$ and stays close to the reference, and a hack that scores $5.00$ by exploiting the reward model while drifting far from it:

With $\beta = 0$ the hack wins, because $5.00$ beats $4.00$ and the reward model is the only judge in the room. The ordering flips at $\beta = 0.10$, where the hack’s distance from the starting model finally costs more than the extra reward it buys. That is reward hacking and the penalty is what prevents it.

Too large a $\beta$ is its own failure, since the policy then stays so close to the reference that preference training changes nothing.

Direct preference optimisation

Direct preference optimisation (Rafailov et al. 2023) reaches a similar destination by a shorter road, with no separate reward model and no reinforcement learning loop. Its insight is that a policy trained under Equation (20.2) already encodes a reward, readable straight off its own log-ratio against the reference:

$$\begin{equation} \hat{r}(x, y) \;=\; \beta \left( \log \pi_\theta(y \mid x) - \log \pi_{\text{ref}}(y \mid x) \right). \label{eq:dpo-reward} \quad\text{(20.3)} \end{equation}$$

Substitute that into Equation (20.1) and the reward model disappears, leaving a loss over preference pairs alone:

$$\begin{equation} \mathcal{L}_{\text{DPO}} \;=\; -\log \sigma\big( \hat{r}(x, y_w) - \hat{r}(x, y_l) \big). \label{eq:dpo} \quad\text{(20.4)} \end{equation}$$

With $\beta = 0.1$, three stages of one pair show what the loss is measuring:

At the start the policy is the reference, so both implicit rewards are zero, the margin is zero and the loss is $\log 2 = 0.6931$. That is the same number the undecided pair gave in the Bradley-Terry table above, and for the same reason, which is that the model has no opinion yet.

One step raises the preferred response and lowers the rejected one, and the loss falls. The third row is the warning that $\beta$ exists to price, because pushing the margin further keeps shrinking the loss while the policy drifts further from where it began.

DPO is simpler to implement and to debug, and the two methods are close enough in practice that the choice is usually made on engineering grounds.

Try it yourself. code/worked_examples/alignment.py produces every table in this section. Run --bt for the reward-model loss, --kl for the reward-hacking sweep and --dpo for the implicit rewards. Change one reward in the first table and watch which pair dominates the mean loss.

Run it in ColabNotebookSource

What alignment actually changes

It is worth being precise here, because the word carries more weight than the technique does. Alignment changes which of the model’s existing behaviours are likely, and it neither adds knowledge nor removes capability.

A model that could write malware before alignment can still write malware after it, and what changed is that the aligned model’s most probable response to the request is a refusal. That distinction explains most of what follows, including why jailbreaks exist, since a behaviour made unlikely is not a behaviour removed.

What still goes wrong

Four failure modes survive all three stages, and each has a known cause.

Hallucination is the model stating false things fluently, and the cause is structural, because it was trained to produce likely text and a plausible falsehood is likely text. Nothing in the objective distinguishes true from true-sounding, and Chapter 21 is the main practical response.

Sycophancy is the model agreeing with the user against the evidence, which is preference tuning working correctly on data where raters preferred agreeable answers. The signal was human approval, and approval and correctness are not the same thing.

Jailbreaks are careful phrasings that recover refused behaviour, and for the reason just given: alignment shifted probabilities rather than removing capabilities, so a sufficiently different framing can shift them back.

Inherited bias is the last. Chapter 9 measured social regularities in static embeddings with WEAT, the same regularities sit in the pretraining corpus of every large model, and alignment reduces their expression without removing their source. Chapter 27 takes this up properly.

None of the four is a bug in the sense of a mistake in the code. Each is a predictable consequence of the objective, which is why each needs a system around the model rather than a fix inside it.

Further reading.

Ouyang et al. (2022) is the InstructGPT paper and the clearest description of the three-stage pipeline. Rafailov et al. (2023) introduce direct preference optimisation. Bai et al. (2022) replace much of the human labelling with a written set of principles, which is a different answer to the same problem.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• alignment.pyPreference tuning, in three tables

  Bradley-Terry turns a ranking into a loss, the KL penalty stops reward hacking, and DPO does it without a reward model.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.