A pretrained language model does exactly one thing. It continues text plausibly, because that is what Equation (17.1) asked of it.
Ask a base model “What is the capital of France?” and a perfectly good continuation is:
What is the capital of Germany? What is the capital of Italy?
Nothing has gone wrong. That is a very likely continuation of a list of questions, and the model was trained to produce likely continuations.
The gap is not capability. The model knows the answer. The gap is that nobody ever told it that a question should be followed by an answer rather than by another question.
Closing that gap is what this chapter is about, and it happens in stages.
| stage | data | teaches |
|---|---|---|
| pretraining | raw text, trillions of tokens | language and facts |
| instruction tuning | instruction and response pairs | the shape of a response |
| preference tuning | human rankings of responses | which response is better |
Pretraining is enormously expensive and happens once. The other two are moderate, and both are repeated.
The stages are not alternatives. Each assumes the one before it, and each teaches something the previous one could not.
Continue training on pairs of instructions and good responses.
The data is what matters. A few tens of thousands of well-written pairs turn a text continuer into something that answers questions, follows formats and stops when it is done.
Note what has not happened. No new knowledge entered the model. The capital of France was already there. What changed is the model’s default behaviour when it sees a question.
That is why the stage is cheap relative to pretraining, and why data quality dominates data quantity here.
Instruction tuning teaches the shape of a response. It cannot easily teach which of two well-shaped responses is better.
Nobody can write a loss function for helpfulness. But people can reliably say which of two answers they prefer, and that comparison is enough.
Reinforcement learning from human feedback turns those comparisons into a training signal in three steps.
Collect pairs of model responses and have people rank them. Train a reward model to predict those rankings. Then optimise the language model against the reward model, with a penalty for drifting too far from where it started.
The penalty term is not an afterthought. Without it the model finds outputs the reward model scores highly and humans do not, which is reward hacking, and it happens reliably.
Direct preference optimisation reaches a similar destination by a shorter road. It optimises the preference data directly, with no separate reward model and no reinforcement learning loop.
DPO is simpler to implement and to debug. The two are close enough in practice that the choice is usually made on engineering grounds.
It is worth being precise, because the word carries more weight than the technique does.
Alignment changes which of the model’s existing behaviours are likely. It does not add knowledge, and it does not remove capability.
A model that could write malware before alignment can still write malware after it. What changed is that the aligned model’s most probable response to the request is a refusal.
That distinction explains most of what follows, including why jailbreaks exist. A behaviour made unlikely is not a behaviour removed.
Four failure modes survive all three stages, and each has a known cause.
The model states false things fluently. The cause is structural. It was trained to produce likely text, and a plausible falsehood is likely text. Nothing in the objective distinguishes true from true-sounding. Chapter 20 is the main practical response.
The model agrees with the user against the evidence. This is preference tuning working correctly on data where raters preferred agreeable answers. The signal was human approval, and approval and correctness are not the same thing.
Careful phrasing recovers refused behaviour. As above, alignment shifted probabilities rather than removing capabilities, so a sufficiently different framing can shift them back.
Chapter 9 measured social regularities in static embeddings with WEAT. The same regularities are in the pretraining corpus of every large model, and alignment reduces their expression without removing their source. Chapter 26 takes this up properly.
None of the four is a bug in the sense of a mistake in the code. Each is a predictable consequence of the objective, which is why each needs a system around the model rather than a fix inside it.
is the InstructGPT paper and the clearest description of the three-stage pipeline. introduce direct preference optimisation. replace much of the human labelling with a written set of principles, which is a different answer to the same problem.
The continuation trap. Write three prompts on which a base language model and an instruction-tuned one would behave very differently. For each, describe the base model’s likely output and say why it is a correct continuation.
Why a reward model. Explain why preference tuning needs a learned reward model rather than direct human feedback in the training loop. Then explain what the penalty term protects against, and predict what its absence would produce.
Shape against substance. Instruction tuning changes behaviour without adding knowledge. Design an experiment that would distinguish the two, using only prompts and no access to the weights.
Diagnose the failure. For each symptom, name which stage is responsible and what you would change: (a) the model invents citations; (b) it reverses a correct answer when contradicted; (c) it refuses a legitimate medical question; (d) it produces a different quality of answer for names from different regions.
Sycophancy as a design consequence. Argue that sycophancy is preference tuning succeeding rather than failing. Then propose one change to the data collection that would reduce it, and name what your change costs.