Translation has a target. Two competent translators produce texts that are recognisably the same.
Summarisation does not work that way. Two competent people summarising the same article will disagree about what mattered, and both can be right.
That makes it the hardest generation task to evaluate and the most useful one to study. Every difficulty in Part V shows up here at once: decoding choices, faithfulness, evaluation that does not measure what you care about, and a context window that is too small.
The word covers two quite different problems.
| extractive | abstractive | |
|---|---|---|
| what it does | selects sentences from the source | generates new text |
| can it hallucinate? | no, by construction | yes |
| how it reads | disjointed | fluent |
| what it is | a ranking problem | a generation problem |
Extractive summarisation cannot state anything the source did not, which is a guarantee no generative system can offer. It reads badly, because sentences lifted out of context do not join up.
Abstractive summarisation reads well and can compress genuinely, by saying in one clause what the source took a paragraph to say. It can also state things the source never said.
Transformers made abstractive summarisation practical. They also made faithfulness the central problem, and the two facts are the same fact.
This is the part most often skipped, and it changes everything downstream.
Two datasets dominate the literature and they are not the same task.
CNN/DailyMail pairs news articles with their bullet-point highlights. Those highlights are largely extractive: most of their content appears near-verbatim in the first few sentences of the article.
XSum pairs BBC articles with their one-sentence opening summary, which is written to be genuinely abstractive and often states something no single sentence of the article states.
Three consequences follow.
A model tuned on one transfers poorly to the other, because they reward different behaviour.
On CNN/DailyMail a lead-sentence baseline, which simply returns the first few sentences, is competitive. If your model cannot beat it, you have not established that it learned anything.
Hallucination rates on XSum are far higher, and this is not a defect of the models. The dataset asks for statements not present in the source, so a model that obliges will sometimes oblige incorrectly.
The practical rule is short. Before choosing a model, measure how extractive your target summaries are. That number tells you what you are actually training.
Two shapes are available and both are in Chapter 17.
An encoder-decoder reads the document with a bidirectional encoder and writes with a causal decoder. The encoder sees the whole source at once, which is the natural fit, and the architecture was designed for exactly this.
A decoder-only model puts the document and the summary in one sequence and continues it. This is what a general instruction-tuned model does when you ask it to summarise.
The encoder-decoder shape has the better inductive bias and needs fine-tuning. The decoder-only shape needs no fine-tuning at all and arrives already competent, which usually wins on engineering grounds.
Pretraining objective matters more than shape. Models pretrained by reconstructing corrupted spans of text, rather than by predicting the next token, transfer to summarisation better, because span reconstruction is the same operation summarisation performs.
Chapter 16 established that attention costs in the sequence length. Summarisation is where that bill arrives.
A news article fits in a few hundred tokens. A scientific paper is five to ten thousand. A transcript or a legal filing is longer still. Many checkpoints cap at 512 or 1024.
Three responses, with honest costs.
Truncate. Cheap, and it silently discards the tail of the document. On news this is nearly free, because news is written with the important part first. On a paper it removes the results.
Sparse attention, as in Longformer, LED and BigBird. Each position attends to a local window plus a few global positions, which brings the cost back to roughly linear. It buys length at some cost in quality.
Hierarchical summarisation. Summarise chunks, then summarise the summaries. It handles any length and it compounds errors, because a mistake in a chunk summary is invisible at the next level.
Measure the length distribution of your inputs and your targets before you set the truncation limits.
Truncating inputs is visible: you know you did it. Truncating targets is silent, and it is a common bug.
A target cut at the limit teaches the model to stop early. The loss looks fine throughout training, because the model is correctly predicting a truncated target. The failure only appears at generation time, as summaries that end mid-thought.
Training is the cross-entropy of Chapter 11, with two details that matter.
Mask the loss. Padding tokens must not contribute. An unmasked loss over a heavily padded batch is mostly measuring the model’s ability to predict padding, and it will look encouragingly low.
Teacher forcing. At each step the decoder is fed the correct previous token rather than its own prediction. That makes training parallel and stable, and it introduces the problem below.
The mismatch is exact.
During training, every previous token the model sees is correct. During generation, the model consumes its own output.
So the model has never been trained on the distribution it faces at inference. One early error shifts the context for everything after it.
The model has no experience of recovering from its own mistakes, because in training it never made any.
Three partial remedies exist and none is a cure.
Label smoothing stops the model becoming over-confident, which makes it less brittle when the context is unfamiliar. Scheduled sampling occasionally feeds the model’s own predictions during training. Sequence-level fine-tuning optimises against the evaluation metric rather than per-token likelihood.
Exposure bias is also the cleanest reason decoding matters here. Sampling at high temperature compounds it, because an unlikely token early is a corrupted context for everything that follows. Summarisation wants the low-entropy end of Chapter 18’s table.
Chapter 21 covered ROUGE, which is the standard here. Two things about using it.
Evaluate with generation, not loss. Validation loss measures next-token prediction under teacher forcing, which is not the task. Generate summaries and score them. It is slower, and it is the number that matters.
Watch loss alone and you watch a quantity that improves while the thing you care about does not. Chapter 21 named that failure.
ROUGE rewards overlap, and overlap is not faithfulness. A summary can score well and state something false, and a correct paraphrase can score near zero. Chapter 21 showed exactly that with the feline rests upon the rug.
For summarisation the gap is specific, and it has a name. Faithfulness is whether every claim in the summary is supported by the source. ROUGE does not measure it, cannot measure it, and a system tuned on ROUGE alone will drift away from it.
Measuring faithfulness needs a different instrument: entailment models asking whether the source entails each summary sentence, question-answering consistency checks, or human annotation. All three are more expensive than ROUGE, which is why ROUGE is still reported.
The chapter has assumed you are training a model. Often you should not.
A general instruction-tuned model summarises well with no training at all, which is Chapter 18’s point. Fine-tuning earns its cost when you need a specific style, a specific length, a specific domain vocabulary, or an on-premises model.
When you do fine-tune, Chapter 23 applies unchanged. LoRA at rank 8 trains four tenths of one per cent of the weights. That is usually enough for a task this close to what the model already does.
The order to work in is Chapter 18’s hierarchy. Prompt first. Measure. Fine-tune only when you can say what prompting failed to achieve.
is ROUGE, and its limitations are best read alongside . introduced the span-corruption pretraining that transfers so well here. cover both extractive and abstractive summarisation with more task detail.
Measure your dataset. For a summarisation corpus of your choice, compute what fraction of summary bigrams appear verbatim in the source. Compare it against a lead-sentence baseline’s ROUGE. What does the gap tell you about what a trained model would have to learn?
Beat the lead baseline. Implement the baseline that returns the first three sentences and score it. Then argue what a model must do to justify its cost, and state the margin you would require.
The silent truncation. Explain why truncating targets produces a training loss that looks healthy while generation fails. Describe the check you would add to a training script to make the bug loud.
Exposure bias, concretely. Trace what happens when a decoder generates one wrong token at position three of a twenty-token summary. Explain why teacher forcing gives the model no experience of this, and say which of the three remedies addresses it most directly.
Two metrics disagree. Construct a summary that scores well on ROUGE and is unfaithful to its source, and one that is faithful and scores poorly. Then describe an evaluation that separates them.
Choose the architecture. You must summarise ten-thousand-token legal filings, on-premises, in a fixed house style. Work through the choices in this chapter: extractive or abstractive, encoder-decoder or decoder-only, truncate or sparse or hierarchical, prompt or fine-tune. Justify each, and name the one you are least sure about.