Chapter 17Pretrained Transformers: BERT and GPT

Two doors from one building block

Chapter 16 built a block. This chapter builds the two families that came out of it.

The only real difference between them is what each one is allowed to see.

Take the attention of Equation (16.2). Apply the causal mask of Equation (16.3) and each position sees only its past. Leave the mask off and each position sees the whole sentence.

That single choice separates GPT from BERT, and everything else follows from it.

The last row is not a limitation anyone chose. It falls out of the second row, and Section 1.3.3 explains why.

GPT: pretraining by prediction

GPT is a stack of decoder blocks, trained on the oldest objective in this book.

Given the tokens so far, predict the next one. That is Equation (10.1) from Chapter 10, with a transformer in place of the count table:

$$\begin{equation} \mathcal{L} \;=\; -\sum_i \log p\big(w_i \mid w_1, \ldots, w_{i-1}\big). \label{eq:gpt-loss} \quad\text{(17.1)} \end{equation}$$

Nothing in that equation is new. The n-gram model of Chapter 10 minimised the same quantity, and so did the recurrent language model of Chapter 12.

What changed is the machinery computing the probability, and how much data it could be fed.

Why the causal mask is not optional here

Without the mask, position $i$ could attend to position $i+1$ and read the answer it is being asked to predict.

The loss would fall to nearly zero and the model would learn nothing.

This is the reasoning that made </s> a scored token in Chapter 10. An objective that can be satisfied trivially is the wrong objective.

So the mask is what makes the task hard enough to be worth solving.

What prediction turns out to teach

The surprise of the GPT line is how much falls out of so little.

To predict the next token well across a large corpus, a model has to represent a great deal. Syntax, facts, discourse structure, and something that functions like reasoning.

None of that was asked for. All of it is instrumental to the one objective.

That claim was made repeatedly and doubted repeatedly. The evidence for it is in Chapter 18, where models trained this way answer questions they were never trained on.

BERT: pretraining by filling gaps

BERT drops the mask and therefore needs a different objective.

If every position can see every other, next-token prediction is trivial. So BERT hides tokens instead, and predicts those.

This is masked language modelling, and it is the cloze task from language teaching.

The masking budget

Two numbers define it, and both are trades worth seeing.

The first is how much to hide. Take a sequence of $512$ tokens.

At one per cent you pay for a full forward pass over $512$ tokens and collect five predictions. Correct, and very slow.

At ninety per cent you collect plenty of predictions with almost no context to make them from. The task degenerates into guessing.

BERT chose $15$ per cent, which gives $76$ gradients per sequence with $85$ per cent of the sentence still readable. Later work has found higher rates workable on larger models, so this is a balance point rather than a constant.

The mismatch, and the fix

The second number is subtler, and it repairs a problem the first one creates.

The token [MASK] appears in every pretraining batch and never once at fine-tuning time. A model that keyed on it would break the moment it was used for real.

So of the $15$ per cent chosen, BERT does three different things.

The last two rows carry the argument.

Because a chosen position might be left unchanged, the model cannot tell which positions were chosen. So it has to build a usable representation of every token rather than only the obviously missing ones.

The random-token row teaches something else again. It forces the model to notice when a token does not fit its context. No amount of [MASK] prediction would ever require that.

Why BERT cannot generate

Generation means producing token $i$ from tokens $1$ to $i-1$, then $i+1$ from $1$ to $i$, and so on.

BERT’s every layer attends in both directions. Position $i$ has been built from the whole sequence, including positions that do not exist yet during generation.

You could mask BERT at inference time. But then it is being used in a configuration it was never trained in, and it performs badly.

This is the same restriction the bidirectional RNN had in Chapter 12, for the same reason. Reading both ways and generating left to right are incompatible requirements, and a model has to choose.

The same objective, scaled

The GPT line’s later chapters are mostly one experiment repeated with more of everything.

The architecture barely changed across that table. The objective did not change at all.

What changed was scale, and something unexpected came with it.

At GPT-3’s size the model could perform tasks from a description and a few examples in its prompt, with no gradient update at all .

That is in-context learning, and nobody designed it. It is the behaviour that turned language models from a component into a product, and Chapter 18 is about using it.

Where the parameters go

The counting of Chapter 16 scales up unchanged. For a transformer with $D$ model dimension, $L$ layers and feed-forward size $H = 4D$, each layer costs roughly $12D^2$.

The quadratic in $D$ is why width is expensive. The linear in $L$ is why depth is comparatively cheap.

Both are dwarfed by the third row, where each of the $96$ layers costs $1.8$ billion parameters on its own.

What a pretrained model is for

Two ways of using one, and the field has moved from the first to the second.

Fine-tuning continues training on a labelled task, updating all the weights. It is what BERT was designed for, and it works well with a few thousand examples. It produces one model per task.

Prompting leaves the weights alone and describes the task in the input. It needs no labelled data and no training run, and one model serves every task.

Fine-tuning generally scores higher when you have the labels. Prompting wins on everything else.

Chapter 23 shows how to get most of fine-tuning’s benefit at a fraction of its cost.

The deeper shift is in what a model is. Before pretraining, a model was built for a task. After it, a model is a general capability that tasks are requested from.

Part V is about making those requests well. Checking the answers, and living with what the model learned from its corpus.

Further reading.

is the first GPT paper and the one that reported in-context learning. is BERT, and its ablations on the masking scheme are worth the read on their own. remains the architecture both are built from. cover the pretraining and fine-tuning pattern with more task detail.