Chapter 18Steering LLMs: Decoding and Prompting

A distribution is not an answer

Part IV ends with a trained model. Given a context it produces a probability distribution over the next token, which is Chapter 10’s task at transformer scale.

A distribution is not an answer. Something has to choose.

That choice is decoding, and it is made outside the model, after training is over. Nothing in this chapter changes a single weight.

Two levers exist at that point. Decoding decides how a token is picked from the distribution. Prompting decides what the distribution is conditioned on.

Both are cheap, both are reversible, and both change the output more than most people expect.

Decoding: from distribution to token

Greedy, and where it fails

The obvious rule is to take the highest-probability token at every step.

It fails, and the failure is easy to construct. Take a tiny model whose whole next-token tree is written out.

Greedy returns the cat sat, with probability $0.40 \times 0.55 \times 0.60 = 0.1320$.

Now rank every three-word sequence the model can produce.

The best sequence is more than twice as likely as the one greedy found.

Greedy went wrong at the first step. It took the at $0.40$ over a at $0.35$, and could not see that a leads to bird at $0.90$ and then sang at $0.95$.

That is the general shape of the failure. A high-probability token can sit in front of a low-probability continuation, and a rule that looks one step ahead cannot know.

Beam search

Beam search keeps the best $w$ partial sequences at every step instead of one.

Width $1$ is greedy by definition. Width $2$ already finds the optimum here, and wider beams cost more for nothing.

Beam search is not guaranteed to find the best sequence. It searches more of the space than greedy and less than all of it, and that is the honest description.

It also has a known bias. Because it optimises total probability, it favours short, safe, generic output. That is acceptable in translation, where there is a right answer. It is a problem in open-ended generation, where the safest continuation is usually the dullest.

Temperature

Sampling introduces variety, and temperature controls how much.

Divide the logits by $T$ before the softmax, which is the same as raising each probability to the power $1/T$ and renormalising.

$T = 1$ leaves the model’s own distribution alone. Below $1$ it sharpens, and at $T = 0$ it is greedy, with zero entropy because nothing is left to choose. Above $1$ it flattens.

Both ends have a characteristic failure. Low temperature gives repetitive, safe text. High temperature gives varied text that drifts off topic and eventually stops making sense.

Top-$k$, top-$p$, and why the second replaced the first

Sampling from the full distribution has a problem. A vocabulary of $50{,}000$ tokens holds a great many implausible options, and their combined tail probability is not negligible.

Top-$k$ keeps the $k$ most likely tokens and renormalises. Top-$p$, or nucleus sampling, keeps the smallest set whose probability mass reaches $p$.

The difference shows the moment you try both on two distributions of different shape.

Now apply a fixed $k$ to both.

Read $k = 5$. On the peaked distribution it admits four tokens the model had all but ruled out. On the flat one it discards three tokens that were as good as the ones it kept.

The same $k$ is too loose on one shape and too tight on the other, because $k$ cannot see the shape.

Top-$p$ can.

At $p = 0.9$ the peaked distribution keeps two tokens and the flat one keeps eight. The candidate set now follows the model’s own confidence, which is exactly what a fixed $k$ could not do.

That is why nucleus sampling became the default.

All of it, on one distribution

Every row is the same model, and none of them changed a weight.

Which row to use follows from the task. A factual answer wants a low-entropy row. A story wants one of the middle ones.

Try it yourself. code/worked_examples/decoding.py produces every table here. --beam shows greedy losing and beam recovering, --temp sweeps the temperature, --truncate contrasts top-$k$ with top-$p$ on both shapes, and --compare prints the last table.

Run it in ColabNotebookSource

Prompting: programming without weights

The second lever changes what the model is conditioned on.

Chapter 17 noted that at sufficient scale a model can perform a task from a description in its input, with no gradient update . That capability is what makes prompting a technique rather than a formatting detail.

Three settings

Zero-shot gives an instruction and nothing else. Few-shot adds a handful of worked examples in the prompt. Chain-of-thought asks the model to produce its reasoning before its answer .

The third is the interesting one, and its mechanism is not mysterious.

A model produces one token at a time, each conditioned on what came before. Asking for an answer directly gives it one forward pass to reach the conclusion. Asking for the steps first gives it many, and each step becomes context for the next.

So chain-of-thought is not the model thinking harder. It is the model being given room to compute.

What actually helps

Four things, in rough order of reliability.

Be specific about the output. Name the format, the length and the allowed values. Most prompt failures are underspecification, not model failure.

Give examples rather than adjectives. Two demonstrations of the format you want beat a paragraph describing it.

Ask for steps on multi-step problems. Free, and it helps most where the task is arithmetic or logical.

Put the constraints last. The instruction nearest the generation point tends to dominate, which is a property of the architecture rather than a rule of style.

What prompting cannot do

Prompting selects from what the model already has. It does not add knowledge the model lacks, and it does not reliably remove behaviour the model learned.

Two consequences follow, and they set up the next three chapters.

If the model does not know a fact, no prompt will supply it. That is Chapter 20.

If the model’s default behaviour is wrong, prompting patches it case by case and never fixes it. That is Chapter 19.

The steering hierarchy

Four ways to change what a model does, cheapest first.

Work down that list, not up. A great deal of expensive fine-tuning has been spent on problems a decoding parameter would have solved.

The order also has a diagnostic use. If prompting cannot get close, the problem is usually knowledge or behaviour rather than phrasing. That tells you which chapter to read next.

Further reading.

report in-context learning. introduce chain-of-thought prompting. is the nucleus sampling paper. Its account of why pure sampling degenerates is the clearest available. cover decoding alongside machine translation, where beam search has its longest history.