Chapter 19Steering LLMs: Decoding and Prompting

A distribution is not an answer

Part IV ends with a trained model which, given a context, produces a probability distribution over the next token, and that is Chapter 10’s task at transformer scale. A distribution is not an answer, though, so something has to choose.

That choice is decoding, and it is made outside the model after training is over, since nothing in this chapter changes a single weight. Two levers exist at that point, where decoding decides how a token is picked from the distribution and prompting decides what the distribution is conditioned on. Both are cheap, both are reversible, and both change the output more than most people expect.

Decoding: from distribution to token

Greedy, and where it fails

The obvious rule is to take the highest-probability token at every step, and it fails in a way that is easy to construct. Take a tiny model whose whole next-token tree is written out.

Greedy returns the cat sat, with probability $0.40 \times 0.55 \times 0.60 = 0.1320$.

Now rank every three-word sequence the model can produce.

The best sequence is more than twice as likely as the one greedy found, and greedy went wrong at the very first step. It took the at $0.40$ over a at $0.35$ and could not see that a leads to bird at $0.90$ and then sang at $0.95$.

That is the general shape of the failure. A high-probability token can sit in front of a low-probability continuation, and a rule that looks one step ahead cannot know.

Beam search

Beam search keeps the best $w$ partial sequences at every step instead of one.

Width $1$ is greedy by definition, width $2$ already finds the optimum here, and wider beams cost more for nothing. Beam search is not guaranteed to find the best sequence, because it searches more of the space than greedy and less than all of it, and that is the honest description.

It also has a known bias, since optimising total probability favours short, safe, generic output. That is acceptable in translation where there is a right answer, and it is a problem in open-ended generation where the safest continuation is usually the dullest.

Temperature

Sampling introduces variety and temperature controls how much of it. Divide the logits by $T$ before the softmax, which is the same as raising each probability to the power $1/T$ and renormalising.

The distribution has seven tokens and the table shows the largest five, so a row sums to $0.99$ rather than $1$, with each and every holding $0.005$ apiece at $T = 1$. The entropy column counts all seven.

$T = 1$ leaves the model’s own distribution alone, below $1$ it sharpens, and at $T = 0$ it is greedy with zero entropy because nothing is left to choose, while above $1$ it flattens. Both ends have a characteristic failure, since low temperature gives repetitive, safe text while high temperature gives varied text that drifts off topic and eventually stops making sense.

Top-$k$, top-$p$, and why the second replaced the first

Sampling from the full distribution has a problem, because a vocabulary of $50{,}000$ tokens holds a great many implausible options and their combined tail probability is not negligible. Top-$k$ keeps the $k$ most likely tokens and renormalises, while top-$p$, or nucleus sampling, keeps the smallest set whose probability mass reaches $p$. The difference shows the moment you try both on two distributions of different shape.

Now apply a fixed $k$ to both.

Read the $k = 5$ row. On the peaked distribution it admits four tokens the model had all but ruled out, and on the flat one it discards three tokens that were as good as the ones it kept. The same $k$ is too loose on one shape and too tight on the other, because $k$ cannot see the shape. Top-$p$ can.

At $p = 0.9$ the peaked distribution keeps two tokens while the flat one keeps eight, so the candidate set now follows the model’s own confidence, which is exactly what a fixed $k$ could not do. That is why nucleus sampling became the default.

All of it, on one distribution

Every row is the same model and none of them changed a weight. Which row to use follows from the task, since a factual answer wants a low-entropy row while a story wants one of the middle ones.

Try it yourself. code/worked_examples/decoding.py produces every table here. --beam shows greedy losing and beam recovering, --temp sweeps the temperature, --truncate contrasts top-$k$ with top-$p$ on both shapes, and --compare prints the last table.

Run it in ColabNotebookSource

Prompting: programming without weights

The second lever changes what the model is conditioned on. Chapter 18 noted that at sufficient scale a model can perform a task from a description in its input with no gradient update (Brown et al. 2020), and that capability is what makes prompting a technique rather than a formatting detail.

Three settings

Zero-shot gives an instruction and nothing else, few-shot adds a handful of worked examples in the prompt, and chain-of-thought asks the model to produce its reasoning before its answer (Wei et al. 2022). The third is the interesting one, and its mechanism is not mysterious.

A model produces one token at a time, each conditioned on what came before, so asking for an answer directly gives it one forward pass to reach the conclusion while asking for the steps first gives it many, with each step becoming context for the next. Chain-of-thought is therefore not the model thinking harder but the model being given room to compute.

What actually helps

Four things help, in rough order of reliability.

Be specific about the output, naming the format, the length and the allowed values, because most prompt failures are underspecification rather than model failure. Give examples rather than adjectives, since two demonstrations of the format you want beat a paragraph describing it. Ask for steps on multi-step problems, which is free and helps most where the task is arithmetic or logical. And put the constraints last, because the instruction nearest the generation point tends to dominate, which is a property of the architecture rather than a rule of style.

What prompting cannot do

Prompting selects from what the model already has, so it does not add knowledge the model lacks and it does not reliably remove behaviour the model learned. Two consequences follow, and they set up the next three chapters. If the model does not know a fact then no prompt will supply it, which is Chapter 21. And if the model’s default behaviour is wrong then prompting patches it case by case and never fixes it, which is Chapter 20.

The steering hierarchy

There are four ways to change what a model does, cheapest first.

Work down that list rather than up it, because a great deal of expensive fine-tuning has been spent on problems a decoding parameter would have solved. The order also has a diagnostic use, since if prompting cannot get close then the problem is usually knowledge or behaviour rather than phrasing, and that tells you which chapter to read next.

Further reading.

Brown et al. (2020) report in-context learning. Wei et al. (2022) introduce chain-of-thought prompting. Holtzman et al. (2020) is the nucleus sampling paper. Its account of why pure sampling degenerates is the clearest available. Jurafsky and Martin (2025) cover decoding alongside machine translation, where beam search has its longest history. Every strategy here is an argument to transformers’ generate, and its LogitsWarper classes are the operations of this chapter written out one per file, which makes them unusually easy to read.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• decoding.pyGreedy, beam, temperature, top-p

  Where greedy decoding loses, and why top-p replaced top-k.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.