Chapter 24Efficiency: Smaller, Cheaper, Faster

The other axis

Most of this book has followed capability, through larger models, longer contexts and better scores. There is a second axis, and outside research it is usually the binding one, because a model that cannot run within your budget, memory or latency is not available to you at any quality.

This chapter is about that axis, and it divides cleanly into three questions. How do I adapt a model without paying for full fine-tuning, how do I make the model smaller, and how do I make it answer faster?

Adapting without paying for it

Full fine-tuning updates every weight, and the memory cost is worse than the parameter count suggests. For a $7$ billion parameter model in half precision:

Each fp32 quantity is four bytes per parameter, so one moment over $7$ billion parameters is $28$ GB and two are $56$. Mixed precision training usually keeps an fp32 master copy of the weights as well, which is another $28$ GB and takes the total past $110$; the table leaves it out, so treat $84$ GB as the floor rather than the figure.

Either way it is several high-end accelerators to fine-tune a model that fits on one for inference, and the result is a whole new $14$ GB of weights per task.

LoRA

Low-rank adaptation (Hu et al. 2022) rests on one observation, which is that the change a fine-tune makes to a weight matrix has low intrinsic rank even though the matrix does not. So do not learn the change directly but factor it:

$$\begin{equation} W' \;=\; W \;+\; BA, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k). \label{eq:lora} \quad\text{(24.1)} \end{equation}$$

Freeze $W$ entirely and train only $A$ and $B$. The arithmetic is the argument, since a full matrix has $dk$ parameters while the adapter has $r(d + k)$.

Those figures are for $d = k = 4096$, a typical attention projection. At $r = 8$ you train four tenths of one per cent of the weights, and the optimiser state shrinks in the same proportion, which is where most of the memory in the table above went.

Three practical consequences follow, and the third is the one people underrate. You can fine-tune a large model on one accelerator. An adapter is megabytes rather than gigabytes, so you can keep hundreds of them. And because Equation (24.1) is an addition to a frozen $W$, adapters can be swapped at serving time without reloading the base model, which gives one model in memory and many behaviours.

QLoRA combines the idea with a quantised base model and puts fine-tuning a $65$ billion parameter model within reach of a single accelerator.

Making the model smaller

Quantisation

Weights are usually trained in $16$ bits and they rarely need $16$ bits to be used.

The int4 row is the interesting one, because a model that needed a data-centre card at fp16 now fits on a consumer GPU. Quality loss at int8 is usually indistinguishable in practice, and at int4 the loss is measurable but often acceptable, which is almost always a better trade than moving to a genuinely smaller model.

What the rounding actually does

A quantised weight is an integer code and a scale shared across a group of weights. For a symmetric scheme the scale is set by the largest magnitude in the group,

$$\begin{equation} S = \frac{\max_i |w_i|}{q_{\max}}, \qquad q_i = \operatorname{round}\!\left(\frac{w_i}{S}\right), \qquad \hat{w}_i = q_i S, \label{eq:quantise} \quad\text{(24.2)} \end{equation}$$

where $q_{\max}$ is $127$ for int8 and $7$ for int4. Take eight weights from one row of a hidden layer and put them through Equation (24.2) twice:

Halving the bit width multiplies the root mean square error by $20.6$, because the step between neighbouring codes grows from $0.004016$ to $0.072857$. The largest weight is exact in both columns, since it is what defines the scale.

Why one weight can ruin a row

Equation (24.2) has a weakness that matters enormously in practice. The scale is set by the largest magnitude, so a single extreme value drags it upward and every other weight in the group inherits a coarser step. Trained transformers reliably produce such outliers in a few coordinates.

Add one weight of $+4.20$ to the same row and the int8 scale moves from $0.004016$ to $0.033071$, which is $8.2$ times coarser. The outlier itself is then stored exactly, with no error at all, while the small weights that make up the rest of the row carry up to $17.32$ per cent error. There are far more of them, and they are the ones doing the work.

The repair is to stop sharing one scale across the whole row:

Thirteen times less error, for a cost that is easy to miss. The scales now have to be stored too, and one fp16 scale for every group of $64$ weights adds two bytes per $64$, which is a quarter of a bit per weight. So int4 with group scales really costs $4.25$ bits, and every quantisation scheme in use is negotiating that same trade between the size of a group and the damage one outlier inside it can do.

Try it yourself. code/worked_examples/quantization.py produces every table in this section. Run --round for the two precisions, --outlier for the damage one weight does and --channel for the repair. Change the outlier from $4.20$ to $1.00$ and watch how much of the error disappears.

Run it in ColabNotebookSource

Distillation and pruning

Distillation trains a small student to match a large teacher’s output distribution, and the teacher’s full distribution carries more information than a hard label does, because it says which wrong answers were nearly right.

Pruning removes weights or whole structures that contribute little, where unstructured pruning gives better compression ratios while structured pruning gives speedups on real hardware, because it removes shapes rather than scattered entries.

The general finding across all three techniques is the same. Models are substantially over-parameterised for inference, and most of that slack can be recovered without retraining from scratch.

Making inference faster

Where the time actually goes

Generation has two phases with completely different cost profiles, and conflating them causes most confusion about latency. Prefill processes the whole prompt in parallel, so it is compute-bound and fast per token. Decoding produces one token at a time, each depending on the last, so it is memory-bound, because every step reads the entire weight matrix to produce one token. The second phase therefore dominates wall-clock time on any answer of reasonable length, and it is the one worth optimising.

The KV cache, and its cost

Each decoding step recomputes attention over the whole prefix, so caching the keys and values avoids that, and it is the single most important inference optimisation there is. It also becomes the memory problem. For $D = 4096$ and $L = 32$ layers in fp16, where one token costs $2 \times L \times D \times 2$ bytes, or half a mebibyte:

At long context the cache can exceed the weights, since $17.18$ GB against the $14$ GB of fp16 weights is a cache larger than the model. That is why grouped-query attention exists, sharing key and value projections across heads, and it is one of the few architecture changes since the original transformer driven purely by serving cost.

Batching, and the quadratic wall

Throughput comes from batching, because decoding is memory-bound and processing many sequences at once costs little more than one. Serving systems therefore batch aggressively and schedule continuously rather than waiting for a batch to fill.

Underneath all of it sits the $O(n^2)$ of Chapter 16, where quadrupling the context multiplies the attention cost sixteen-fold, and every long-context technique is an attempt to pay less than that. FlashAttention does not change the complexity but the memory traffic, computing attention in tiles that fit in fast on-chip memory, which is a large practical speedup for no change in the result.

Living with a deployed model

Two effects only appear after shipping, and neither is a modelling problem. Drift is the first, because the world changes and the training data does not, so a model’s accuracy on live traffic decays even though the model has not changed, and the response is monitoring rather than retraining on a schedule. Feedback loops are the second, because a deployed model influences the data it will later be trained on, so a recommender changes what people read and a text model changes what people write, and that loop is now measurable in web corpora.

The engineering discipline here is the empirical method of Chapter 14 applied to a system rather than an experiment. State the metric, hold out the evaluation, log the failures, and look at them.

Further reading.

Hu et al. (2022) introduce LoRA and its rank ablations are worth reading for the intuition about intrinsic dimension. Dettmers et al. (2023) combine it with quantisation. Dao et al. (2022) explain why memory traffic rather than arithmetic is the bottleneck. LoRA is peft, quantised loading is bitsandbytes, and vLLM is the serving stack that made paged attention and continuous batching standard.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• quantization.pyWhat rounding a weight costs

  The affine map, the error it leaves, and why one outlier ruins a whole row.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.