Chapter 23Efficiency: Smaller, Cheaper, Faster

The other axis

Most of this book has followed capability. Larger models, longer contexts, better scores.

There is a second axis, and outside research it is usually the binding one. A model that cannot run within your budget, memory or latency is not available to you at any quality.

This chapter is about that axis. It divides cleanly into three questions.

How do I adapt a model without paying for full fine-tuning? How do I make the model smaller? How do I make it answer faster?

Adapting without paying for it

Full fine-tuning updates every weight, and the memory cost is worse than the parameter count suggests.

For a $7$ billion parameter model in half precision:

That is several high-end accelerators to fine-tune a model that fits on one for inference. And the result is a whole new $14$ GB of weights per task.

LoRA

Low-rank adaptation rests on one observation. The change a fine-tune makes to a weight matrix has low intrinsic rank, even though the matrix does not.

So do not learn the change directly. Factor it:

$$\begin{equation} W' \;=\; W \;+\; BA, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k). \label{eq:lora} \quad\text{(23.1)} \end{equation}$$

Freeze $W$ entirely. Train only $A$ and $B$.

The arithmetic is the argument. A full matrix has $dk$ parameters. The adapter has $r(d + k)$.

Those figures are for $d = k = 4096$, a typical attention projection.

At $r = 8$ you train four tenths of one per cent of the weights. The optimiser state shrinks in the same proportion, which is where most of the $84$ GB went.

Three practical consequences follow, and the third is the one people underrate.

You can fine-tune a large model on one accelerator.

An adapter is megabytes rather than gigabytes, so you can keep hundreds of them.

And because Equation (23.1) is an addition to a frozen $W$, adapters can be swapped at serving time without reloading the base model. One model in memory, many behaviours.

QLoRA combines the idea with a quantised base model, and puts fine-tuning a $65$ billion parameter model within reach of a single accelerator.

Making the model smaller

Quantisation

Weights are usually trained in $16$ bits. They rarely need $16$ bits to be used.

The int4 row is the interesting one. A model that needed a data-centre card at fp16 now fits on a consumer GPU.

Quality loss at int8 is usually indistinguishable in practice.

At int4 the loss is measurable but often acceptable. It is almost always a better trade than moving to a genuinely smaller model.

Distillation and pruning

Distillation trains a small student to match a large teacher’s output distribution. The teacher’s full distribution carries more information than a hard label does, because it says which wrong answers were nearly right.

Pruning removes weights or whole structures that contribute little. Unstructured pruning gives better compression ratios; structured pruning gives speedups on real hardware, because it removes shapes rather than scattered entries.

The general finding across all three techniques is the same. Models are substantially over-parameterised for inference, and most of that slack can be recovered without retraining from scratch.

Making inference faster

Where the time actually goes

Generation has two phases with completely different cost profiles, and conflating them causes most confusion about latency.

Prefill processes the whole prompt in parallel. It is compute-bound, and it is fast per token.

Decoding produces one token at a time, each depending on the last. It is memory-bound, because every step reads the entire weight matrix to produce one token.

So the second phase dominates wall-clock time on any answer of reasonable length, and it is the one worth optimising.

The KV cache, and its cost

Each decoding step recomputes attention over the whole prefix. Caching the keys and values avoids that, and it is the single most important inference optimisation there is.

It also becomes the memory problem. For $D = 4096$ and $L = 32$ layers, in fp16:

At long context the cache can exceed the weights.

That is why grouped-query attention exists, sharing key and value projections across heads. It is one of the few architecture changes since the original transformer driven purely by serving cost.

Batching, and the quadratic wall

Throughput comes from batching. Because decoding is memory-bound, processing many sequences at once costs little more than one.

So serving systems batch aggressively, and schedule continuously rather than waiting for a batch to fill.

Underneath all of it sits the $O(n^2)$ of Chapter 16. Quadrupling the context multiplies the attention cost sixteen-fold, and every long-context technique is an attempt to pay less than that.

FlashAttention does not change the complexity. It changes the memory traffic, computing attention in tiles that fit in fast on-chip memory. That is a large practical speedup for no change in the result.

Living with a deployed model

Two effects that only appear after shipping, and neither is a modelling problem.

Drift. The world changes and the training data does not. A model’s accuracy on live traffic decays even though the model has not changed. The response is monitoring, not retraining on a schedule.

Feedback loops. A deployed model influences the data it will later be trained on. A recommender changes what people read; a text model changes what people write. That loop is now measurable in web corpora.

The engineering discipline here is the empirical method of Chapter 14, applied to a system rather than an experiment. State the metric, hold out the evaluation, log the failures, and look at them.

Further reading.

introduce LoRA and its rank ablations are worth reading for the intuition about intrinsic dimension. combine it with quantisation. explain why memory traffic rather than arithmetic is the bottleneck.