Chapter 8Neural Network Fundamentals

When rules run out

The methods of Chapter 7 never had to learn anything. They counted, reweighted and factored. Every step was a fixed recipe applied to data.

Many problems yield to that style. Many do not.

Nobody can write down the rules that tell a photograph of a cat from one of a dog. Nor the rules that map an acoustic waveform to the sentence it encodes. Nor, to stay in this book, the rules that decide whether a film review is favourable.

The knowledge is real but tacit. We recognise the cat without being able to say what makes it one.

For such problems there is only one way forward. Learn the mapping from examples. Show the machine thousands of labelled cases and let it adjust itself until it gets them right. The hope is that it captured the pattern rather than memorising the cases.

This chapter builds the machine that does the adjusting. Every model in the rest of this book is this machine enlarged, from the word2vec of Chapter 9 to the transformers of Part IV.

The neuron

The atom of the machine is deliberately simple.

A single artificial neuron takes an input vector $x$. It forms a weighted sum with learnable weights $w$ and a bias $b$. It passes the result through a nonlinear activation function $f$:

$$\begin{equation} z \;=\; w \cdot x + b, \qquad a \;=\; f(z). \label{eq:neuron} \quad\text{(8.1)} \end{equation}$$

The weighted sum $z$ is called the pre-activation. It is a linear function of the input.

The activation $f$ is what lets the neuron represent anything more interesting than a line. Without it, stacking neurons would be pointless, because a composition of linear maps is just another linear map. So the choice of $f$ is not cosmetic. It is what lets a network bend and curve and carve up its input space.

Activation functions

The historical first choice was a hard threshold. Output $1$ if $z > 0$, and $0$ otherwise.

It captures the all or nothing firing of a real neuron. It is also useless for learning. Its derivative is zero everywhere it is defined, so it gives no signal about which way to nudge the weights. Learning by gradients demands a smooth activation.

The sigmoid was the classic smooth replacement: $\sigma(z) = 1/(1 + e^{-z})$. It squashes any real number into $(0, 1)$, which is convenient when the output should read as a probability. Some values worth knowing: $\sigma(0) = 0.5$, $\sigma(2) \approx 0.88$, and $\sigma(-2) \approx 0.12$.

It has two defects, and this book has already felt both.

Its outputs are not centred on zero, which slows learning. More seriously, it saturates. For large $|z|$ the curve flattens and its derivative approaches zero.

That derivative is $\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$. Its maximum sits at $z = 0$ and is only $0.25$. So even at its best a sigmoid shrinks the gradient passing through it by a factor of four. Stack that across many layers and you have the vanishing gradient that crippled the recurrent networks of Chapter 12.

The tanh function fixes half the problem. It is zero centred, ranging over $(-1, 1)$, and its derivative at the origin is $1$ rather than $0.25$. It preserves gradient better, which is why the recurrent networks of Chapter 12 and the gates of Chapter 13 used it.

The modern default fixes the other half. The rectified linear unit is $\operatorname{ReLU}(z) = \max(0, z)$. For positive inputs its derivative is exactly $1$, so it does not saturate on that side at all. It is also trivially cheap to compute.

It has its own flaw. A neuron stuck at $z < 0$ has zero gradient and can die. Leaky variants patch this by allowing a small negative slope.

The through line is the same for all of them. An activation must be nonlinear enough to give the network expressive power, and smooth enough to give learning a gradient to follow.

The perceptron and its limit

Before layers, meet a single neuron on its own. What it can and cannot do is why every network after it has a middle.

The perceptron is Equation (8.1) with a hard threshold. Frank Rosenblatt built one in 1958, in hardware, with motors adjusting potentiometers for weights. It takes inputs, weights them, adds a bias, and fires if the total clears zero.

It also learns, by a rule so simple you can run it in your head. Show it an example. If it is right, change nothing. If it is wrong, nudge every weight in the direction that would have helped:

$$\begin{equation} w \;\leftarrow\; w + \eta\,(y - \hat{y})\,x. \label{eq:perceptron-rule} \quad\text{(8.2)} \end{equation}$$

Rosenblatt proved this converges, provided a solution exists. That proviso is the whole story of this section.

What one neuron can do

A perceptron computes $w \cdot x + b > 0$. That is the equation of a line in two dimensions, a plane in three, a hyperplane in general. So a perceptron draws one straight boundary and answers according to which side you fall on.

Plenty of useful functions are that shape. Here are two, with weights you can check by hand.

AND at $(1,1)$ gives $1 + 1 - 1.5 = 0.5 > 0$, so it fires. At $(0,1)$ it gives $0 + 1 - 1.5 = -0.5$, so it does not. One line, drawn between the single corner where both inputs are on and the three where they are not.

This is why perceptrons were exciting. Character recognition. Simple sensor decisions. Any judgement where the classes fall on opposite sides of a boundary. All of it became learnable from examples rather than programmed.

Where it fails: XOR

Now try exclusive or. Fire when exactly one input is on.

Draw the four points. The two that should fire, $(0,1)$ and $(1,0)$, sit on one diagonal. The two that should not, $(0,0)$ and $(1,1)$, sit on the other. They interleave.

No single straight line separates one diagonal from the other. The failure is not lack of cleverness, and it can be proved in four lines.

Suppose some $w_1, w_2, b$ computes XOR. Then:

Add the second and third requirements: $w_1 + w_2 + 2b > 0$. Rearrange: $w_1 + w_2 + b > -b$. The first requirement says $b \le 0$, so $-b \ge 0$, and therefore $w_1 + w_2 + b > 0$.

That contradicts the fourth requirement. No such weights exist.

Minsky and Papert published this in 1969, and the field’s funding and enthusiasm collapsed for a decade. The result is correct. The conclusion drawn from it was not, because the fix was already known in principle. Add a layer.

The fix: remap the input

Here is the network. Two hidden units with ReLU, then one output unit. All weights are small integers, so you can verify every number.

$$\begin{align} h_1 &= \operatorname{ReLU}(x_1 + x_2 + 0) \nonumber \\ h_2 &= \operatorname{ReLU}(x_1 + x_2 - 1) \label{eq:xor-net} \\ y &= h_1 - 2\,h_2 \nonumber \end{align}$$

Run all four inputs through it.

It works. But how it works is the lesson, and it is visible if you throw away the input columns and look only at where each point landed.

That table is the whole idea of deep learning.

Four input points became three hidden points. The two inputs that share a target, $(0,1)$ and $(1,0)$, were mapped onto the same place. The hidden layer decided they were the same kind of thing.

And now the problem is linear. In hidden space the target-1 point sits at $(1,0)$, between the two target-0 points at $(0,0)$ and $(2,1)$. The output unit computes $h_1 - 2h_2$, which scores them $0$, $1$ and $0$. One threshold at $0.5$ separates them cleanly.

The output unit is still just a perceptron. It still draws one straight line. What changed is the space it draws that line in.

Why the remap is the whole idea

A hidden layer does not classify. It moves the data to a place where classifying is easy.

The last layer of almost every network in this book is a linear map followed by a softmax. That is a perceptron with more outputs. It was never made cleverer. Everything before it exists to hand it a representation where a straight line will do.

Once you see that, the rest of the book is one idea in many costumes.

Every row is the XOR trick at a different scale. Chapter 6 made the same complaint about one-hot vectors, that every word is equidistant from every other. The answer there was a better space too.

The word for what a hidden layer produces is a representation, and learning good representations is what the field means when it says deep learning. XOR is the smallest example where you can watch it happen and check every number by hand.

Layers, loss, and cost

A single neuron is a weak learner. The power comes from arranging many into layers and stacking the layers, so each layer’s activations feed the next.

Such a network is called feed-forward, or a multilayer perceptron. It applies an affine map and a nonlinearity at every layer. With enough hidden units it can approximate almost any function. That is what makes it a universal tool.

You have already met one such network. The lookup, mix and softmax stack of the neural language model in Chapter 11 is exactly this. So, at bottom, is every architecture that follows it.

To train a network we need a number saying how wrong it currently is.

For a single training example the loss compares the prediction $\hat{y}$ with the true target $y$. For classification the standard choice is cross-entropy, which for a binary target is

$$\begin{equation} L \;=\; -\big[\,y \log \hat{y} + (1 - y)\log(1 - \hat{y})\,\big]. \label{eq:nn-ce} \quad\text{(8.3)} \end{equation}$$

It behaves exactly as a good loss should. Take the truth to be $y = 1$.

Cross-entropy rewards confident correctness and punishes confident error. That is the same pressure we met in the perplexity of Chapter 10, and it is no coincidence. Perplexity is the exponentiated cross-entropy of a language model.

The loss is defined for one example. What we actually minimise is the cost. It is the average loss over the whole training set, seen as a function of all the parameters $\theta$:

$$\begin{equation} J(\theta) \;=\; \frac{1}{N}\sum_{i=1}^{N} L\big(\hat{y}_i, y_i\big). \label{eq:cost} \quad\text{(8.4)} \end{equation}$$

Learning is now a concrete goal. Find the $\theta$ that makes $J(\theta)$ as small as possible.

Gradient descent and backpropagation

We minimise $J$ the way you descend a hill in fog. Feel which way is steepest downhill, then step that way.

The direction of steepest increase is the gradient $\nabla_\theta J$. So we step the opposite way:

$$\begin{equation} \theta \;\leftarrow\; \theta - \eta\,\nabla_\theta J, \label{eq:gd} \quad\text{(8.5)} \end{equation}$$

where the learning rate $\eta$ sets the step size. Too small and training crawls. Too large and it overshoots and diverges.

Repeat this for many steps, in practice on small random batches of examples. Gradient descent then walks the parameters downhill towards a good fit.

Equation (8.5) needs one thing: the gradient. That means the partial derivative of the cost with respect to every weight, often millions of them.

Computing those efficiently is the job of backpropagation. It is nothing more exotic than the chain rule of calculus, applied to the network’s layered structure. The error at the output travels backwards layer by layer. Each layer multiplies the gradient arriving from above by its own local derivative.

One result is worth carrying away, because it recurs throughout the book. Pair a sigmoid or softmax output with the cross-entropy loss, and the gradient of the loss with respect to the pre-activation collapses to

$$\begin{equation} \frac{\partial L}{\partial z} \;=\; \hat{y} - y, \label{eq:pred-minus-target} \quad\text{(8.6)} \end{equation}$$

the prediction minus the target.

A nearly correct neuron, $\hat{y} = 0.9$ against $y = 1$, gets a gentle push of $-0.1$. A confidently wrong one, $\hat{y} = 0.1$ against $y = 1$, gets a hard shove of $-0.9$.

This prediction minus target signal drove the neural language model of Chapter 11 and the classifiers of Chapter 14. It is why cross-entropy and the sigmoid or softmax are always found together.

One training step, all the way through

Nothing makes this concrete like doing it once with numbers. Here is a tiny network with two inputs, two hidden ReLU units, and one sigmoid output.

The parameters start at these values:

The training example is $x = (1, 0)$ with target $y = 1$. The learning rate is $\eta = 0.5$.

Step 1: forward.

Push the input through, one layer at a time.

The network is barely better than a coin flip, which is what an untrained network should look like.

Step 2: backward.

Start at the output and use Equation (8.6).

The ReLU derivative is a gate. Both units have $z > 0$, so both gates are open and multiply by $1$. Had either $z$ been negative, its gate would multiply by $0$. Every weight below it would then get no signal at all on this example.

The signs differ because $v_1$ and $v_2$ have opposite signs. Hidden unit 1 pushed the output up and gets rewarded. Hidden unit 2 pushed it down and gets corrected.

Step 3: update.

Apply Equation (8.5) to each parameter. A negative gradient makes the weight go up.

Step 4: check it worked.

Run the same example forward again with the new weights.

The prediction moved towards the target and the loss fell. That is the whole algorithm. Everything after this chapter is the same four steps, run on larger networks, over more examples, many millions of times.

Generalisation: the whole point

Given enough capacity, a network trained by Equation (8.5) will drive its training cost towards zero. That is not the goal. Mistaking it for the goal is the central hazard of the field.

The goal is generalisation, meaning low error on data the model has never seen.

A model that merely memorised its training examples would score perfectly on the training set and uselessly on anything new. That is overfitting, fitting the noise and idiosyncrasy of the training sample instead of the pattern that also governs future data.

We detect it with the discipline the whole field lives by, which Chapter 14 makes concrete. Hold out a test set the model never trains on, and judge the model only by its error there.

Everything that combats overfitting serves that one end. Regularisation. Early stopping. Dropout. And above all more data.

A neural network is not valuable because it can fit its training data. It is valuable exactly to the extent that what it learned transfers beyond it.

One machine, many chapters

That is the entire apparatus. A neuron. A nonlinearity. A loss averaged into a cost. Gradient descent to minimise it. Backpropagation to supply the gradients. And a test set to keep the whole enterprise honest.

Nothing in the chapters ahead adds a new principle to that list. They add architecture, meaning particular wirings of neurons suited to particular data.

The first use is immediate. Turn this machine loose on the distributional hypothesis of Chapter 6. Train vectors to predict their contexts. Out come the word embeddings of Chapter 9. That is the same dense geometry the counting of Chapter 7 produced, now trained rather than computed.

From there the story is one of ever cleverer architectures, recurrent, gated and attentional, built from these identical parts.

Further reading.

is the paper that made backpropagation and gradient trained multilayer networks widely known. is the standard modern textbook and treats activations, loss functions, optimisation and regularisation in full depth. is a primer on the same material aimed at natural language processing. is a classic study of how initialisation and momentum shape the optimisation this chapter only sketches.