How a neural network learns: the error that travels backward

A newborn neural network is a mess. Its weights are set at random, so if you show it a photo of a cat it answers "toaster" with the same confidence as "cat" or "eclipse." It knows nothing. And yet, a few hours later, that same network gets it right nine times out of ten. No miracle happened in between: what happened was a method to measure how far off it was and hand out the blame for that error across all of its numbers. That method is called backpropagation, and it is the reason why how a neural network learns is no longer a mystery today, but a calculation.

What backpropagation is and how a neural network learns, said plainly and without frills: the network makes a prediction, compares its answer with the correct one, measures the gap between them, and then walks that gap backward through the whole network to know which weight to nudge, in which direction and by how much. Learning is correcting course, one small step at a time.

01 · the errorFirst you have to measure how far off you were

Before you correct anything, you have to put a number on the failure. When the network gives an answer, that answer is compared against the known truth of the example, and out of that comparison comes a single value: the loss (the loss, in the field's own term). It is a grade in reverse: the further the prediction strays from the correct answer, the higher the loss. Zero loss would be a perfect hit.

Picture it as a distance. The correct answer is a point on the map; the network's answer, another point. The loss is how many meters lie between the two. All of learning consists of bringing those points closer, and to bring them closer you first have to know which way to pull. That direction is not guessed: it is calculated.

Learning is not getting it right on the first try. It is measuring the error precisely and knowing exactly which way to move to shrink it.

02 · the slopeThe derivative says which way to go down

Here is where the elegant idea hiding behind the ugly name comes in. Think of the loss as a landscape of mountains and valleys. Each weight of the network is a coordinate; move it a little and the loss rises or falls. The network stands at some point on that terrain, and it wants to reach the bottom of the valley, where the loss is minimal. How does it descend without seeing the whole map? By feeling the slope beneath its feet.

That slope has a name: the gradient. It is the derivative of the loss with respect to each weight, and it answers a very concrete question: if I move this weight a touch, does the loss rise or fall, and how strongly? With that answer the network takes a step in the direction opposite to the slope (downhill) and repeats. That patient descent is called gradient descent [1]. The size of each step is the learning rate: big steps descend fast but overshoot; tiny steps are safe but slow.

03 · backwardBackpropagation: sharing the blame layer by layer

The central piece is still missing, the one that gives it its name. The error is measured at the end, in the output layer. But the blame does not belong to the last layer alone: it has been dragged along from the start, because each layer fed the next. To correct it well you have to know how much each weight contributed to the error, even the ones at the bottom. And that accounting is done in reverse: you start from the final error and propagate it backward, layer by layer, all the way to the input.

The mathematical trick that makes it possible is the chain rule from calculus: to know how much a distant weight influences the final error, you multiply the influences of each intermediate link. Backpropagation is, at bottom, applying that rule in an orderly way and reusing calculations, so that with a single pass forward and another backward all the gradients are computed [2]. Without that order, the calculation would be impractical; with it, it is almost cheap.

The idea did not fall from the sky. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams published in Nature the paper that taught the world to use backpropagation to adjust the weights of networks with several layers [2]. They did not invent the chain rule, which is centuries old, nor were they the first to brush against the idea, but they showed that with it a deep network could learn useful representations on its own. That work broke decades of stagnation and is, in a straight line, the ancestor of almost everything we call artificial intelligence today.

04 · the cycleRepeat the course until the network gets it right

Put the three pieces together and you have the full heartbeat of learning. The network looks at a batch of examples and predicts (forward). It measures its loss. It propagates the error backward and obtains, for each weight, its gradient. It adjusts each weight one small step in the direction that lowers the loss. And it starts over with the next batch. Each full sweep over the data is called an epoch, and many are needed.

Seen from afar, it is one same gesture repeated to exhaustion: predict, measure, correct course. There is no instant of sudden understanding; there is a trickle of millions of tiny corrections that, added up, leave the weights exactly where the network gets it right. What the network "knows" in the end is not an idea: it is the exact position of its weights, sculpted by the error that kept pushing it downhill.

The network does not understand all at once. It descends the mountain by touch, one corrected step at a time, until the valley is the right answer.

And that is the whole secret, without the ugly name getting in the way. Backpropagation is nothing more than the honest accounting of error: each weight receives the share of blame it is due and moves just enough to do better next time. Repeat it enough and a table of random numbers ends up recognizing a face, translating a language or writing this sentence. What is astonishing is not that the machine learns: it is that learning, looked at up close, turns out to be no more than knowing how to measure an error and walk backward to hand it out.