Backpropagation

1 · What it is

A neural network makes a prediction by pushing numbers forward through its layers. A loss scores how far that prediction landed from the right answer. Backpropagation then asks a question about every weight: if this one moved a little, how much would the loss change? That number is the weight’s gradient.

The trick is to start at the end. The calculation starts at the last layer and works back to the first. The chain rule from calculus lets each layer combine the slope handed back to it with its own local slope, then pass the product one layer further down. You rarely write this yourself. TensorFlow records the forward operations and walks them in reverse to get the gradients. PyTorch does the same for any function that ends in a single number.

The idea is older than its fame. Backpropagation is a special case of a wider family of techniques called automatic differentiation. Surveys often credit Linnainmaa’s 1970 work as the earliest account of the reverse method in print. In 1974 Werbos wrote a formal version for systems that move in discrete time steps. Machine-learning researchers reinvented it several times. A 1986 Nature paper by Rumelhart, Hinton and Williams made it famous for training neural networks. That paper showed hidden units learning useful features of the task on their own.

On this page, backpropagation only measures. In a library such as PyTorch, the backward pass fills in the gradients, and only afterwards does an optimiser step use them to move the weights. Backpropagation is what makes gradient descent practical for networks with many layers. Some guides use the word backpropagation for the whole training procedure, weight updates included. Backpropagation also has a weak spot. In deep networks, gradients for the layers near the input can shrink towards zero. Those layers then barely learn, or stop learning. With very large weights, the gradients can instead grow too large for training to settle.