Autograd by Hand

The How It Learns board draws the loop: loss, slope, step, repeat. This lesson writes the part that finds the slopes, in Python: the first of six lessons that end with a small GPT built from scratch.

Before you start. A weight is one adjustable number inside a network. The loss is one number for how wrong the network’s answer is. A weight’s slope says how much the loss moves when you nudge that weight; one slope per weight, all together, is the gradient. In One Knob, One Slope you found one slope by hand: nudge the weight, see how much the loss changed, divide by the nudge. In How It Learns (loss, gradient descent, backprop) a number’s slope is its blame, its share of the wrongness. From What a Neural Network Is: a neuron multiplies each input by a weight and adds them up, plus one extra number called the bias. Also from What a Neural Network Is: XOR is a door with two switches that opens when exactly one is on, which one neuron cannot learn and a hidden layer, a row of neurons in the middle, can.

The idea in 30 seconds

Autograd finds every slope on its own: each number records what it was made from, and one backward walk hands each number its blame, the slope of the loss with respect to that number.
Each operation knows its own small slope: addition passes blame on unchanged, multiplication hands each input the other input times the blame.
The multiplication rule is why a weight’s gradient includes its input: an input of zero gives its weight none.
A hidden neuron’s blame is the weight above it times the blame above that, which is how training reaches weights that never touch the answer.
Blame from every use of a number is added, so the walk visits every number only after everything built from it.
A gradient check compares every slope with nudging the number and dividing the change in the loss by the nudge, because a wrong rule never crashes; it only trains worse.

Read the story · Skip to the numbers