One Knob, One Slope

The How It Learns board draws the whole training loop: loss, slope, step, repeat. This lesson takes the smallest slice of it, one neuron with one input and one target, and runs the loop by hand with real numbers until “gradient” stops being a word and becomes a direction you can point at.

Before you start. You need How It Learns (loss, gradient descent, backprop), which gave you the loop: boil wrongness down to one number called the loss, find each dial’s slope, and step every dial against its slope by an amount the learning rate sets. From What a Neural Network Is you need one more thing: a neuron multiplies its input by a dial called a weight, adds up the result, and passes the total on.

The idea in 30 seconds

A weight is a knob: a plain number the input is multiplied by. The bias is a second knob that adds instead. In 1960 the knobby ADALINE had real ones, turned by hand.
The loss says how wrong one answer is, as a number that is never negative: the miss, squared.
A knob’s derivative is how fast the loss changes as you turn it. Nudge it, measure, divide, and you have it.
The formula gives the same number without a second run: how wrong, times how steep the sigmoid is there, times the input.
The gradient is one derivative per knob, and training turns every knob against its own, over and over.
The sigmoid squashes any total into 0 to 1, and its flat ends make the last turns tiny, so training slows as it closes in and never quite arrives.

Read the story · Skip to the numbers