Why the Loss Won’t Go Down

The How It Learns board draws how a network learns. This lesson is what to do when the learning stalls.

Before you start. From What a Neural Network Is: a neuron is a unit that adds up its inputs, each multiplied by a weight. A weight is one adjustable number inside the network, a dial. Neurons sit in rows called layers. From How It Learns (loss, gradient descent, backprop): the loss is one number for how wrong the network is. Also from How It Learns (loss, gradient descent, backprop): training nudges every weight, over and over, to make the loss smaller, like walking downhill on a landscape whose height is the loss. From One Knob, One Slope: a weight’s slope says which way and how fast the loss changes if you nudge that weight, and the gradient is all the slopes together. The learning rate is how far each nudge goes.

The idea in 30 seconds

A misconfigured network rarely crashes; it learns worse. The loss curve, the loss printed every step, is how you find out.
The learning rate, how far each nudge goes, decides a lot: too small and the loss crawls down for ages, right and it falls fast, a bit too big and it jumps up before recovering, far too big and the numbers overflow into nan, “not a number”.
Adam, a standard training method (2014), gives every weight its own step size: its recent slopes set the direction, their typical size shrinks the step. It costs two extra numbers per weight.
Initialisation sets the size of the random starting weights. Too small and the signal fades to nothing across many layers, too big and it explodes; a size of √(2 / inputs per neuron) keeps it steady.
Layer normalisation resets each layer’s outputs to a standard level on every pass, so the network stays trainable as weights move.
Dropout switches off random neurons during training so none can memorise alone, which keeps the validation loss (loss on unseen examples) falling for longer.

Read the story · Skip to the numbers