The learning rate
Every step of training walks the dials downhill. How far should each step go - and what happens if it goes too far?
Five points sit on the line y = 2x + 1, and two dials start at zero. Every step walks both dials downhill against their slopes, and one control sets how far. The panels show the verdict, the steps to a loss under 0.01, and the loss per step. Too small crawls; too big overshoots, then blows up.
Board: How It Learns →On a narrow screen the three panels stack. The two dials stay above them, and the table scrolls sideways.
try this Drag How far each step goes from 0.001 up to 0.5 and watch Steps to a loss under 0.01 fall, then vanish.
The lesson's line fit: five points, x = 0 to 4, where the right answer is y = 2x + 1. The slope w and the offset b both start at 0, so the first loss is 33. Each step moves every dial against its slope by the learning rate times that slope. The target is a loss under 0.01, a teaching value.
Verdict
Converge
The loss gets under 0.01 at step 29 and keeps falling, to 5.8 x 10^-5 by step 200. The dials settle on w = 2.005, b = 0.987.
Steps to a loss under 0.01
29
of 200 steps allowed
Fastest notch at this budget: 0.2, in 8 steps.
Where the dials ended
w = 2.005, b = 0.987
the answer is w = 2, b = 1
Loss after 200 steps: 5.8 x 10^-5.
The loss, step by step
Log scale: each gridline is a thousand times the one below. The dashed line is the target.
The slope dial w, against its answer
The loss can fall on every step while w swings across its answer and back. That swing is what oscillation looks like.
The dial, swept
Every notch of the learning rate at the current budget. Same points, same start, same loss; only the step size changes from row to row.
| Learning rate | Loss at step 10 | Steps to 0.01 | Final loss | w crossed 2 | Verdict |
|---|---|---|---|---|---|
| 0.001 | 28.9 | never | 2.26 | 0 | Crawl |
| 0.002 | 25.2 | never | 0.168 | 0 | Crawl |
| 0.005 | 16.7 | never | 0.0131 | 1 | Crawl |
| 0.01 | 8.26 | 145 | 7.2 x 10^-3 | 1 | Converge |
| 0.02 | 1.88 | 72 | 2.2 x 10^-3 | 1 | Converge |
| 0.05 | 0.0269 | 29 | 5.8 x 10^-5 | 1 | Converge |
| 0.1 | 0.0129 | 15 | 1.3 x 10^-7 | 1 | Converge |
| 0.2 | 6.9 x 10^-3 | 8 | 4.8 x 10^-13 | 3 | Converge |
| 0.25 | 0.0179 | 12 | 8.0 x 10^-16 | 9 | Oscillate |
| 0.3 | 40.6 | never | 2,125 | 200 | Diverge |
| 0.5 | 8.8 x 10^8 | never | 1.0 x 10^150 | 200 | Diverge |
At a learning rate of 0.05 the loss gets under 0.01 in 29 steps. Turn it up ten times, to 0.5. How many steps now?
A bigger step is faster, until it is wider than the valley
From the bottom of the dial up, every notch arrives sooner: 145 steps at 0.01, 29 at 0.05, 15 at 0.1, 8 at 0.2. Then it turns. At 0.25 it takes 12, because the slope dial overshoots its answer and swings back 9 times on the way down. At 0.3 the loss is 40.6 by step 10, above where it started, and at 0.5 it is 8.8 x 10^8: the lesson's 875 million.
The edge can be computed. Each step multiplies the distance along the valley's steepest direction by one minus the rate times that direction's steepness. For these five points that steepness puts the edge at 0.298. Below it, every step lands closer. Above it, every step lands further out on the other side. A real network has billions of directions, so in practice you try a few rates and watch the loss curve.
Tiny steps never fail, and never finish
At 0.001 nothing goes wrong. After 200 steps the loss is 2.26, down from 33, and the dials are at w = 1.542, b = 0.558. Shrink the budget and the slow notches drop out one by one: at 50 steps even 0.02, which arrives at step 72 with 200, is still short. A slow rate can still run out of time.
Limits of this model
Five points and two dials. The landscape here is a smooth bowl with one bottom. A real network's loss has billions of dials, flat stretches, ridges and many good-enough bottoms, so the same rate can be fine in one region and too big in the next.
Every point, every step. This is full-batch gradient descent: each step uses all five points, so the run is exactly repeatable. Real training uses a batch, a random handful of examples per step, and its loss curve is noisy around the smooth one drawn here.
One fixed rate. Real optimisers change it as they go (warm-up, then decay), and Adam gives every dial its own rate. Here the rate stays fixed, so the dial is the only thing that changes.