What deep learning actually is: backprop, and why language was hard

Here is a program that converts miles to kilometres. One line, one constant, and you knew the constant before you started. Now write one that decides whether a photograph contains a cat. Same shape, same input and output. Nobody can write that line. The difference is a number you do not know. Call it a parameter. Instead of writing the constant, you leave a blank in the program and let a search fill it in. A modern model is that same trick, repeated billions of times. The smallest learnable piece looks like this. Take the inputs, multiply each by its own weight, add them up, add one bias. Those weights and that bias are the parameters. Everything else in the expression you wrote by hand. One unit, a handful of dials. Stack those units into layers and the whole network is still one function. Data in, prediction out, and a very large vector of parameters sitting in the middle. Call that vector theta. Every question about training is a question about theta. Nothing else moves. So what tells you which theta is right? Examples. Thousands of inputs paired with the answer you wanted. The pairs are the specification. You never state the rule; you state the behaviour, and the rule is what gets fitted to it. That inversion is the whole idea. Training is a search over theta, and the search is a loop. Predict. Measure how wrong you were. Change every parameter a little in the direction that makes it less wrong. Repeat. The next two chapters are what those middle two steps mean. Picture standing on a hillside in thick fog. You cannot see the valley. But you can feel which way the ground tilts under your feet, and you can take one step downhill. Do that repeatedly and you get somewhere low. That is gradient descent, and the rest is arithmetic. First the height. Loss is one number saying how wrong the model is on the data. For a number you are predicting, squared error. For a choice between classes, cross entropy, which punishes confident and wrong far harder than unsure and wrong. Either way: one number. Now the landscape. Every parameter is an axis, and the loss is the height above that point. Two parameters give you a surface you can draw. A billion parameters give you the same object in a billion dimensions, which you cannot draw and never need to. The gradient is the tilt. For each parameter, a partial derivative: if I nudge this one dial up, does the loss go up or down, and how fast? Collect one of those per parameter and you have a vector pointing straight uphill. So walk the other way. Descent is one line. Subtract a small multiple of the gradient from every parameter at once. That multiple is the learning rate. Too large and you bounce out of the valley you were in. Too small and you are still descending next week. Computing the exact gradient means running every example in the dataset. Nobody does that. You take a batch of a few hundred, average their gradients, and step on that estimate. It is noisy, and the noise is useful: it shakes you out of shallow dips. A part comes off the end of a production line and fails inspection. To fix it you have to walk back up the line and work out how much each station contributed to the fault. Backpropagation is that walk, done exactly, with calculus. Write the network as a graph. Each node is one small operation: a multiply, an add, an activation, the loss at the very end. The forward pass runs left to right, computing values, and it keeps every intermediate result, because the way back needs them. The backward pass is one rule applied over and over. The chain rule. If the loss depends on b, and b depends on a, then the sensitivity of the loss to a is the two sensitivities multiplied. Blame flows backwards as a product. So every node does the same tiny job. It receives one number from the node in front: how much the loss changes when my output changes. It multiplies that by its own local derivative, and hands the result to the nodes behind it. When that reaches a weight, you have its gradient: how much this one dial is responsible for the error. Every weight in the network gets one, in a single sweep. Then the optimiser subtracts a small step, and the forward pass runs again. This is why any of it is affordable. The obvious alternative is to nudge one parameter, rerun the model, and see what happened, which costs one full run per parameter. Backpropagation gets every gradient in one backward sweep, at roughly the cost of the forward pass. Look at what a vision model learns, layer by layer. The first layer finds edges. A few layers later, corners and textures. Later still, wheels and windows and faces. Nobody designed that ladder. It is what falls out of stacking the same simple operation. A layer does not classify anything. It re-describes its input. Pixels go in, and what comes out is a different set of numbers, in which the thing you actually care about is easier to separate. That output has a name: a representation. Depth is representations stacked. Stack them and the descriptions compose. Early layers learn things nearly every task needs, so they transfer: take a trained network, replace the last layer, and it learns your task from a fraction of the data. Late layers are the ones that are specific to the job. Now the detail that makes the whole stack legal. A layer without an activation is a matrix multiply. Two of them in a row is one matrix multiplied by another, and the product of two matrices is just another matrix. Two layers, one layer. Identical. That argument does not stop at two. A hundred stacked linear layers collapse to a single matrix. All that depth, all those parameters, and the model can only draw a straight line. The fix is to bend the signal between every pair of layers. The bend can be almost anything. In practice it is the rectified linear unit: pass positives straight through, flatten everything negative to zero. One kink. But a kink cannot be folded into a matrix, so the layers stop collapsing and depth finally buys you something. Imagine a whisper passed down a line of a hundred people, where each one is told to double it or halve it. By the end you have a scream or silence. A deep network does exactly that, to its signal going forward and its gradients coming back. So start it right. Set the weights to zero and every unit in a layer computes the same thing forever. Set them too large and activations explode. Initialisation schemes scale the random starting weights by the number of inputs, so the variance stays roughly constant all the way down. Then keep it right while it moves. Normalisation sits between layers and rescales the activations back to a sane mean and spread, with two learned parameters so the layer can undo it if it wants. Training gets faster and tolerates a much larger learning rate. Plain descent treats every parameter the same. Momentum adds a running average of past steps, so you keep rolling through flat stretches. Adam goes further and gives each parameter its own step size, scaled by how big its gradients have recently been. It is the default for a reason. Now the failure that looks like success. Keep training and the loss on your training data keeps falling. On data the model has never seen, it falls, bottoms out, and starts climbing. The model stopped learning the pattern and started memorising the examples. Only held out data shows it. Regularisation is everything you do about that. Weight decay pulls parameters toward zero, so the model prefers a simpler fit. Dropout switches off random units during training, so no single unit becomes load bearing. Early stopping just stops at the bottom of the held out curve. More data beats all three. Take an image. Two neighbouring pixels are nearly the same colour, and a slightly darker photo is a slightly different set of numbers. Now take a dictionary. Word two hundred and word two hundred and one have nothing whatsoever to do with each other. That is the first problem. Vision arrives as numbers with distance already built in. Language arrives as symbols from a list, where every pair is equally far apart. So you learn the distance: give each token a vector of numbers, and train those vectors along with everything else. Second problem: length. An image crops to a fixed grid. A sentence is four words or four hundred, and the network has to accept both. The answer for years was to process one token at a time, carrying a hidden state forward. A recurrent network. Third problem, and the hard one. The word that decides the verb can be three hundred words back. Backpropagation through a recurrent network multiplies the same term once per step, so a signal three hundred steps old is that factor raised to the three hundredth. It vanishes, or it explodes. Gated units bought some back. An L S T M carries a separate memory line that is added to rather than multiplied through, so the gradient has a path that survives. What it could not fix was speed: every step still waits for the one before it. Attention removed both limits at once. Every position looks directly at every other position, in one step, so a dependency three hundred words back is one hop instead of three hundred. And because the positions do not wait for each other, the whole sequence runs in parallel on a G P U. So: parameters are the numbers you do not write. Loss says how wrong you are, the gradient says which way is down, and backpropagation hands every weight its share of the blame. Depth builds representations, the nonlinearity keeps them from collapsing, and language needed attention to arrive.