Non-linearity: why a hundred layers can equal one

Stack a hundred layers in a neural network. Forget one tiny detail, and all a hundred of them collapse into a single one. You would have learned nothing at all. Start with the atom. One neuron takes its inputs, multiplies each by a weight, adds a bias, and pushes the total through an activation. The weights are the knobs, and they are the only part the network learns. Wire those neurons into layers, and data flows left to right. Each layer never sees the raw input. It only ever sees what the layer below already worked out. Pixels become edges, edges become shapes, and shapes become objects. But here is the catch. A layer is really just a matrix multiply. And a matrix times a matrix is only ever another matrix. So three layers, ten layers, a hundred layers, all of it folds straight back down into one. The fix is almost insultingly small. After each layer, bend the line. Keep the positives, and flatten everything below zero. That single kink is what stops the layers collapsing. Now the geometry changes. The linear part stretches and rotates the space. The kink folds it. Stretch, fold, stretch, fold, and you can carve out a boundary of very nearly any shape you like. That folding is where the intelligence actually comes from. So a neural network is one enormous function with billions of knobs. Learning is nothing more than turning them. Everything after this, attention, transformers, all of it, is a rearrangement of the same unit.