What a Neural Network Is

WHAT A NEURAL NETWORK IS How LLMs Work · 01 — a LEARNED function: neurons, layers, and why non-linearity is the whole trick traditional code: a human writes the RULES. a neural net: you show it EXAMPLES and it LEARNS the mapping (input → output). an LLM (like the one you're reading) is one enormous version of this. this track opens that black box, no hand-waving. 1 · THE NEURON (the atom) one neuron = a WEIGHTED SUM of its inputs + a bias, passed through an activation. that's the whole unit: output = activation( w1·x1 + w2·x2 + w3·x3 + b ) x1 x2 x3 Σ + act w1 w2 w3 output the WEIGHTS (w) are what the network LEARNS — the knobs. big weight = 'this input matters a lot'; near-zero = 'ignore it'. the bias (b) shifts the threshold. one neuron is basically a tiny voter that weighs its evidence and speaks up or stays quiet. (this is loosely inspired by a brain neuron firing — but it's really just arithmetic.) 2 · LAYERS & THE FORWARD PASS wire neurons into LAYERS: input → hidden layer(s) → output. each connects to all in the next (fully-connected). input hidden layers output the FORWARD PASS: data flows left → right, each layer transforming it. that's just a matrix multiply + activation, per layer. DEEP = many hidden layers (hence 'deep learning'). more layers → more abstract features built from simpler ones. vision: pixels → edges → shapes → objects. language: characters → words → phrases → meaning. each layer doesn't see the raw input — it sees what the layer BELOW already figured out. abstraction, again (l.03 gates). a small net like this is a Multi-Layer Perceptron (MLP); it's still the feed-forward block inside every transformer (l.04). 3 · WHY NON-LINEARITY IS THE WHOLE TRICK (the activation function) here's the catch: without an activation, stacking layers is POINTLESS — a chain of matrix multiplies collapses back into ONE matrix = a single linear function. 100 layers would equal 1. you'd have learned nothing deep. the ACTIVATION adds a KINK — a non-linearity — so layers DON'T collapse, and the net can bend to fit any shape: 0 ReLU = max(0, x) ReLU: pass positives, zero out negatives. dead simple, and the modern default. (older: sigmoid, tanh — S-curves.) UNIVERSAL APPROXIMATION: with enough neurons + non-linearity, a net can approximate ANY continuous function. that's the theorem that makes all of this possible — the weights pick WHICH function. so: linear layer stretches/rotates the space, activation FOLDS it. stack enough stretch-and-fold and you can carve out astonishingly complex decision boundaries. that folding is where the 'intelligence' geometrically comes from. 4 · A NET = A GIANT TUNABLE FUNCTION the mental model that makes everything else click: a neural net is one big mathematical function with millions–BILLIONS of knobs (weights). 'learning' = turning the knobs so the function maps inputs → the right outputs (that's l.02). · PARAMETERS = all the weights + biases. GPT-scale LLMs have 10⁹–10¹² of them. · a layer = MATRIX MULTIPLY (weights) + add (bias) + activation → this is WHY GPUs run AI: they're matrix machines (How-Computers-Work l.09) — thousands of multiplies in parallel. · the weights aren't meaningful individually — the FUNCTION they collectively define is. everything ahead — embeddings, attention, transformers, an LLM — is a clever ARRANGEMENT of these same neurons, trained on enough data. the atom never changes; only how you wire it and what you feed it. NEXT (l.02): the knobs start RANDOM and the net is useless. how does showing it examples tune billions of weights to something that works? → gradient descent & backpropagation. JARGON, DECODED neuron = weighted sum of inputs + bias, through an activation · weight = a learned knob (input importance) · bias = a learned offset · activation function = the non-linearity (ReLU / sigmoid / tanh) layer = a row of neurons · input / hidden / output layers · fully-connected (dense) = every neuron wired to the next layer · forward pass = compute output from input · deep learning = many hidden layers · MLP = multi-layer perceptron non-linearity = the kink that stops layers collapsing into one · ReLU = max(0,x) · universal approximation = enough neurons can fit any function · feature = a learned intermediate representation (edges, words) parameters = all weights + biases (LLMs: billions) · matrix multiply = a layer's core op (→ GPUs) · 'a net is a tunable function' = the whole mental model · training (l.02) = finding the weights