The MLP Language Model

The Tokens and Embeddings board draws text turned into lists of numbers. This lesson learns one for every character: the third of six lessons that end with a small GPT built from scratch.

Before you start. A weight is one adjustable number inside a network; training nudges every weight, over and over, to shrink the loss, one number for how wrong the guesses were. Training finds, for every weight at once, which way to nudge it (Autograd by Hand builds the code that does this). In The Bigram Model a model read 3,000 returns-desk replies and guessed each next character from only the one before it, turning 51 scores (one per possible character) into chances with softmax; its loss on replies it never trained on was 1.97, where knowing nothing scores 3.93. From Tokens & Embeddings: an embedding is a short list of numbers that stands for one piece of text, kept as one row of a table and picked out by that piece’s id. From What a Neural Network Is: a hidden layer is a row of neurons between the input and the answer, each adding up its inputs times weights plus a bias, an extra adjustable number that shifts where the neuron fires. From Why the Loss Won't Go Down: the learning rate is how far each nudge goes, and the loss curve, the loss printed as training runs, shows whether it is right.

The idea in 30 seconds

A model that looks several characters back cannot use a table of counts: it needs a row for every possible run of characters (about 46 million million for the last eight characters), almost all empty. That blow-up is the curse of dimensionality.
An embedding table gives each character a short list of learned numbers; the model reads its context, the last eight characters, by looking up their eight lists and laying them side by side.
A hidden layer (neurons that each add up their inputs times weights) followed by tanh, a squash into −1 to 1, turns those lists into one score per possible next character; softmax turns scores into chances. That stack is an MLP, a multilayer perceptron, published by Bengio and colleagues in 2003.
Validation loss is the loss on text held back from training. On the practice text of 3,000 short customer-service replies the MLP scores 0.18, against 1.97 for a model that looks one character back.
The learning rate, how far each training nudge goes, is chosen by reading the loss: too small barely moves, too big ends worse than where it started.
Characters that play the same part end up with similar embeddings: after training the ten digits end up with nearly the same list of numbers, though no one said they were digits.

Read the story · Skip to the numbers