Attention by Hand

The Attention and the Transformer board draws each word looking back at the words before it. This lesson builds that look by hand, then trains it: the fourth of six lessons that end with a small GPT built from scratch. A GPT is the kind of model behind ChatGPT: it writes text one piece at a time, each time guessing what comes next.

Before you start. A weight is one adjustable number inside a model, and training nudges every weight to make the loss smaller (Autograd by Hand). The loss is one number for how wrong the model’s guesses were, and lower is better. In The MLP Language Model a model read the last eight characters of a returns-desk reply (a short customer-service reply about a returned order) and guessed the next one. That model was an MLP, a multilayer perceptron: layers of neurons stacked one after another. From The Maths Floor, Part 2: Probability, Softmax and Temperature: softmax turns a list of scores into shares that add up to 1. From Vectors, Dot Products and Matrix Multiply: multiply two lists of numbers pair by pair and add the products, and you get one number that is big when the lists match. From Attention & the Transformer: attention lets each word take a share of its meaning from each word before it.

The idea in 30 seconds

A long-range dependency is when the next character depends on something further back than a model can see. Example: in “it is past the 30-day window, so we cannot take it back”, the word after “so” depends on “past”, about 28 characters earlier. A model that reads only the last eight characters guesses that word right half the time.
Self-attention lets every position in a text take information from any earlier position. Each position has three lists of numbers, made by learned weights: a query (what it is looking for), a key (what it has), and a value (what it hands over when chosen).
Multiplying a query by a key pair by pair and adding gives a score. Divide by the square root of the list’s length, then softmax turns each position’s row of scores into shares that add to 1.
The causal mask sets every score a position gives to a later position to minus infinity before softmax, so its share is 0. Training shows the model the whole text at once, later characters included, so without the mask it copies the answer it is meant to guess.
Each position’s output is every earlier value times its share, added up. A head is one set of queries, keys and values; a model with several heads can look in several places at once.
Attention helps where the deciding word is far back. After “window, so ”, a model reading only the last eight characters gave the right next character a 0.50 chance; with attention added, 0.86. Of 100 replies each wrote, 38 were correct without attention, 90 with it.

Read the story · Skip to the numbers