Attention & the Transformer

Attention and the Transformer · Open board

The keystone board of the How LLMs Work track. It tells the story of a translation machine that forgot the beginning of every long sentence, and of the fix that became modern AI.

Before you start. Tokens & Embeddings turned every word into a list of numbers, each looked up on its own, so the “bank” in river bank and the “bank” in bank loan get the identical list. You also want the one unit from What a Neural Network Is: each input is multiplied by its own dial and the results are added up. Nothing in the story half is harder than that multiply and add.

The idea in 30 seconds

The old design squeezed a whole sentence into one fixed-size note, the bottleneck, and long sentences did not survive the squeeze.
Attention is the fix: keep every word, and let whoever is working now decide what share of its meaning to take from each.
Each word brings a query, what I am looking for; shows a key, what I am about; offers a value, what I contribute if chosen. All three are learned.
Scores become shares adding to 1, and every word rebuilds itself as a blend of the other values in those shares.
Dropping the read-in-order part left the transformer: attention alone, parallel on GPUs, with distance free.
The causal mask hides the future, which makes it a generator; attention costs n², which makes a context window a number.

Read the story · Skip to the numbers