The GPT Block

The From Transformer to LLM board draws the stack of identical layers inside a language model. This lesson builds that stack at toy size: the fifth of six lessons that end with a small GPT built from scratch.

Before you start. A weight is one adjustable number inside a model, and training nudges every weight, again and again, to shrink the loss, one number for how wrong the guesses were. Validation loss is that same score measured on text held back from training; in The MLP Language Model a model that read the last eight characters of a returns-desk reply scored 0.1784 on it. From Tokens & Embeddings: an embedding is a short list of numbers that stands for one piece of text, looked up by its id. From What a Neural Network Is: a hidden layer is a row of neurons between input and answer, each adding up its inputs times weights, and ReLU after it turns every negative total into zero. From The Maths Floor, Part 2: Probability, Softmax and Temperature: softmax turns a list of scores into shares that add to 1. From Attention by Hand: in attention, each position in the text scores every earlier position, turns the scores into shares, and takes a blend of what those positions offer, weighted by the shares; each copy running side by side is a head.

The idea in 30 seconds

Stacking more layers can make a network worse, even on the examples it trained on. The fix is a residual connection: each layer adds a correction to its input instead of replacing it, so a layer that learned nothing changes nothing.
In a GPT, each spot in the text (a character here, a word-piece in a real model) carries one list of numbers from input to output, and every layer adds its correction into it. That running list is the residual stream.
Layer norm hands each layer a rescaled copy of the list: subtract the average, divide by the typical distance from it. A list and ten times it give the same copy, so every layer reads at a steady size however much the list grows.
A feed-forward layer is a small ordinary neural-network layer run on each spot alone. Attention, where each spot takes a weighted blend of the earlier spots, is the only step where spots share anything.
A transformer block is attention then a feed-forward layer, each reading a rescaled copy and adding its result into the list. A GPT is that block stacked: 12 deep in OpenAI’s first GPT (2018), 4 deep in this lesson’s toy.
The 4-block toy, trained on short customer-service replies, gives the right next character a chance of 0.85 on average. It ties a simpler model and pulls ahead only where the clue sits far back: 0.98 against 0.86.

Read the story · Skip to the numbers