Sampling and What Scale Buys

The From Transformer to LLM board draws the machine that fills in the next blank. This lesson makes the small one write, then counts the distance to a big one: the last of six lessons that build a small GPT from scratch.

Before you start. In The GPT Block you trained a small GPT, the kind of model built in the last two lessons, which writes one character at a time. Trained on 3,000 short returns-desk replies, it scores each of the 51 characters that could come next. From The Maths Floor, Part 2: Probability, Softmax and Temperature: softmax turns scores into chances that add up to 1. Sampling draws one outcome in proportion to its chance. Temperature divides every score first. From Tokens & Embeddings: a real model reads tokens, chunks of text about four characters long. A weight is one number inside the model. fp16 stores each weight in 2 bytes, so 7 billion weights take 14 GB. From The GPU (thousands of cores, and the memory that feeds them): an A100 is a GPU chip for AI work, and it reads its memory at 2 TB/s.

The idea in 30 seconds

A language model only scores the next piece of text. Writing is a loop: start from the prompt, the opening text you hand it, choose one piece, add it to the end, and ask again.
Greedy decoding, always taking the top score, gives one prompt the same reply every time. Drawing by chance gives variety; temperature, a number every score is divided by before the draw, sets how bold it is.
Top-k sampling (2018) keeps only the k highest-scoring choices and draws among them, so a very unlikely piece can never be picked.
A real model reads tokens, pieces of text about four characters long. Llama 2 7B chooses among 32,000 and trained on 2 trillion, about 27 million times this lesson’s whole training text.
Training costs about 6 sums per weight (one number inside the model) per token read: 8.4 × 10²² for 7 billion weights on 2 trillion tokens, 8.5 years of one A100 AI chip at full speed.
Writing one token with a 7-billion-weight model needs 45 µs of maths on an A100 but 7 ms to read its 14 GB of weights from memory, so the chip mostly waits: memory, not maths, sets the pace.

Read the story · Skip to the numbers