The Bigram Model

The From Transformer to LLM board draws the machine that fills in the next blank. This lesson builds the smallest one: the second of six lessons that end with a small GPT built from scratch.

Before you start. A weight is one adjustable number inside a model. Its slope says how much the loss, the model’s wrongness, moves if you nudge that weight. Training nudges every weight a little against its slope, over and over. In Autograd by Hand you built code that finds every slope automatically, by remembering how each number was made and walking that record backwards. From The Maths Floor, Part 2: Probability, Softmax and Temperature: a probability is a share between 0 and 1, the shares add to 1, and sampling draws an outcome in proportion to its share. Also from The Maths Floor, Part 2: Probability, Softmax and Temperature: softmax turns a list of scores into probabilities. From Tokens & Embeddings: every piece of text gets an id, and an id can pick one row out of a table of numbers.

The idea in 30 seconds

A bigram model guesses the next character from the single character before it, using a table of how often each character followed each other one.
Dividing each row of counts by its total turns it into chances, and drawing from those chances, one character after another, writes new text that is right in pairs and wrong beyond them.
A language model is scored by the chance it gave to the real text: take −log of each chance, which is 0 for a sure thing and grows as the chance shrinks, and average. That loss is the average negative log-likelihood.
One pair the table never saw makes the loss infinite, and smoothing, adding a small count to every cell, makes nothing impossible at a small cost.
The same model can be one grid of adjustable numbers, trained by nudging each downhill. Turning its scores into chances (softmax) and scoring them the same way (together, cross-entropy) lands on the counting’s answer.
Counting needs a row for every possible past, so looking further back needs a learned formula with adjustable numbers, not a bigger table.

Read the story · Skip to the numbers