Looking Inside the Model

The Tokens and Embeddings board ends on a map where directions mean things. This lesson hunts for such arrows deep inside a trained model, where nobody drew or labelled them.

Before you start. What a Neural Network Is showed that a network is layers of neurons, each multiplying inputs by weights and adding them up. Training sets billions of those weights, and nobody writes them. Tokens & Embeddings showed that each word is stored as a list of numbers, and moving in one direction through that list changes one meaning, such as male to female. Precision and Recall (where to set the spam filter's bar) built a spam filter: a weighted sum, squashed into a 0-to-1 score, then a threshold that says yes or no. Attention & the Transformer showed a transformer is a stack of layers, each handing the next a row of numbers per word.

The idea in 30 seconds

A trained model is opaque because none of its numbers is labelled, not because they are hidden.
A probe is a small classifier trained on a layer’s numbers; beating a coin flip means the concept is readable there, not that the model uses it.
A feature is one concept a model tracks, stored as a direction across many neurons, not as one neuron.
Superposition packs more features than neurons by overlapping their directions, so one neuron fires for several unrelated things.
A sparse autoencoder unpacks those overlapping directions into a much longer list of candidate features, where each entry fires for roughly one thing.
Steering adds a feature’s direction while the model runs; the behaviour changes, which proves the feature matters.

Read the story · Skip to the numbers