RNNs and LSTMs

The Attention and the Transformer board opens on a translation machine that reads one word at a time and forgets the beginning of long sentences. This lesson is that machine, and the repair that nearly saved it.

Before you start. From What a Neural Network Is you need one unit: inputs multiplied by weights, added up, then passed through an activation. From How It Learns (loss, gradient descent, backprop) you need backpropagation: when the answer is wrong, the blame is handed backwards through the network, multiplied by each weight it crosses on the way. From Tokens & Embeddings you need one fact: every word becomes a list of numbers before the machine sees it.

The idea in 30 seconds

A recurrent neural network (RNN) reads text one word at a time and carries a short running summary of everything so far, a fixed-size list of numbers, into the next word. It uses the same learned settings at every word, which is why it can read text of any length.
Training sends a correction back from the answer to every learned setting that helped produce it. In an RNN that correction passes back through every earlier word, and is multiplied by roughly the same factor at each one.
If that factor is under 1, the correction shrinks towards zero over long stretches of text. This is the vanishing gradient: the network still trains, but cannot learn to link words that sit far apart.
The LSTM adds a second running note, the cell state, which is only added to or crossed out, never rewritten. The correction travelling back along it is multiplied by nearly 1 at each word, so it survives.
Three gates, each a learned number between 0 and 1, decide how much of that second note to keep, how much of the new word to write onto it, and how much of it to pass on.
Sequence to sequence translation chains a reading LSTM to a writing LSTM through one fixed-size summary of the whole sentence. That single summary is the limit removed by the next lesson’s fix, attention, which lets the writer look back at every word.

Read the story · Skip to the numbers