Attention Is All You Need
A translation model with the recurrence taken out, which turned out to matter far more than the translation.
Read the original ↗This is a paper about machine translation. It contains no language model, no scaling law, no chatbot and no claim about what happens if you make it a thousand times bigger. It is worth reading precisely because of how little it claims.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardAttention and the Transformerthe architecture, drawn block by block
- BoardTokens and Embeddingswhat the model is given before any attention happens
- BoardConcurrency and the Hardwarethe ceiling the sequential part imposes, which is the paper's real motive
- LabTokens and embeddingsthe input side, run on your own sentence
- LabThe batch size dialthe serving cost the paper never had to pay
- LabThe parallel ceilingwhy removing a sequential dependency is worth this much trouble
- VideoAttention from scratch: the mechanism that ate the fieldthe mechanism itself
- VideoAttention in production: KV cache, MQA, GQA and MLAwhat the industry did with it afterwards
- VideoRNNs and LSTMs: the memory that almost workedthe recurrent models the paper is replacing
- LessonAttention & the TransformerUses the board Attention and the Transformer
- LessonRNNs and LSTMsUses the board Attention and the Transformer
- LessonAttention by HandUses the board Attention and the Transformer
- LessonTokens & EmbeddingsUses the board Tokens and Embeddings
- LessonLooking Inside the ModelUses the board Tokens and Embeddings
- LessonThe MLP Language ModelUses the board Tokens and Embeddings
- LessonConcurrency & the HardwareUses the board Concurrency and the Hardware
- LessonWaiting without spinningUses the board Concurrency and the Hardware
Further reading
Read the original for the full argument and its evidence.