The Bitter Lesson
Seventy years of AI research say the same thing every time: a general method that rides falling compute cost beats a method built from what humans know, and the people who built in the knowledge never see it coming.
Read the original ↗About eleven hundred words, posted on a personal website in 2019, with no citations and no figures, and it is now the most quoted argument in the field. It is quoted, almost always, as saying that scale is all you need. It does not say that. It names two specific methods and one specific economic premise, and the premise is the part that has aged.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardHow It Learnslearning, the second of the two methods the essay says scale without limit
- BoardAttention and the Transformerthe architecture the lesson is credited with, and the human choice inside it the essay does not admit
- BoardReasoning and Test-Time Computesearch coming back at inference time, which is the essay's chess argument one level up
- BoardMany Experts One Routerspending the compute budget wider instead of deeper, which is still the essay's bet
- BoardConcurrency and the Hardwarethe falling cost per unit of compute the whole argument rests on, and where it stopped falling
- LabThe parallel ceilingthe part of any method that does not scale with more machines, which is the essay's silent assumption
- LabHow cost growscompute moves the constant and never the exponent - the limit the lesson does not mention
- LabMany experts, one routerone way the field kept the compute growing after the chips stopped
- VideoHow we got here: from the perceptron to the transformerthe same seventy years, told forward - the essay is that history read backward
- VideoMixture of experts, reasoning, and test-time computemixture of experts and test-time compute, the two moves the field made once training compute alone ran short
- VideoAttention in production: KV cache, MQA, GQA and MLAwhat happened when one general method was given every GPU on earth
- LessonHow It Learns (loss, gradient descent, backprop)Uses the board How It Learns
- LessonOne Knob, One SlopeUses the board How It Learns
- LessonWhy the Loss Won't Go DownUses the board How It Learns
- LessonAutograd by HandUses the board How It Learns
- LessonAttention & the TransformerUses the board Attention and the Transformer
- LessonRNNs and LSTMsUses the board Attention and the Transformer
- LessonAttention by HandUses the board Attention and the Transformer
- LessonReasoning & Test-Time Compute (why "thinking longer" works)Uses the board Reasoning and Test-Time Compute
- LessonMixture of Experts (a big brain with a cheap forward pass)Uses the board Many Experts One Router
- LessonConcurrency & the HardwareUses the board Concurrency and the Hardware
- LessonWaiting without spinningUses the board Concurrency and the Hardware
Further reading
Read the original for the full argument and its evidence.
- The essay ↗Sutton's own site, plain http and eight paragraphs. Eleven hundred words and worth reading in full before arguing with anyone about it, because the two sentences that matter most are the shortest ones.
- A Better Lesson, Rodney Brooks ↗Published days after the essay and still the strongest objection to it: that convolution and every successful architecture since are themselves human knowledge, so none of the victories were ever as pure as described.
- Scaling Laws for Neural Language Models ↗Kaplan and colleagues, January 2020. The empirical vindication that arrived ten months later, and the source of the smooth power law everyone now cites instead of the essay.
- Will we run out of data? ↗Epoch AI on the stock of human-generated public text and when frontier training exhausts it. The constraint the essay has no mechanism for, because falling compute cost does not produce more text.
- DeepSeek-R1 ↗January 2025, and the clearest public account of moving the spending to inference time. Read it as the essay's chess argument recurring: search returning as the way to convert a budget into performance.