Training Compute-Optimal Large Language Models
For a fixed training budget, grow the model and the data in equal proportion, and a 70 billion parameter model on 1.4 trillion tokens beats the 280 billion one trained with the same compute.
Read the original ↗Given a fixed amount of compute, how big should the model be, and how much should it read? The field had an answer, and every large model of 2021 was built on it. This paper trained over four hundred models to check, found the answer was off by a wide margin, and proved it with one model a quarter the size of its predecessor that beat it almost everywhere.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardFrom Transformer to LLMscaling laws and the three levers, with Chinchilla's balance of parameters and data named in one line
- BoardTraining at Scalesixteen bytes per parameter and the ways a run is split across GPUs, the machinery a parameter-first race was spending on
- BoardMany Experts One Routerparameters that do not all run for every token, which splits the one number this paper optimises into two
- LabThe batch size dialevery generated token reads every weight, which is why a model a quarter the size is cheaper to serve: the bill this paper names and leaves out
- LabMany experts, one routertotal parameters against the ones that actually run, the distinction a dense-only scaling law never had to make
- VideoMixture of experts, reasoning, and test-time computethe scaling-law line, the compute-optimal correction of 2022, and the finite supply of text that came next
- LessonFrom Transformer to LLMUses the board From Transformer to LLM
- LessonThe Bigram ModelUses the board From Transformer to LLM
- LessonThe GPT BlockUses the board From Transformer to LLM
- LessonSampling and What Scale BuysUses the board From Transformer to LLM
- LessonFour Ways to Split a ModelUses the board Training at Scale
- LessonAI/ML Ops - the deep-dive setUses the board Training at Scale
- LessonMixture of Experts (a big brain with a cheap forward pass)Uses the board Many Experts One Router
Further reading
Read the original for the full argument and its evidence.
- The paper (arXiv) ↗Section 3 and Table 3 are the argument; appendix D.2 has the fitted loss formula and appendix A shows how many times Chinchilla repeated each part of its data.
- Chinchilla Scaling: A replication attempt (Besiroglu et al., 2024) ↗A refit of the paper's third approach from its own plots, finding the published constants inconsistent with the first two approaches.
- Beyond Chinchilla-Optimal (Sardana et al., 2023) ↗The same scaling law with the cost of inference added, which moves the answer towards smaller models trained for longer.
- Scaling Data-Constrained Language Models (Muennighoff et al., 2023) ↗What happens once the fresh text runs out: about four passes are nearly free, and past that more compute buys less and less.