Training language models to follow instructions with human feedback
Fine-tune GPT-3 on written answers, then on which of several answers people preferred, and a 1.3 billion parameter model is preferred to the 175 billion parameter one it came from.
Read the original ↗A small model that was told what the job is beat a model a hundred times its size that was not. That is the headline, and it is true. The more useful paper is underneath it: forty contractors, a scoring network that is right about seven times in ten, a leash, and a candid section on whose preferences all of it encodes.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardFine-tuning and Alignmentthe three stages drawn in order, with the reward model, PPO and the KL leash that are this paper's step two and step three
- BoardFrom Transformer to LLMthe base model that continues a question instead of answering it, which is the gap this paper was written to close
- BoardHow Do You Know It's Goodhuman preference as a judge that measures feel rather than truth, which is what this paper's headline number is
- LabThe LoRA rankthe thin correction that later made fine-tuning fit on one card; this paper moved every weight of GPT-3 instead
- LabThe temperature dialthe drum the answers are drawn from; the paper samples its reinforcement learning rollouts at temperature 1
- VideoRL fine-tuning: reward models, RLHF and DPOpreference pairs, a reward model trained on score differences, PPO and the KL leash, which is this paper's recipe drawn step by step
- VideoWhat fine-tuning actually is: prompting, retrieval, LoRA and the order to try them infine-tuning placed after prompting and retrieval in the order to try them, which is where this paper's step one sits
- LessonFine-tuning & AlignmentUses the board Fine-tuning and Alignment
- LessonA LoRA RunUses the board Fine-tuning and Alignment
- LessonFrom Transformer to LLMUses the board From Transformer to LLM
- LessonThe Bigram ModelUses the board From Transformer to LLM
- LessonThe GPT BlockUses the board From Transformer to LLM
- LessonSampling and What Scale BuysUses the board From Transformer to LLM
- LessonEvaluating Models (how do you *know* it's good?)Uses the board How Do You Know It's Good
- LessonPrecision and Recall (where to set the spam filter's bar)Uses the board How Do You Know It's Good
Further reading
Read the original for the full argument and its evidence.
- The paper (arXiv) ↗Twenty pages of body and a long appendix. Section 5.2, on who the model is aligned to, is the part worth reading twice; appendix C.4 has the reinforcement learning settings.
- Scaling Laws for Reward Model Overoptimization (Gao et al., 2022) ↗The measurement of what the KL leash is protecting against: the true score rising and then falling as you optimise harder against a proxy.
- Direct Preference Optimization (Rafailov et al., 2023) ↗The same objective with the reward model and the sampling loop removed, trained as a classification loss on the preference pairs.
- Towards Understanding Sycophancy in Language Models (Sharma et al., 2023) ↗Evidence that human preference data itself rewards telling people what they already believe, which is the hedging this paper noticed, generalised.