Direct Preference Optimization: Your Language Model is Secretly a Reward Model
The best model for the RLHF objective has a closed form, so the reward model and the reinforcement learning loop collapse into one classification loss on the preference pairs.
Read the original ↗A Stanford paper from 2023 that took the most complicated stage of training a chat model, a reward model plus a reinforcement learning loop, and showed it could be replaced by one loss function on the preference pairs. The algebra fits in a paragraph. What it quietly gave up took the field about a year to measure.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardFine-tuning and Alignmentthe objective with its KL leash and beta, which is the equation DPO solves in closed form
- VideoRL fine-tuning: reward models, RLHF and DPOpreference pairs, the reward model and the KL leash, then the interpreter chapter where DPO drops the reward model and gives up exploration
- LessonFine-tuning & AlignmentUses the board Fine-tuning and Alignment
- LessonA LoRA RunUses the board Fine-tuning and Alignment
Further reading
Read the original for the full argument and its evidence.
- The paper (arXiv) ↗NeurIPS 2023. Section 4 is the whole derivation in a page, and the loss code is in Appendix B.
- A General Theoretical Paradigm to Understand Learning from Human Preferences ↗Azar and others, DeepMind, 2023. The IPO paper, and the clearest account of why the KL term goes slack when preferences look certain.
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms ↗Rafailov and others, 2024. DPO's first author measuring the question the DPO paper left open.