Videos / AI

Episode 19 of 22 · AI, end to end

RL fine-tuning: reward models, RLHF and DPO

Chapters

1. Copying the stroke0:00
2. The same loss as before0:16
3. One target, many0:36
4. The demonstrator's ceiling0:53
5. What one answer cannot say1:08
6. Stop writing, judge1:26
7. Better, or worse1:43
8. Making the pairs2:00
9. A direction, not a score2:16
10. One difference2:33
11. A paid coin flip2:49
12. The label carries the labeller3:05
13. The judge, learned3:22
14. Swap the head3:39
15. Only differences count3:55
16. Taste you can call4:14
17. It learned the artefacts4:30
18. Confident about nowhere4:46
19. Three copies in the gym5:03
20. One step of the loop5:21
21. The clip in PPO5:38
22. The leash, in KL5:56
23. Take the leash off6:13
24. Beta, and the bill6:29
25. The interpreter leaves6:47
26. The reward was in there7:04
27. One loss, one line7:23
28. Count what is gone7:41
29. Offline, by design7:58
30. The simpler line8:16
31. The answer key8:35
32. How reasoning is trained8:52
33. The group is the baseline9:09
34. The goal and the measure9:26
35. What it looks like9:42
36. Unglamorous defences10:02
37. The whole thing10:21

RL fine-tuning: reward models, RLHF and DPO. Picture an apprentice copying a master's brushstrokes, line for line. That is supervised fine tuning. You show the model a prompt and one written answer, and it learns to reproduce

Mark as watchedSaved in this browser only.