Videos / AI

Episode 21 of 22 · AI, end to end

Mixture of experts, reasoning, and test-time compute

Chapters

1. The straight line0:00
2. What a power law promises0:16
3. Too big for its own data0:35
4. The bill, not the physics0:55
5. The text runs out1:12
6. Two levers, not one lane1:30
7. Every specialist, every patient1:48
8. Where the parameters sit2:08
9. Many blocks, one router2:25
10. Top k, and the weights2:41
11. Total versus active2:59
12. A different path every layer3:16
13. Every book stays on the shelf3:33
14. It buys compute, not footprint3:51
15. The router collapses4:07
16. Capacity, and the dropped tokens4:27
17. Routing is a network hop4:44
18. The ledger, both sides5:01
19. The back of the exam paper5:18
20. Thinking is tokens, tokens are compute5:34
21. Answer it more than once5:51
22. A verifier picks the winner6:10
23. Searching, not just sampling6:26
24. Easy now, hard after a think6:45
25. Marked by machine7:03
26. The format, then the truth7:20
27. A reward that is an opinion7:39
28. A reward you cannot argue with7:58
29. The loop, and the caveat8:16
30. Where the checker runs out8:35
31. Saved at a lower quality8:55
32. Fewer bits per weight9:11
33. A student, not a summary9:30
34. Draft, then verify9:49
35. Why the checking is free10:06
36. Three different bills10:24
37. The whole argument10:43

Mixture of experts, reasoning, and test-time compute. Plot a language model's loss against the compute spent training it, on log axes, and you do not get a cloud. You get a straight line. That one finding organised the last few years,

Mark as watchedSaved in this browser only.