Videos / AI

Episode 20 of 22 · AI, end to end

Evals: how you know an agent is good

Chapters

1. The case on your screen0:00
2. The four you were not watching0:16
3. Code has a diff0:33
4. Four words first0:49
5. Three parts, two shipped1:05
6. The number is a comparison1:21
7. Open the logs1:38
8. Hold it out1:55
9. Versioned beside the code2:11
10. Keep the hard ones2:27
11. Somebody has to label2:44
12. Slice the set3:02
13. Four kinds of ruler3:19
14. Exact match3:37
15. Assert what you can3:55
16. A rubric and a person4:12
17. A model as judge4:29
18. The judge's biases4:44
19. A flight recording5:02
20. Did the world move5:19
21. The path it actually took5:37
22. Right tool, right arguments5:56
23. Steps compound6:13
24. Cost is a score too6:31
25. Five runs, five scores6:49
26. Where the spread comes from7:07
27. Report an interval7:25
28. Fifty examples is noise7:40
29. Three ways to get it back8:00
30. Wins, losses, and luck8:17
31. The blocked merge8:35
32. Two tiers8:53
33. Budget the bill9:10
34. Thresholds that hold9:28
35. Print what flipped9:45
36. Close the loop10:00
37. The whole thing10:18

Evals: how you know an agent is good. One case is failing on your screen. You reword the prompt, run it again, and it passes. That feels like progress. It is one data point, chosen by you, graded by you, on the example

Mark as watchedSaved in this browser only.