Videos / AI

Episode 16 of 22 · AI, end to end

Computer use and multimodal agents

Chapters

1. One task0:00
2. Picture in, click out0:16
3. The interface everything has0:32
4. Eight verbs0:49
5. A pair of numbers1:06
6. Four questions1:24
7. Cut into squares1:38
8. A patch becomes a token1:57
9. One sequence, mixed2:14
10. What a screenshot costs2:31
11. Grounding2:49
12. Why grounding misses3:05
13. The glance back3:24
14. Observe means now3:41
15. Plan one step3:57
16. Act has no dry run4:15
17. Verify is not optional4:31
18. The receipt, traced4:48
19. A photograph of a room5:07
20. A step costs a call5:24
21. Seconds, not milliseconds5:41
22. Keep words, drop pictures5:58
23. No undo6:15
24. The screen moved under it6:32
25. A reader that sees no pixels6:50
26. The accessibility tree7:07
27. And in a browser7:25
28. Structured beats pixels7:43
29. Where the tree runs out8:01
30. Take both8:19
31. The mouse is real8:35
32. Sandbox8:53
33. Allowlist9:10
34. The last click is yours9:28
35. The screen is untrusted9:48
36. Keep the receipts10:06
37. The whole thing10:25

Computer use and multimodal agents. Here is the task, and it does not change for the next ten minutes. There is a receipt sitting in the downloads folder. Open it, type its total into the expenses app, attach the fil

Mark as watchedSaved in this browser only.