Videos / AI

Episode 9 of 22 · AI, end to end

Instrumenting LLM calls: traces, spans and what to log

Chapters

1. All green, still wrong0:00
2. No stack trace for wrong0:20
3. You cannot rerun it0:37
4. Every step returned fine0:55
5. What the three pillars miss1:14
6. Instrument the content1:31
7. One printed itinerary1:47
8. Three fields, one tree2:05
9. Six spans, one answer2:21
10. One span per model call2:38
11. The context that travels2:54
12. The unit of debugging3:13
13. The flight recorder3:29
14. The rendered prompt, not the template3:46
15. The completion, and why it stopped4:04
16. The identity of the call4:21
17. Tokens, latency, cost4:39
18. Every tool call, both sides4:59
19. Corners everyone agreed on5:16
20. One field, three spellings5:34
21. Three separable parts5:53
22. The generative AI conventions6:11
23. Be honest: still moving6:31
24. One collector, in the middle6:49
25. The itemised bill7:07
26. Cost, at the tail7:25
27. First token is not total7:43
28. What breaks the cache8:02
29. Errors, broken down8:21
30. Turns per resolved task8:39
31. The copier in the corner8:57
32. Read the prompt you just logged9:14
33. Redact before it leaves9:33
34. Three tiers of content9:53
35. Keep the failures, not a slice10:12
36. Retention, and the clock10:31
37. The whole thing10:51

Instrumenting LLM calls: traces, spans and what to log. Your dashboard is entirely green. Two hundred OK. Ninety milliseconds. Zero errors. And the answer the model just gave a customer was confidently, fluently wrong. Every signal you

Mark as watchedSaved in this browser only.