Videos / AI

Episode 7 of 22 · AI, end to end

Attention in production: KV cache, MQA, GQA and MLA

Chapters

1. Two bills0:00
2. Query, key and value0:16
3. The n by n square0:36
4. One token at a time0:53
5. Arithmetic intensity1:13
6. Busy, then idle1:30
7. Never add it twice1:49
8. Keys and values, not queries2:04
9. Quadratic becomes linear2:22
10. The cache, as a formula2:38
11. Twenty one gigabytes2:54
12. The cache is the batch3:12
13. One shared document3:28
14. Broadcast, not replicate3:46
15. Divide by the head count4:04
16. And it decodes faster4:22
17. What it costs4:39
18. The honest summary4:57
19. One per table5:14
20. Sixty four into eight5:29
21. The dial has a number5:45
22. The arithmetic of eight6:01
23. Why everyone shipped it6:18
24. It fits the hardware too6:37
25. Shorthand you expand6:54
26. Down, then up7:10
27. A different trade7:30
28. Fold the up projection7:51
29. The rotary snag8:07
30. The deal, stated plainly8:25
31. The desk and the warehouse8:43
32. The round trips8:57
33. Tiles, and one word9:13
34. Same answer, fewer trips9:30
35. A window, which is not9:49
36. The map of the toolbox10:09
37. The whole thing10:27

Attention in production: KV cache, MQA, GQA and MLA. Here is the thing that decides every design in this video. Training a language model and serving one are not the same problem. Training is bound by arithmetic. Serving is bound by

Mark as watchedSaved in this browser only.