Videos / AI

Episode 4 of 22 · AI, end to end

RNNs and LSTMs: the memory that almost worked

Chapters

1. Order is information0:00
2. A box of fixed width0:16
3. One cell, one state0:30
4. Unrolled, but shared0:47
5. What is in the state1:03
6. Four useful shapes1:19
7. A hundred layers, by accident1:36
8. Blame, travelling back1:51
9. The same factor, a hundred times2:07
10. Vanishing, quietly2:26
11. Exploding, loudly2:44
12. One is loud, one is silent3:00
13. The belt3:17
14. Add, do not transform3:32
15. Gate one: forget3:49
16. Gate two: input4:05
17. Gate three: output4:21
18. The cheaper bet4:37
19. Lengths that disagree4:54
20. Encoder, decoder, one vector5:10
21. Feeding itself back5:29
22. The bottleneck5:46
23. Patches that missed6:02
24. Nothing to look at6:17
25. Keep the page open6:33
26. A score per source word6:49
27. Softmax, then a mixture7:07
28. Alignment you can draw7:24
29. The patch outgrew it7:44
30. Which half was essential8:00
31. Step nine, then ten8:16
32. The idle chip8:34
33. The whole sequence at once8:52
34. The bill for it9:12
35. Where recurrence wins9:29
36. What actually survived9:48
37. The whole thing10:04

RNNs and LSTMs: the memory that almost worked. Three words. Dog bites man. Now the same three words in the other order. Man bites dog. Nothing in the vocabulary changed. Everything in the meaning did. Order is information, and

Mark as watchedSaved in this browser only.