AI
O(n²)
attention's cost in sequence length (the context bottleneck)
Source board
This definition comes from the board below. Open it to see the term in context.
Attention and the TransformerAIMore terms from this board
- attention matrixn×n who-attends-to-whom
- causal maskcan only attend to earlier tokens (for generation)
- cross-attentionattend to another sequence
- feed-forward (FFN/MLP)the per-token 'thinking' layer
- layer normstabilizes activations
- multi-headparallel attentions capturing different relations
- Query/Key/Value'what I seek / what I offer / what I give'
- residual connectionadd the input back (deep-net enabler)
- scaled dot-product attentionsoftmax(QKᵀ/√d)V
- self-attentioneach token attends to the others in its own sequence
- stack of N blocksthe depth of the model
- transformer blockattention + FFN + residual + LayerNorm