AI
causal mask
can only attend to earlier tokens (for generation)
Source board
This definition comes from the board below. Open it to see the term in context.
Attention and the TransformerAIMore terms from this board
- attention matrixn×n who-attends-to-whom
- cross-attentionattend to another sequence
- feed-forward (FFN/MLP)the per-token 'thinking' layer
- layer normstabilizes activations
- multi-headparallel attentions capturing different relations
- O(n²)attention's cost in sequence length (the context bottleneck)
- Query/Key/Value'what I seek / what I offer / what I give'
- residual connectionadd the input back (deep-net enabler)
- scaled dot-product attentionsoftmax(QKᵀ/√d)V
- self-attentioneach token attends to the others in its own sequence
- stack of N blocksthe depth of the model
- transformer blockattention + FFN + residual + LayerNorm