AI
emergent ability
capability appearing abruptly at scale
Source board
This definition comes from the board below. Open it to see the term in context.
From Transformer to LLMAIMore terms from this board
- alignmentteaching assistant behavior on top of the base model
- autoregressivegenerate one token at a time, feeding output back
- base modelraw pretrained predictor (not an assistant)
- Chinchillabalance params & data
- decoder-onlya stack of causal transformer blocks (GPT-style)
- distributionprobabilities over the vocab
- in-context learninglearning from prompt examples, no weight change
- logitsraw pre-softmax scores over the vocab
- next-token predictionthe objective (predict the following token)
- parameters / data / computethe three scale levers
- pretrainingthe big self-supervised training run
- scaling lawsloss falls predictably with params/data/compute
- self-supervisedlabels come free from the text itself