AI
next-token prediction
the objective (predict the following token)
Source board
This definition comes from the board below. Open it to see the term in context.
From Transformer to LLMAIMore terms from this board
- alignmentteaching assistant behavior on top of the base model
- autoregressivegenerate one token at a time, feeding output back
- base modelraw pretrained predictor (not an assistant)
- Chinchillabalance params & data
- decoder-onlya stack of causal transformer blocks (GPT-style)
- distributionprobabilities over the vocab
- emergent abilitycapability appearing abruptly at scale
- in-context learninglearning from prompt examples, no weight change
- logitsraw pre-softmax scores over the vocab
- parameters / data / computethe three scale levers
- pretrainingthe big self-supervised training run
- scaling lawsloss falls predictably with params/data/compute
- self-supervisedlabels come free from the text itself