AI
pretraining
the big self-supervised training run
Source board
This definition comes from the board below. Open it to see the term in context.
From Transformer to LLMAIMore terms from this board
- alignmentteaching assistant behavior on top of the base model
- autoregressivegenerate one token at a time, feeding output back
- base modelraw pretrained predictor (not an assistant)
- Chinchillabalance params & data
- decoder-onlya stack of causal transformer blocks (GPT-style)
- distributionprobabilities over the vocab
- emergent abilitycapability appearing abruptly at scale
- in-context learninglearning from prompt examples, no weight change
- logitsraw pre-softmax scores over the vocab
- next-token predictionthe objective (predict the following token)
- parameters / data / computethe three scale levers
- scaling lawsloss falls predictably with params/data/compute
- self-supervisedlabels come free from the text itself