AI
base model
raw pretrained predictor (not an assistant)
Source board
This definition comes from the board below. Open it to see the term in context.
From Transformer to LLMAIMore terms from this board
- alignmentteaching assistant behavior on top of the base model
- autoregressivegenerate one token at a time, feeding output back
- Chinchillabalance params & data
- decoder-onlya stack of causal transformer blocks (GPT-style)
- distributionprobabilities over the vocab
- emergent abilitycapability appearing abruptly at scale
- in-context learninglearning from prompt examples, no weight change
- logitsraw pre-softmax scores over the vocab
- next-token predictionthe objective (predict the following token)
- parameters / data / computethe three scale levers
- pretrainingthe big self-supervised training run
- scaling lawsloss falls predictably with params/data/compute
- self-supervisedlabels come free from the text itself