AI
logits
raw pre-softmax scores over the vocab
Source board
This definition comes from the board below. Open it to see the term in context.
From Transformer to LLMAIAlso defined on Inference.
More terms from this board
- alignmentteaching assistant behavior on top of the base model
- autoregressivegenerate one token at a time, feeding output back
- base modelraw pretrained predictor (not an assistant)
- Chinchillabalance params & data
- decoder-onlya stack of causal transformer blocks (GPT-style)
- distributionprobabilities over the vocab
- emergent abilitycapability appearing abruptly at scale
- in-context learninglearning from prompt examples, no weight change
- next-token predictionthe objective (predict the following token)
- parameters / data / computethe three scale levers
- pretrainingthe big self-supervised training run
- scaling lawsloss falls predictably with params/data/compute
- self-supervisedlabels come free from the text itself