AI
the KV cache grows with context
the memory cost of long context
Source board
This definition comes from the board below. Open it to see the term in context.
InferenceAIMore terms from this board
- context windowmax tokens the model sees at once
- greedyalways take the top token
- groundingRAG/tools to tie answers to real sources
- hallucinationconfident, fluent, but false output
- inferencegenerating text from the trained model
- KV cachestored Keys/Values of past tokens (speeds generation)
- prefill vs decodeprocess the prompt vs generate token-by-token
- samplingpicking the next token from the distribution
- statelessno memory between calls (history is re-sent)
- temperaturerandomness dial (0 = deterministic)
- throughputtokens/sec
- top-k / top-p (nucleus)sample only from the top tokens / the top cumulative mass
- TTFTtime to first token