AI
KV cache
stored Keys/Values of past tokens (speeds generation)
Source board
This definition comes from the board below. Open it to see the term in context.
InferenceAIMore terms from this board
- context windowmax tokens the model sees at once
- greedyalways take the top token
- groundingRAG/tools to tie answers to real sources
- hallucinationconfident, fluent, but false output
- inferencegenerating text from the trained model
- prefill vs decodeprocess the prompt vs generate token-by-token
- samplingpicking the next token from the distribution
- statelessno memory between calls (history is re-sent)
- temperaturerandomness dial (0 = deterministic)
- the KV cache grows with contextthe memory cost of long context
- throughputtokens/sec
- top-k / top-p (nucleus)sample only from the top tokens / the top cumulative mass
- TTFTtime to first token