Inference (sampling, context, KV cache, hallucination)

The Inference board of the How LLMs Work track. Training is over and the weights will never change again. This is what happens in the seconds after you hit enter.

Before you start. You need to know that the model is a huge pile of fixed numbers, its parameters (a 7-billion-parameter model is a pile of 7 billion of them), which scores every word that could come next; From Transformer to LLM is where that pile came from. It deals in tokens, chunks of roughly three quarters of a word, from Tokens & Embeddings. No maths beyond adding and dividing.