The GPU (thousands of cores, and the memory that feeds them)

The Inference Engineering board follows one request through a serving system. This lesson is the chip at the bottom of it.

Before you start. A matrix multiply is many dot products at once, multiply the pairs and add, and one neural-network layer is one, from Vectors, Dot Products and Matrix Multiply. A model reads your prompt in one burst, prefill, then writes the reply one token at a time, decode, and each weight costs a multiply and an add per token, two operations counted as FLOPs, from Inference Engineering (serving an LLM is a systems problem). An A100 is NVIDIA’s data-centre GPU, the AI track’s reference card. A batch is the group of conversations served on one decode step, each carrying its KV cache, the model’s saved working for every earlier word so it never recomputes it, and on an A100 a step costs at least 7 milliseconds (The Two Clocks (TTFT, TPOT, and why goodput is the number) measures it later). A processor keeps a little fast memory close and a lot of slow memory far away, and a far read costs many close ones, from The Memory Hierarchy (cache, locality, the memory wall) in How Computers Work.

The board leaves a puzzle on that same chip: one half of a request is limited by arithmetic, the other by fetching numbers from memory. This lesson is why.