The batch size dial
One GPU is serving one chat, and its cores are 99% idle. How many more chats can you add before anything slows down, and what stops you?
Generating one token drags every weight off memory while the cores mostly wait. The panels show how much arithmetic each byte buys, which of the two units is actually busy, and how many chats fit before the cache holding each chat's context fills the card. Add chats and the throughput climbs for almost nothing, until the wall arrives.
Board: Inference →Three panels stack on a narrow screen. The two dials and the lever tabs stay above them, and the swept table scrolls sideways.
try this Drag Chats sharing one read of the weights to the right and watch operations per byte rise until The wall stops it.
Included with a plan
This lab is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
see also Attention Is All You Need, Training Compute-Optimal Large Language Models and Efficient Memory Management for Large Language Model Serving with PagedAttention in Papers.