Chunked Prefill (one long prompt must not freeze the room)

Serving LLMs, the third board. It draws the batch’s schedule twice: once when a long paste runs whole and freezes the room, once sliced into pieces that ride along with everyone else’s replies.

Before you start. The Two Clocks (TTFT, TPOT, and why goodput is the number) gave you the room: 32 people sharing one GPU, each getting a new token every 39 ms, a promise called TPOT that a busy GPU can still break. A token is a word or a piece of one, and producing one means pushing it through the model’s weights, the 7 billion numbers the model learned while it was trained. Prefix Caching (the same start, computed once) then showed the machine skip re-reading a system prompt it has already read once; this lesson is about the prompt it has never read before.

The idea in 30 seconds

Reading a prompt before answering is prefill; writing the answer out afterward, one word at a time, is decode.
A long prompt’s prefill, run whole, is one uncuttable block of pure arithmetic that can freeze every other reply for as long as it takes to finish.
A busy GPU is not the same thing as a kept promise: the freeze looks like 100% utilisation on the one dashboard anyone was watching.
Chunked prefill cuts the prompt into fixed-size slices and runs one per decode step; piggybacking is why that ride is nearly free, not merely fair.
Stall-free scheduling, from the 2023-2024 work at Microsoft Research and Georgia Tech, always fills decode tokens first and lets a prompt’s slices take whatever room is left.
The slice size is a dial with no free setting: smaller protects everyone else’s replies, larger gets the pasted document its own first word sooner.

Read the story · Skip to the numbers