Routing and Cascades (not every request needs the big model)

The last board of the Serving LLMs run. Five earlier boards each made one fixed-size model faster: The Two Clocks (TTFT, TPOT, and why goodput is the number) measured its two clocks, Prefix Caching (the same start, computed once) and Chunked Prefill (one long prompt must not freeze the room) fixed the first and second, Prefill / Decode Split (two pools, one KV in flight) split the pool in two, Constrained Decoding (the grammar holds the pen) reached into its sampler. This board steps back to the front door and asks the question none of them asked: does this request even need that model? One queue, three price tags 75 times apart, and two different ways of deciding which one a question gets.

Before you start. A model charges you for every word it reads and every word it writes. A bigger model charges more for each of those words, and answers harder questions better. The five lessons before this one took one model of one fixed size and made its words cheaper. This lesson lets the size change too, by picking which model a question goes to.

The idea in 30 seconds

A fleet of models can be priced tens of times apart, and most requests don’t need the priciest one.
A tier is one size of model, in its own pool of machines, at its own price.
A router decides which tier a request goes to before any model answers; it’s cheap and fast, and fails silently when a hard question looks easy.
A cascade lets the cheapest tier try first and only escalates when it doubts itself, reading a log-prob it already computed as a confidence score.
Confidence and correctness are not the same thing - that gap is calibration, and it lets a cascade wave through a smooth wrong answer or pay twice for a right one it doubted.
The two mechanisms fail on opposite sides of the same moment, which is why using them together closes more gaps than either alone.

Read the story · Skip to the numbers