The stall, built: rush hour (load testing, p99 and the knee in Go)

The rail of slips over the stove, and the chart where the wait leaves the ground as the stove gets busy. This lesson draws that chart again, from a real program, by sending it a crowd and timing every cup.

Before you start. The stall, built: copies and shards (replication and sharding in Go) finished the stall’s back room: a copy of the book that runs a few milliseconds behind, and the book cut into three by a hash. The front of the stall is still the one from The stall, built: one counter in code (load balancing, health checks and caching in Go): three stalls, each cooking one cup at a time, and a gate that sends each customer to the least busy one. Thirty stalls on one order (availability, the nines, and the tail at scale) said the number that matters is how slow the unlucky customers are, not the average one, and that an order split across many stalls meets its unluckiest stall far more often. The stall that turns people away (load shedding and admission control) showed that when orders arrive faster than the stoves cook, the line grows without end. This lesson measures both on the stall you built.

The idea in 30 seconds

A load generator that waits for each answer before sending the next goes quiet whenever the server stalls, so it records almost none of the slow moments; this is coordinated omission.
A fair load generator sends on a fixed schedule and times each request from when it was due, so a stall is charged to every request that should have been sent during it.
p50 is the middle request’s time and p99 is the time only one request in a hundred exceeds; the p99 is what the unlucky users feel.
Response time stays flat as load rises, then bends sharply near full use; here the bend, the knee, came around 80% of the servers’ capacity on paper.
Losing one of three servers at 60% load leaves two at 90%, past the knee: nothing fails after the first moments, and the slow requests get twenty times slower.

Read the story · Skip to the numbers