Thirty stalls on one order (availability, the nines, and the tail at scale)

One search fanned out to many machines at once, and panel B, “the slowest leaf wins”. This lesson walks the same idea through the chai stall, one wedding order split across thirty stalls. The lab at the end of the story lets you move the number of stalls yourself.

Before you start. You need the stall from The chai stall that got famous (load balancing): one famous stall that became a chain, with the crowd shared between stalls so no single one drowns. One wala, many kettles (concurrency vs parallelism) helps too: it showed a whole chain of stalls running off one gas cylinder, and that cylinder comes back here.

The idea in 30 seconds

Availability is the share of time something answers; each extra nine divides the down time by ten, and 99.9% is 43 minutes a month.
When a request needs several parts all working, their availabilities multiply, so thirty parts at 99.9% make a whole at 97%.
Redundancy, a spare copy of each part, turns “down if one fails” into “down if both fail”, but only when the copies do not share what fails.
Tail latency is how slow the rare slow answers are, and fan-out to many parts turns a 1% tail into the common case: 26% at thirty parts, 63% at a hundred.
A hedged request sends a second copy only when the first is late, and cuts the slow share sharply for a few percent more calls; asking fewer parts and a timeout are the other two ways out.

Read the story · Skip to the numbers