The Tail at Scale
At a thousand-way fan-out your typical request is your server's one-in-a-thousand bad day, so stop making components fast and start making slow ones not matter.
Read the original ↗Six pages in CACM that renamed a problem, and you cannot unsee it afterwards. Fan a request out to a thousand machines and your typical response time stops being your server's typical response time - it becomes your server's one-in-a-thousand bad day. Nothing you do to the average touches it.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardQuery Servingpanel B is this paper, cited on the board by name: the slowest leaf sets the latency of the whole query
- BoardEverythingthe tail sitting in the x-ray over query serving, next to the ~200 ms budget it threatens
- BoardThe Order Slip Railqueueing at high utilisation - the variability source the paper cannot design away, because it is the utilisation you are paying for
- BoardMicroservices & Resiliencethe same fan-out arithmetic run on availability instead of latency, reaching the same conclusion: stop believing any one dependency
- BoardThe Thundering Herda correlated tail - everybody slow at the same instant, which is the case a second replica does not rescue
- BoardLoad Balancing & DNSchoosing where to send a request, which is the cheaper half of the paper's answer
- LabThe load balancertwo requests in a hundred cost sixty times the rest - p99 from 7.4s to 2.9s by asking two backends instead of none
- LabThe order slip railthe wait at 95% busy, which is where most of the paper's variability list actually comes from
- VideoQueues: the order slip rail and the cliffthe cliff a queue sits on as it nears capacity
- VideoMicroservices and resilience: containing failure at scalefan-out maths that says the dependencies cannot all be working, and what to do about it
- VideoThe thundering herd: when everyone reconnects at onceone real outage where every client was slow at the same moment
- VideoMonitoring and observability: knowing what is happening, and whypercentiles, and watching an average lie - the measurement this whole argument depends on
- LessonGoogle Search - the case studyUses the board Query Serving
- LessonThirty stalls on one order (availability, the nines, and the tail at scale)Uses the board Query Serving
- LessonThe order slip rail (queues & the utilization cliff)Uses the board The Order Slip Rail
- LessonThe stall that turns people away (load shedding and admission control)Uses the board The Order Slip Rail
- LessonThe stall, built: rush hour (load testing, p99 and the knee in Go)Uses the board The Order Slip Rail
- LessonNetflix - the case studyUses the board Microservices & Resilience
- LessonSlack - the case studyUses the board The Thundering Herd
- LessonLoad Balancing & DNS (the operator's half)Uses the board Load Balancing & DNS
Further reading
Read the original for the full argument and its evidence.
- The paper ↗Communications of the ACM 56(2), February 2013. Six pages, no equations to speak of. The table of fan-out against slow-request probability on the first page is the whole argument; the rest is the catalogue.
- The paper (PDF) ↗The copy most citations point at, if the ACM link asks you for a login.
- Addressing cascading failures ↗The Google SRE book chapter that supplies what the paper is missing: retry budgets, deadline propagation, load shedding, and the reason a retrying client is a load generator.
- Timeouts, retries and backoff with jitter ↗The operational companion piece from the other side of the industry. Read it directly after the hedging section, because it is about what the duplicate request does when the service was slow because it was overloaded.