One request, forty services (distributed tracing)

The Contract at the Counter · Open board

Two services meeting at a counter across a written list of what you may ask. This lesson goes one step past it: what happens when a single request crosses forty such counters, and how to follow it from the front door to the last one.

Before you start. You need Thirty stalls on one order (availability, the nines, and the tail at scale), where one request fanned out to many parts, the slowest part set the pace, and a 1% tail became the usual case. You also need The Contract at the Counter (API design), where services meet only across a written list of what you may send, and adding a new line to that list breaks nobody who ignores it.

In this lesson

June 2012: Twitter publishes a tool for finding where a request went

One request, forty services: nobody can see the whole trip

The trace: one label stuck on at the front door

The span: one service’s slice, with a parent

Trace id propagation: the label must be handed on

The critical path: the only chain whose speed matters

Sampling: you cannot keep every trace

Terms it explains: trace, trace id, span, trace id propagation, critical path, sampling, head sampling, Tail sampling, Retries and hedges, Run the code above.

Index

Open everything on chalk.

One request, forty services (distributed tracing)

In this lesson