Ship it: the alarm

The tower that watches every ship in the harbour without boarding one. This lesson builds that tower over the dock app, sets the one alarm worth waking for, breaks the app on purpose, and writes up what happened.

Before you start. Ship it: the sync put the dock app on a laptop cluster, run by Argo CD from a git repository, and ended on a gap: a version that starts and says “ok” to its health check, but fails some customers, rolls out to every copy. The Watchtower (Observability) gave you the error budget, the small share of requests a service may fail in a month before it breaks its promise. The Page at 3 a.m. (Incident Response) gave you the postmortem, the calm write-up after an incident that asks what failed, never who.

The idea in 30 seconds

Prometheus reads each copy’s counters every 15 seconds and stores them, so it can say what share of requests failed over any recent window.
An SLO is a written target for what customers get, such as 99.9% of requests succeeding over 30 days; the 0.1% left over is the error budget.
The burn rate is how many times faster than allowed the budget is being spent; at 50, a month’s budget lasts 14.4 hours.
A good paging rule checks the burn rate over a long window, to be sure, and a short window, so the alarm stops soon after the fix.
A health check can say “ok” while customers fail, so only measuring their answers catches a quiet failure.
A blameless postmortem records what happened, what went well, what let it through, and what changes, without naming anyone as the cause.

Read the story · Skip to the numbers