Build Your Own Distributed Scheduler - From Scratch

The whole build, laid out in the order its own walls forced it.

Boards

Code: teaching/code/from-scratch/scheduler/ - one Go module, go run ./NN-stage. Go is the programming language every stage is written in. The demos run on a pretend clock, so every run prints the same result; the later stages show a stand-in taking over when the one in charge dies, crashed work being handed out again, and no job running twice.

Before you start. This series builds a small copy of a job scheduler: a program that runs other programs at the right time - a backup every night, a report every Monday - and keeps doing that correctly across several machines, any of which can die mid-job. It borrows two ideas from earlier builds in this track, and you need neither to follow along. From the small database: a file on disk that can find any one saved record quickly, used here to remember every job. From the small message log: a waiting line of work that helpers take from, where a task a helper dropped goes back in line, used here to get the jobs done.

The idea in 30 seconds

A scheduler is one loop repeated forever: what’s due right now, do it, work out when it’s due next.
Going distributed means several copies of that loop share one job list, so the list itself has to survive a crash - a durable store, not one clerk’s memory.
Checking every job every minute doesn’t scale; keeping the soonest one on top, a heap, means you only ever look at one.
Letting every copy act at once means every job runs once per copy, so only one copy - the leader, chosen by a lease - is allowed to fire jobs at all.
Doing the work and deciding what’s due are different jobs; the leader hands work to a pool of workers through a queue, and a dead worker’s job goes to someone else.
A job handed out twice can run twice, so an idempotency key makes a repeat a no-op, and a fencing token stops an already-replaced clerk from acting anyway.
The clock itself has policies to pick, not just a bug to fix: what happens after downtime, what happens when a run overlaps the next, and everyone agreeing on one shared time.

Read the story · Skip to the numbers