Scheduler 07: The Hard Part - Time

From Scratch · Distributed Scheduler.

Before you start. This lesson is part of a build that makes a small distributed job scheduler - a program that fires jobs at the right moment, on the right machine, even when a machine crashes mid-run. Six earlier stages already gave it a lot. It can tell which job is due next without scanning the whole list. It survives a crash, because every job lives in a database instead of only in memory. Several machines run the scheduler, but only one of them, the leader, decides which jobs fire. It stays leader by holding a lease, a permission that runs out unless it keeps renewing it. The leader does not run the jobs itself. It drops each one into a queue, a waiting line that other machines take work from. And every new lease comes with a bigger number, so the database can refuse a write from an old leader that has not noticed it was replaced. All five of those were built on one thing nobody checked: that the clock telling them the time is telling the truth.

The idea in 30 seconds

A wall clock tells you the date; a stopwatch - a monotonic clock - tells you how long. Answer a “how long” question with the wall clock, and a routine correction can wreck the answer.
A lease is a “how long” question - has my time run out - so it has to be checked against the stopwatch, not the wall clock.
Daylight saving deletes an hour or repeats one, twice a year; a schedule stored as a rule survives that, a schedule stored as one fixed instant does not.
Time zones and daylight-saving dates are set by governments, and governments change them, sometimes on a few weeks’ notice.
A leap second is the same problem at world scale: an instant that shouldn’t exist, which a program that never expected it cannot represent.
What happens to a missed job, and what happens to a job that overlaps itself, are decisions this build makes on purpose, not facts the clock can hand you.

Read the story · Skip to the numbers