Scheduler 02: Durable Job Store

The board that moves the schedule off one machine and onto disk. Follows Scheduler 01: The Tick Loop, and

Before you start. This series builds a small distributed job scheduler, a program whose only task is to fire off other jobs - a nightly backup, an hourly report - on a repeating schedule, without missing one and without running it twice. So far, our small scheduler can check a list of jobs against the clock and run whichever ones are due. It computes each job’s next run as a fixed point in time rather than “five minutes from now,” which is what stops the schedule from creeping late. And when a job runs longer than the gap between its own runs, it already has three answers to choose from: skip that run, hold it until the current one finishes, or start it anyway alongside the one still going. All three live in one list, inside the memory of one running program, and that is exactly what breaks next.

The idea in 30 seconds

A scheduler that keeps its jobs only in memory loses the whole schedule, and the whole history, the moment it restarts.
Move the jobs into a store, a table on disk, and a restart just reopens the same rows.
What was missed while you were down is a policy you choose: run each missed one once, backfill every slot, or skip ahead to the next one.
Taking a job has to be a claim: one write that both picks it and marks it taken, so two workers can’t grab the same one.
A claim isn’t forever. It’s a lease, and it expires on its own if nobody renews it - the only reason a dead worker’s job ever comes back.
The store can’t always tell a dead worker from a slow one, so it has to choose: hand the job out again and risk running it twice (at-least-once), or risk losing it silently forever. At-least-once is the honest choice.

Read the story · Skip to the numbers