The Chubby lock service for loosely-coupled distributed systems
Consensus sold as an operable service rather than a library, because teams discover they need availability after the program is written.
Read the original ↗The paper that made lease-based locking a standard building block, and the reason so much Google infrastructure has a single sentence in it that begins 'we elect a master using Chubby'. Its best material is the section admitting what its users did with it.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardLeader Electionthe talking stick on a lease - acquire, renew, expire - which is Chubby's session with the names changed
- BoardIdempotency and Fencingthe rising token a resource checks, published here in 2006 as a sequencer, caveat included
- BoardWho Holds the Penthe five-replica Paxos cell underneath the lock, which the paper treats as the boring part
- BoardConsistency and Consensuscoordination services as the chapter names them: outsource the hard agreement, keep the elections
- BoardStorage & Compute InfraGFS and Bigtable sit on this shelf, and neither elects a master without it
- LabWho holds the penthe split brain a lease cannot prevent and a sequencer can only make refusable
- LabCharged twicewhat you do when the resource cannot check a token, which the paper concedes is most of them
- VideoConsensus: who holds the penwhy one writer at a time costs a whole replicated cell
- VideoIdempotency keys: how a retry stops being a second chargethe client-supplied key that protects a resource with no version parameter
- LessonScheduler 04: Leader ElectionUses the board Leader Election
- LessonRaft 04: A Key-Value Store on TopUses the board Leader Election
- LessonScheduler 06: Idempotency and FencingUses the board Idempotency and Fencing
- LessonRaft 01: Leader ElectionUses the board Who Holds the Pen
- LessonRaft Capstone: EverythingUses the board Who Holds the Pen
- LessonWho holds the pen (consensus & Raft)Uses the board Who Holds the Pen
- LessonConsistency and ConsensusUses the board Consistency and Consensus
- LessonBuild Your Own RaftUses the board Consistency and Consensus
- LessonRaft 02: Log ReplicationUses the board Consistency and Consensus
- LessonGoogle Search - the case studyUses the board Storage & Compute Infra
Further reading
Read the original for the full argument and its evidence.
- The paper ↗OSDI 2006. Sections 2.4 and 2.9 are the ones the scheduler lesson leans on - why the lock hands out a sequencer, and why a service beat a library.
- Paxos Made Live - An Engineering Perspective ↗The companion, 2007. What it took to build the consensus layer this paper treats as a solved component, written by the people who found out.
- How to do distributed locking ↗Kleppmann, 2016. The canonical modern statement of the pause-then-wake split brain, and of why a lock without a fencing token protects nothing.