ZooKeeper: Wait-free coordination for Internet-scale systems
Do not implement the lock. Ship a wait-free store with ordering and a notification, and let clients assemble the primitives.
Read the original ↗Chubby put locks in the server. ZooKeeper's argument is that you should not: give clients a wait-free store with ordering guarantees and a notification, and let them build locks, elections, membership and barriers on top. It is the design that ran a decade of open-source infrastructure, and the one Kafka spent five years leaving.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardConsumer Groupsthe group coordinator and committed offsets, both of which lived here in 2011 and both of which moved out
- BoardReplication and Durabilitythe controller that elects a partition leader from the ISR - the job this held until KRaft took it
- BoardEverythingthe panel that names KRaft as the replacement, in the list of what real Kafka adds
- BoardLeader Electionthe ephemeral node under the lease, and the TTL trade the session model makes for you
- BoardConsistency and ConsensusZab named beside Paxos and Raft, and coordination-as-a-service as a category
- BoardAirbnb, Monorail to SOASmartStack registering each host as an ephemeral node and watching the tree - service discovery, the workload that actually dominated
- LabThe consumer leasea lease over partitions with a durable bookmark, which is the recipe this paper's primitives were assembled into
- LabWho holds the penthe election the ephemeral sequential node implements, and the split brain a session timeout produces
- VideoThe thundering herd: when everyone reconnects at onceone event waking every waiter - the herd the watch-your-predecessor recipe exists to avoid
- VideoConsensus: who holds the penthe quorum under the ensemble, and why writes do not get faster when you add servers
- VideoEvent-driven architecture: the ledger everyone readstopic, offset and replay - the Kafka model whose coordination this used to hold
- LessonKafka 05: Consumer GroupsUses the board Consumer Groups
- LessonKafka: Replication and DurabilityUses the board Replication and Durability
- LessonKafka 08: EverythingUses the board Everything
- LessonScheduler 04: Leader ElectionUses the board Leader Election
- LessonRaft 04: A Key-Value Store on TopUses the board Leader Election
- LessonConsistency and ConsensusUses the board Consistency and Consensus
- LessonBuild Your Own RaftUses the board Consistency and Consensus
- LessonRaft 02: Log ReplicationUses the board Consistency and Consensus
- LessonCompany Platforms - Netflix / Spotify / AirbnbUses the board Airbnb, Monorail to SOA
Further reading
Read the original for the full argument and its evidence.
- The paper (PDF) ↗USENIX ATC 2010. Section 2 is the API and the two guarantees; section 2.4 is the recipes, and the election recipe is the one to read twice.
- Zab: High-performance broadcast for primary-backup systems ↗The replication protocol underneath, published separately - read it if you want to know why it is not Paxos and not Raft.
- KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum ↗The largest user's exit, argued in public. The operational case against a second consensus system in the stack, written by people who ran one for a decade.
- ZooKeeper Recipes ↗The recipes as maintained documentation, including the explicit herd-effect warning that motivates watching your predecessor rather than the lock.