Kafka: Replication and Durability

The toy’s three filing cabinets holding copies of the same partition, and the line marking what every one of them is guaranteed to hold. Follows Kafka 05: Consumer Groups, which split the reading work across many programs, and hands off to Kafka 07: Exactly Once.

Before you start. This series builds a small copy of Kafka, a program that other programs send records to, in order, and read back later in that same order. So far your toy can: write records one after another into a file that only ever grows (the log); cut that file into numbered pieces so old ones can be thrown away (segments); split one topic’s traffic across several of those logs at once, so more than one machine can share the writing (partitions); and let a group of reader programs split up the reading between them (consumer groups). Every one of those logs, so far, has lived on exactly one machine. This lesson is about the day that machine is gone.