Kafka 08: Everything

The finale of the Kafka build, tying together everything before it and closing out Build Your Own Kafka - From Scratch.

Before you start. This series builds a small copy of Kafka, a program that lets one part of a system hand a stream of events to many other parts without losing any of them, even if a reader falls behind or a machine restarts. Picture it as a newsroom’s shelf of published articles. So far the copy can do four things. It writes every article onto the shelf and never removes one when someone reads it. It splits the shelf into several side-by-side shelves when one gets too busy. It keeps spare copies of each shelf in other rooms, in case one room burns down. And it lets several teams read the same shelf at their own pace, each keeping its own bookmark. This lesson doesn’t add a new piece. It follows one single event all the way through, so you can watch everything built so far work as one machine instead of several separate ideas.

The idea in 30 seconds

One article’s trip touches every piece the earlier lessons built: which shelf it lands on, who confirms it’s safe to read, who’s allowed to read it, and when it finally gets thrown away.
Reading never deletes anything. A bookmark belongs to whoever is reading, not to the shelf.
Splitting one shelf into partitions is how the system grows, at the cost of only promising order within one partition.
A leader takes every write, replicas copy behind it, and nothing is shown to a reader until the waterline says enough replicas already have it.
A consumer group hands each partition to exactly one of its members, so two teams reading the same shelf never step on each other, and either team can rewind on its own.
All of that is one refusal, not seven tricks: reading must never destroy the record.

Read the story · Skip to the numbers