Build Your Own Kafka - From Scratch

The whole build laid out as one map, in the order each stage’s own problem forces the next.

Before you start. This series builds a small copy of Kafka: a program that keeps a list of everything that happened, in order, and lets any number of programs each read that list from wherever they left off. You need to know only that a file is a named place on disk you can add bytes to and read back later; if you’ve already built the first system in this track, you’ve seen the same length-prefixed record trick this one reuses.

Code: teaching/code/from-scratch/kafka/ - one Go module, each stage runs alone: go run ./01-queue … go run ./07-semantics. 02 and 03 write real log/segment files you can inspect and delete.

The idea in 30 seconds

Twenty private pipes between twenty teams is worse than it sounds; one shared record that anyone can read at their own pace replaces all of them.
A queue throws a note away once it’s read, which breaks the moment a second reader, a crash, or a slow reader shows up.
A log never throws a note away. Its position, the offset, belongs to the reader, not the book.
One notebook caps out, so the record is split into partitions, held by a broker, and shared out among readers by a consumer group.
A broker also copies each partition to another machine, and catches the same write arriving twice - the two problems every copy-and-retry system runs into.
Kafka is named after the writer Franz Kafka, because it’s a system built for writing things down.

Read the story · Skip to the numbers