Kafka: a Distributed Messaging System for Log Processing
The log as the primitive: a broker that remembers nothing about its readers, and consumers that pull at their own pace.
Read the original ↗Six pages at a workshop, and no replication in any of them. The Kafka paper is far smaller than Kafka's reputation, and reading it is the quickest way to find out which famous parts of the system were designed in 2011 and which arrived later because 2011 was not enough.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardThe Naive Queuedeliver-and-delete, and its three walls - the design the paper is arguing against
- BoardThe Logthe append-only file the paper makes the primitive, with the offset as its only address
- BoardSegments and Offset Indexsegment files named by base offset, which is the paper's entire storage engine
- BoardPartitionsthe unit of parallelism, and the only place the paper promises ordering
- BoardConsumer Groupsthe group coordinator and committed offsets - in 2011 both lived in ZooKeeper
- BoardReplication and Durabilitythe ISR and the acks dial, none of which the paper has - replication is its future work
- BoardExactly Oncethe idempotent producer, added six years after the paper declined to build it
- BoardStream Processingthe field the paper's primitive turned into, with the two broker families named
- BoardThe Order Slip Railthe kitchen pulling at its own pace, which is the paper's whole argument for pull
- LabThe order slip railhow fast the wait grows as a puller nears capacity - the lag the pull model hands the consumer
- LabCharged twiceat-least-once and the dedup the paper explicitly makes the application's job
- LabThe write-ahead logcheckpoint against replay time, the same retention trade one layer down
- LabThe slate next to the stovethe eviction policy the paper refuses to implement, on the grounds that the OS already has one
- LabSplitting the ledgera partitioning key that sends every message to one partition
- VideoEvent-driven architecture: the ledger everyone readstopic, offset, retention, replay, CDC - the paper's model taught in one episode
- VideoQueues and pub-sub: work that outlives the requesta queue divides, a log copies, and exactly-once is not a thing you can buy
- VideoQueues: the order slip rail and the cliffproducer, broker, consumer, and the cliff a slow consumer sits on
- VideoExactly-once: the outbox, the retry chain, and reconciliationthe outbox and the saga - what the paper's one-sentence dismissal actually costs
- VideoIdempotency keys: how a retry stops being a second chargethe client-supplied key that makes at-least-once survivable
- LessonKafka 01: The Naive QueueUses the board The Naive Queue
- LessonKafka 02: The LogUses the board The Log
- LessonKafka 03: Segments and the Offset IndexUses the board Segments and Offset Index
- LessonKafka 04: PartitionsUses the board Partitions
- LessonKafka 05: Consumer GroupsUses the board Consumer Groups
- LessonKafka: Replication and DurabilityUses the board Replication and Durability
- LessonKafka 07: Exactly OnceUses the board Exactly Once
- LessonStream ProcessingUses the board Stream Processing
- LessonThe order slip rail (queues & the utilization cliff)Uses the board The Order Slip Rail
- LessonThe stall that turns people away (load shedding and admission control)Uses the board The Order Slip Rail
- LessonThe stall, built: rush hour (load testing, p99 and the knee in Go)Uses the board The Order Slip Rail
Further reading
Read the original for the full argument and its evidence.
- The paper (PDF) ↗NetDB 2011, six pages. Section 3 is the whole design; section 6's future-work list is the most useful page in it, because most of what it promises is what Kafka later became.
- The Kafka documentation ↗The current system, for reading against the paper. Replication, log compaction and exactly-once semantics are all here and none of them are there.
- KRaft ↗Later work: the removal of the ZooKeeper dependency the paper's entire coordination section is built on.
- Exactly-once semantics in Kafka ↗Later work, 2017: the idempotent producer and transactions, and what they do and do not actually promise.