Kafka 07: Exactly Once

Sits under the project this build is part of, Build Your Own Kafka - From Scratch, filling in the one paragraph that lesson gives to exactly-once delivery.

Before you start. This series builds a small copy of Kafka, a program that keeps one shared, ordered notebook of everything that happened and lets any number of other programs read it at their own pace, instead of passing notes hand to hand. So far the notebook has copies on more than one machine called a broker, so one machine dying doesn’t lose it (Kafka: Replication and Durability), and many readers can split the work of going through it without re-reading each other’s pages (Kafka 05: Consumer Groups). A producer is any program adding new lines to it; a topic is the notebook’s name, split into ordered slices called partitions, one or more per broker. This lesson is about the one thing none of that fixes: a program can never be completely sure whether a line it tried to write actually landed.

The idea in 30 seconds

You can never tell a lost line from a lost reply, so you must resend, and resending guarantees duplicates.
Kafka’s fix is a number that only counts up per producer, per slice; a resend carrying an already-stored number is dropped, and the sender is still told it succeeded.
That number only protects one run of one producer writing to one slice; a restart or a write spanning several slices needs more.
A transaction is the bigger unit: an id you choose yourself survives a restart and locks out any old copy still running, and a commit lands everywhere or nowhere.
A given-up-on transaction’s lines stay physically in the notebook; a reader asking for only the real ones skips them, and pays for that by sometimes waiting.
None of this reaches past Kafka. Once a line lands somewhere else, a database, an API call, the same check-number idea is your job again.

Read the story · Skip to the numbers