Kafka 05: Consumer Groups

Follows Kafka 04: Partitions and hands off to Kafka 07: Exactly Once.

Before you start. Kafka 04: Partitions split one topic, a named stream of messages such as every order a shop takes, into several partitions - separate ordered logs a producer can write to in parallel, so writes stopped being capped at one machine’s disk. That lesson never said who is allowed to read a partition, and that gap is this lesson’s whole subject.

The idea in 30 seconds

A consumer group is a promise: within the group, each partition belongs to exactly one member at a time.
The number of partitions is the ceiling. More group members than partitions just means idle members with nothing to do.
A coordinator, an ordinary broker with one extra job, computes the split, so members stop guessing from a shared record and start receiving an answer.
Membership changing at all - a join, a crash, a deploy - triggers a rebalance, and the safest kind stops the whole group until it’s done.
The offset, each member’s bookmark, lives in Kafka itself, not in the consumer’s own memory, because the consumer is the thing that’s going to crash.
Processing and marking-it-done are two separate steps, and a crash between them means you either lose a record or duplicate one - never neither, at this layer alone.

Read the story · Skip to the numbers