Kafka 04: Partitions

From Scratch: Kafka. Follows Kafka 03: Segments and the Offset Index, which made one log survive a crash and search fast, and hands off to Kafka 05: Consumer Groups.

Before you start. You need to know that Kafka keeps every message it’s given in one long, append-only file - a log - and that each message gets a number called an offset marking its place in that file, counting up from zero. The last lesson cut that file into numbered, throwaway-able chunks so it could survive a crash and be searched fast; this lesson is about the file itself running out of room to go any faster.

The idea in 30 seconds

One log lives on one machine, so it has a ceiling; splitting a topic into several independent logs, called partitions, splits the ceiling across several machines instead.
More partitions means more machines writing at once, which is where the extra throughput comes from - nothing about any one partition got faster.
The price is exact: order is guaranteed only inside one partition, never between two of them, and nothing records a “true” order across lines.
Which partition a message goes to is decided by its key, turned into a number by a hash, and that number picking the line - the whole rule is the partitioner.
Choosing the key is choosing what stays ordered. Everything else is free to interleave, and that freedom is exactly what buys the throughput.

Read the story · Skip to the numbers