Kafka 03: Segments and the Offset Index

It draws the log as a shelf of bound volumes instead of one endless scroll, and that picture is this whole lesson. The project this stage belongs to is Build Your Own Kafka - From Scratch; it follows Kafka 02: The Log and hands off to Kafka 04: Partitions.

Before you start. This series builds a small copy of Kafka, the program big companies use to pass events (every click, every order) from the programs that make them to the programs that need them. Other programs send it a stream of records and read them back later, in order. The last stage turned that stream into one file that only grows, where a record’s distance from the start of the file - its offset - doubles as its address.