Where Data Systems Are Heading

The last board in the Data Systems track. It asks what holds a product’s several copies of the same facts together, and closes on what all that remembering is for.

Before you start. The previous lesson, Stream Processing, showed you the log: an add-only list of what happened, read from the start whenever you want to replay it. This lesson asks what happens when four different systems each want their own copy built from that same list, and what it costs when they don’t.

The idea in 30 seconds

Several stores holding the same fact must be kept in step, and updating each one separately - a dual write - can drift forever without ever throwing an error.
The fix is one add-only list, the log, treated as the source of truth. Every other store is a derived view: nothing but the log, replayed.
A derived view can always be deleted and rebuilt, because it was never more than a function of the log.
The same log, split by key, stops two people claiming the same name without a lock or a message between workers.
“Is this consistent” hides two questions: timeliness (has it caught up - it will) and integrity (is it even correct - nothing but the design fixes that).
The same rule, that only the two ends of a trip can confirm it happened, guards money outside databases too, and what a log is allowed to remember has real costs to real people.

Read the story · Skip to the numbers