Stream Processing

Data Systems track, data-in-motion module. This lesson is about what happens once your data stops arriving as a finished file and starts arriving one event at a time, with no end in sight.

Before you start. The previous lesson, Batch Processing, waited until a whole day’s file sat still on disk, then worked through it once, start to finish - that stillness is what let it sort, join, and stop. Nothing here holds still: the input never ends, so a stream processor has to give you an answer before it has seen everything, and stay honest about how wrong that answer might still turn out to be.

The idea in 30 seconds

A stream never finishes, so a stream processor has to answer before it has seen everything, and stay honest about that.
Every event carries two clocks: when it happened (event time) and when the processor heard about it (processing time), and they disagree.
Counting by processing time is easy and can be flatly wrong - a network delay reads as a spike of activity that never happened.
A window is a fixed stretch of time you count events into; a watermark is the processor’s running guess at “I’ve seen everything up to here,” and it is what tells a window when to close.
An event that arrives after its window has closed is late, and the system has to choose: drop it, or correct the count after the fact.
Giving a window more patience catches more late events and delays every answer; there is no setting that is both fast and right.

Read the story · Skip to the numbers