The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing
Stop treating a stream as a dataset that will eventually be complete: ask what, where, when and how of every pipeline, and a batch job becomes the case where the answers are all of it, at the end, once.
Read the original ↗Eleven authors, no benchmark, and one refusal: stop treating a stream as a dataset that will eventually be complete. What the paper offers instead is four questions that any pipeline has to answer, and the observation that a batch job is just the pipeline whose answers are 'all of it, at the end, once'.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardStream Processingthe board's panel on event time against processing time, windows, watermarks and late events is this paper's vocabulary in one picture; the paper is where it is argued rather than listed
- BoardBatch Processingthe board's batch philosophy rests on bounded input so that the job completes; the paper treats that as the special case a stream subsumes
- BoardWhere Data Systems Are Headingthe board says the Lambda architecture used both batch and stream and engines then unified them; the paper is the argument for that unification and names the two-systems cost
- LabThe late tapa fixed wait for stragglers is the crudest watermark; drag it and see taps thrown away fall while the seconds waited climb, the paper's too-fast and too-slow trade as one dial
- LessonStream ProcessingUses the board Stream Processing
- LessonBatch ProcessingUses the board Batch Processing
- LessonWhere Data Systems Are HeadingUses the board Where Data Systems Are Heading
- LessonData About PeopleUses the board Where Data Systems Are Heading
Further reading
Read the original for the full argument and its evidence.
- The paper (VLDB 2015, PDF) ↗PVLDB 8(12), pages 1792-1803. Section 2.3 is triggers and refinement modes; section 2.4 runs one data set through every choice; section 3.3 is the evidence.
- Structured Streaming (SIGMOD 2018, PDF) ↗Later work: Spark's reply, which adopts event time, triggers and watermarks and objects to the three-knob API.
- Apache Beam programming guide ↗Later work: the windowing, watermark, trigger and accumulation-mode chapters, with allowed lateness set in the examples.