Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Fault-tolerant shared memory on a cluster is cheap if nobody may write to it one cell at a time, because then you can log how data was made instead of the data.
Read the original ↗Spark is remembered as MapReduce kept in memory. The paper's actual claim is stranger and more useful: you can have fault-tolerant shared memory on a cluster, as long as nobody is allowed to write to it one cell at a time. Everything good in the design, and most of what has aged, comes from that one restriction.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardBatch Processingthe board's dataflow-engines panel names RDDs and lineage in a line and complains that MapReduce materialises every step; this paper is where that line comes from
- BoardStream Processingthe board's microbatching and checkpointing line is the recovery story this paper's section on batched stream processing starts in one sentence
- BoardWhere Data Systems Are Headingthe board says engines like Spark and Flink unified batch and stream, which is the arc that starts here and takes three more papers to arrive
- LabThe write-ahead loga lineage chain is a log to replay after a crash, and the paper checkpoints long chains for the reason a database does; the lab runs that trade with writes in place of transformations
- LessonBatch ProcessingUses the board Batch Processing
- LessonStream ProcessingUses the board Stream Processing
- LessonWhere Data Systems Are HeadingUses the board Where Data Systems Are Heading
- LessonData About PeopleUses the board Where Data Systems Are Heading
Further reading
Read the original for the full argument and its evidence.
- The paper (USENIX NSDI 2012, PDF) ↗Section 2.3 and Table 1 are the argument against shared memory; section 4 is narrow and wide dependencies; section 6.1's 'Understanding the Speedup' is the paragraph the abstract leaves out.
- Spark SQL: Relational Data Processing in Spark (SIGMOD 2015, PDF) ↗Later work, by the same group: the DataFrame API and the Catalyst optimizer, the reply to the RDD's closures.
- Discretized Streams (SOSP 2013, PDF) ↗Later work: streaming built from small batch jobs on this paper's recovery mechanism, which the Spark documentation now calls legacy.
- Structured Streaming (SIGMOD 2018, PDF) ↗Later work: event time, triggers and watermarks, arrived at by way of the Dataflow model and a SQL-shaped API.