Data Systems
RDD / lineage
recompute lost data from its recipe
Source board
This definition comes from the board below. Open it to see the term in context.
Batch ProcessingData SystemsMore terms from this board
- batch vs service vs streamoffline bulk / online requests / near-real-time
- bounded inputfinite dataset, so the job finishes
- dataflow engine (Spark/Flink)one DAG, in-memory, no forced materialization
- distributed filesystem (HDFS)replicated blocks across shared-nothing nodes
- map-side (broadcast hash) joinsmall table in every mapper's RAM
- MapReducemap → shuffle(sort) → reduce over a cluster
- materializationwriting full intermediate output to disk
- reduce-side (sort-merge) joinjoin in the reducer after a shuffle
- saltingsplit a hot key
- shufflepartition + sort by key
- Unix pipestream one tool's output into the next
- workflow/DAGchained jobs