Data Systems
dataflow engine (Spark/Flink)
one DAG, in-memory, no forced materialization
Source board
This definition comes from the board below. Open it to see the term in context.
Batch ProcessingData SystemsMore terms from this board
- batch vs service vs streamoffline bulk / online requests / near-real-time
- bounded inputfinite dataset, so the job finishes
- distributed filesystem (HDFS)replicated blocks across shared-nothing nodes
- map-side (broadcast hash) joinsmall table in every mapper's RAM
- MapReducemap → shuffle(sort) → reduce over a cluster
- materializationwriting full intermediate output to disk
- RDD / lineagerecompute lost data from its recipe
- reduce-side (sort-merge) joinjoin in the reducer after a shuffle
- saltingsplit a hot key
- shufflepartition + sort by key
- Unix pipestream one tool's output into the next
- workflow/DAGchained jobs