Data Systems
Unix pipe
stream one tool's output into the next
Source board
This definition comes from the board below. Open it to see the term in context.
Batch ProcessingData SystemsMore terms from this board
- batch vs service vs streamoffline bulk / online requests / near-real-time
- bounded inputfinite dataset, so the job finishes
- dataflow engine (Spark/Flink)one DAG, in-memory, no forced materialization
- distributed filesystem (HDFS)replicated blocks across shared-nothing nodes
- map-side (broadcast hash) joinsmall table in every mapper's RAM
- MapReducemap → shuffle(sort) → reduce over a cluster
- materializationwriting full intermediate output to disk
- RDD / lineagerecompute lost data from its recipe
- reduce-side (sort-merge) joinjoin in the reducer after a shuffle
- saltingsplit a hot key
- shufflepartition + sort by key
- workflow/DAGchained jobs