Batch Processing

Data Systems track, opens the data-in-motion module. This lesson is about a program that runs once over a pile of data that already fully exists, and writes out a brand new copy of the answer - no request in, no reply out, only a start and a finish.

Before you start. You need to know what a hash function does. Feed it anything and it gives back a number: feed it the word “the” and it might give back 10. Divide that number by the number of machines and keep the remainder: with 4 machines, 10 divided by 4 leaves 2, so “the” goes to machine 2, and every copy of “the” lands on that same machine, because the number never changes. Partitioning used this to spread a table’s rows across machines; this lesson reuses it to spread words.

The idea in 30 seconds

A batch job runs once, start to finish, over a pile of data nobody is changing while it runs, and writes out a fresh copy.
Unix pipes are the pattern at one-machine scale: small tools, connected by a stream of text. MapReduce is the same pattern scaled to a cluster.
Map turns each record into a key and a value; reduce combines every value that shares a key.
The shuffle’s whole job is putting every copy of one key on one machine - that’s what lets reducers run with no coordination.
A hot key sends far more rows to one reducer than the rest, which straggles while others idle; hashing usually prevents this, but not for a genuinely popular key.
Writing every step to disk is MapReduce’s own bottleneck. Dataflow engines like Spark fix it by keeping data in memory and recomputing only what’s lost.

Read the story · Skip to the numbers