The stall, built: copies and shards (replication and sharding in Go)

The Chai Stall That Got Famous · Open board

The famous stall, now a row of stalls behind one gate. Every stall in that picture still writes into one book. This lesson copies the book, then cuts it into three, in code.

Before you start. The stall, built: one counter in code (load balancing, health checks and caching in Go) built the stall as a Go program: a gate that sends each customer to the least busy of three stalls, a slate that remembers prices, and a rail of order slips, all with one book behind them. The copy clerks (database replication) gave that book copy clerks: the leader’s book takes every write, and each clerk copies it a moment later, so readers can use the copies, and a copy is always slightly behind. Splitting the ledger (sharding) cut the book itself: different customers in different books, with one field, the shard key, deciding which book a customer lives in. This lesson builds both and measures them.

The idea in 30 seconds

A replica is a second copy of the data that readers can use, and it runs behind the original by the replication lag, measured here at about 5 ms in the middle with a 10 ms copy round.
A stale read is a reader shown a value that has already been replaced; it is what a replica does by design, not a fault.
Reading your own write back from the original, not the copy, removes the stale read for the person who just wrote.
A shard holds one slice of the data; cutting by ranges of id let the busiest slice take 53% of the writes here, and cutting by a hash kept every slice near a third.
Copies protect against losing a machine and spread reads, but they do not relieve a shard that is too full, because every write reaches every copy.
Moving from three shards to four by remainder moves about three quarters of the keys.

Read the story · Skip to the numbers