Index

Free, with Google

Keep going, free.

A free account opens lessons 2 and 3 of every track, 33 more lessons, and keeps your progress and review cards on every device.

Start free with Google

Launch price

Open everything on chalk.

Every lesson, board, video, lab and paper, and the full review decks. Yearly is $39/year, or $19.50 for your first year with LAUNCH50, ends 1 Nov, 18:30 UTC.

See plansStart free instead

Module 05 · System Design · Deeper than the interview

Watch: Sharding: splitting the ledger
▶ Video

Changing the engine in flight (live schema and data migrations)

Splitting the Ledger
Splitting the Ledger · Open board

The ledger cut into books, drawn at rest. This lesson goes past it: what it takes to change the shape of a ledger that is being written to every second, with no night to close the shop.

Before you start. You need three ideas from earlier. A ledger can be cut into pieces so that more hands write at once, which is Splitting the ledger (sharding). A ledger can have copies that trail the original by a moment, which is The copy clerks (database replication). And different jobs want different kinds of book, which is The Right Book for the Job (databases, deeply). This lesson is about the day you must move from one book to another, or change what a page looks like, while people are still writing in it. What a page looks like is its schema: the list of fields a record has, such as a name, a price and a date.

Lesson 27 of 73 in System Design

The rest of this lesson is part of a plan.

A plan opens every lesson, board, video, lab and paper on chalk. Yearly is $39/year, or $19.50 for your first year with LAUNCH50, ends 1 Nov, 18:30 UTC.

See plansAlready a member? Sign in

Or start free: a free account opens lessons 2 and 3 of every track.

In this lesson

  • 2017: Stripe, and a record that assumed one of something
  • Why you cannot just run one big change
  • Step one: expand, add the new road beside the old
  • Step two: dual write, every new change goes to both roads
  • Step three: backfill, fill in the history
  • Step four: shadow read, trust but check
  • Step five: cutover, then contract

Terms it explains: schema, row, column, online migration, expand and contract, dual write, backfill, shadow read, cutover, consistent hashing, cross-shard query, hot shard, range vs hash partitioning, resharding, shard, shard key (partition key).

Lab

Names are mostly A–D. You split A–M / N–Z. What happens?

Start on Split by letter, read the load bars under The books, then switch to Scrambled name and press Add a book.

Open Splitting the ledger

The ring rehomes only 1/N of the keys when a server dies. So why does everyone run two hundred virtual nodes per server?

Drag Points on the ring per server from one upward, stay on Walk clockwise to the next server, and watch What the balance costs.

Open The ring

Related papers

Review System Design