Encoding and Schema Change

A board in the Data Systems track’s first module, data on one machine. It draws the shape a program’s data takes the instant it leaves memory, and what has to be true about that shape for a stranger, or a future version of the same program, to read it back correctly.

Before you start. The previous lessons, Storage Engines and Storage Extras: richer indexes, warehouse schemas, column tricks, were about how a database lays bytes out on disk so it can find them again fast. This one is about what those bytes look like in the first place, and why a byte written five years ago still has to make sense to code written this afternoon.

The idea in 30 seconds

Encoding is flattening an object sitting in memory into a row of bytes, for a file, a network, or another program to read later.
Rolling deploys mean old and new code run side by side, so an encoding needs two separate promises: backward compatible, meaning new code reads old data, and forward compatible, meaning old code reads new data without breaking.
JSON keeps its promise the honest way, by spelling every field’s name out in full, at the cost of repeating those names in every record.
Protocol Buffers and Thrift use a schema instead: a legend, shared by every version of the code, giving each field a number, its field tag, so an old reader can skip a field it has never seen just by knowing how wide it is.
Avro drops even the tag, and has the reader carry two schemas, the writer’s and its own, matching fields up by name.
The same looseness that makes JSON friendly to read also broke Twitter’s ids in 2010: a plain JSON number can’t hold a 64-bit integer without silently rounding it.

Read the story · Skip to the numbers