The Google File System
A file system designed from four written-down assumptions about the workload, one of which is that its own machines are always broken.
Read the original ↗GFS is worth reading for its first two pages, before any mechanism appears. It is one of the few systems papers that writes down the assumptions it is designing for, in order, in plain language - and every strange decision later in the paper is one of those assumptions cashed in.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardThe Logappend-only as a storage model, which is GFS's central workload assumption
- BoardCharged Twicethe duplicate record a GFS reader must be written to tolerate, on the payments side of the same problem
- BoardWho Holds the Penone writer, chosen and fenced - the lease GFS hands a primary replica
- BoardThe Disks Are Lying To Yousilent corruption and checksums, the assumption GFS opens on
- BoardReplicationthe three copies of every chunk, and what keeps them in step
- LabCharged twiceat-least-once and the reader's dedup problem, which is GFS's record append seen from above
- LabWho holds the penthe single master, and the split-brain the lease exists to prevent
- LabThe write-ahead logoperation log against checkpoint frequency - exactly the master's recovery trade
- VideoIdempotency keys: how a retry stops being a second chargethe client-supplied id that lets a reader filter a duplicate, which is what GFS asks applications to do
- VideoConsensus: who holds the penwhy one writer at a time is worth this much machinery
- LessonKafka 02: The LogUses the board The Log
- LessonCharged Twice (idempotency & exactly-once)Uses the board Charged Twice
- LessonRaft 01: Leader ElectionUses the board Who Holds the Pen
- LessonRaft Capstone: EverythingUses the board Who Holds the Pen
- LessonWho holds the pen (consensus & Raft)Uses the board Who Holds the Pen
- LessonDropbox - the case studyUses the board The Disks Are Lying To You
- LessonReplicationUses the board Replication
Further reading
Read the original for the full argument and its evidence.
- The paper (PDF) ↗SOSP 2003. Section 2.1 is the assumptions list and section 2.7 the consistency model - those two pages are the paper.
- GFS: Evolution on Fast-forward (ACM Queue, 2009) ↗Later, and honest: Kirk McKusick interviewing Sean Quinlan about what the single master cost as the system grew, and why the metadata layer had to be redesigned.