Module 11 · System Design · Five more whole designs
Design a web crawler (a billion pages a month, politely)
Before you start. You need the six steps from Design the Whole Thing (the interview, assembled): ask, estimate, API, data, design, bottlenecks, with nothing on the whiteboard until a number pays for it. Four Systems in One Night (worked designs) showed that the biggest number in a prompt names the hinge of the design, and here that number is a rate of pages per second. Google Search, the case study, later in this track, tells what a search engine does with the pages after they arrive. This lesson is only the part before: getting them.
Lesson 51 of 73 in System Design
The rest of this lesson is part of a plan.
A plan opens every lesson, board, video, lab and paper on chalk. Yearly is $39/year, or $19.50 for your first year with LAUNCH50, ends 1 Nov, 18:30 UTC.