Write-heavy and streaming systems

These prompts test whether you notice that nobody is waiting on any single write. A metric sample, a click, a log line: each one is cheap, there are hundreds of thousands a second, and what anyone reads is an aggregate minutes later. So the design is a log you append to, partitioned by some key, read by consumers at their own pace. The trap is drawing a database behind an API server and writing every event into it synchronously, which works on the whiteboard and falls over at the first number you compute. The second trap is forgetting that a consumer will fall behind, and having no answer for where the backlog goes.

When writes dominate, the log is the spine

Start every one of these the same way: compute the write rate, then put an append-only log (Kafka, Kinesis, Pulsar, whatever you can defend) between the producers and everything that does work. Producers append and return. Consumers read at their own pace and keep their own offset. The log is what lets a slow indexer, a crashed aggregator or a provider outage turn into a backlog instead of lost data or a stalled producer.

Then say the two things the rest of the answer hangs on. First, the partition key: it decides ordering (only within a partition), parallelism (one consumer per partition in a group) and hot spots (one key with a third of the traffic is one partition with a third of the traffic). Second, what happens when consumers fall behind: how much retention covers the backlog, what you drop or sample first, and how the consumer catches up without redoing work twice. An answer that names both of those out loud in the first five minutes is already ahead of most.

Partition key: Pick the key whose events must be processed together or in order: a series, an ad, a user, a host. Then ask what happens if one key is enormous, because one always is.
Consumer lag: The distance between the newest offset written and the offset a consumer has processed. Retention has to be longer than the worst lag you plan to survive, or the backlog is deleted before it is read.
Idempotent consumer: A consumer that restarts from its last committed offset will see some events again. Make processing safe to repeat (dedup by event id, or write with a key that overwrites) rather than promising it never happens.

Questions

Answer each one yourself before you open it.

01Design a metrics and alerting system for a fleet of 10,000 servers.
The hinge is cardinality, not throughput: the thing that breaks a metrics system is the number of distinct time series, so the design is a time-series store keyed by series, appended in time order and compressed, with alerts evaluated against recent data as it arrives. Agents push samples into a partitioned log, ingesters partitioned by series id write compressed chunks, old data is downsampled, and a separate rule evaluator queries the last few minutes and fires alerts.
Interview assumptions: 10,000 hosts, 500 series each, one sample every 10 seconds. That is 10,000 × 500 = 5 million series, and 5,000,000 / 10 s = 500,000 samples/s. A raw sample is a timestamp and a value, 16 bytes, so 500,000 × 16 B × 86,400 s ≈ 700 GB/day raw. Delta-of-delta timestamps and XOR'd float values get a sample down to a couple of bytes, so 500,000 × 2 B × 86,400 ≈ 90 GB/day, which is why compression is the storage design, not an optimisation. Components: agents, a log partitioned by series, ingesters holding the open chunk per series in memory, object storage for closed chunks, a query layer, and an alert evaluator with its own deduplicating notifier. Replace the series-per-host number first with real traffic: it multiplies everything, and it is the one teams underestimate.
Follow-up A team adds user_id as a label on their request-latency metric. What happens?
Every distinct user becomes a new series, so one metric with a million users is a million series overnight and the ingesters run out of memory holding open chunks. Enforce a per-metric series limit at ingest and reject or drop the offending label, and tell the team to use logs or traces for per-user detail. Cardinality is a budget you police at the front door, not something you scale your way out of.
Learn more The Watchtower (observability) · LSM vs B-tree
02Design a system that counts ad clicks in real time, per ad per minute, for billing and a live dashboard.
The hinge is that billing needs exact counts and the dashboard needs fast ones, so you keep the raw click log as the source of truth and aggregate from it in a stream processor keyed by ad id over one-minute windows. Clicks land in a log partitioned by ad id, a stream job deduplicates by click id and sums per window, the small aggregates go to a store the dashboard reads, and a periodic batch job recounts from the raw log to reconcile what gets billed.
Interview assumptions: 1 billion clicks a day, peak five times the average, 1 million active ads, 100 bytes per raw click. Average is 1,000,000,000 / 86,400 ≈ 10,000 clicks/s, so peak ≈ 60,000/s. Raw log is 1,000,000,000 × 100 B = 100 GB/day, cheap to keep. The aggregate is tiny by comparison: at most 1 million counters per minute, and at roughly 50 bytes each that is 1,000,000 × 50 B = 50 MB of window state, which fits in memory on a handful of stream workers. That gap, 100 GB in and 50 MB of state, is the whole argument for aggregating in the stream rather than counting rows in a database. Replace the peak multiplier first: ad traffic is spiky around events, and it sizes the partitions.
Follow-up One ad runs during a live final and takes 30% of all clicks. Your ad-id partitioning now has a hot partition. What do you do?
At a 60,000/s peak, 30% is about 20,000 clicks/s on one partition and one consumer. Salt the key for hot ads (ad id plus a suffix 0 to N), pre-aggregate each salted slice in parallel, and sum the slices per window in a second small stage. You pay a second hop and a slightly later count for that one ad, and every other ad is unaffected.
Learn more Stream Processing · Charged Twice (idempotency & exactly-once) · Kafka 04: Partitions
03Design a notification system that sends push, email and SMS for a product with 100 million users.
The hinge is that the bottleneck is not your servers but the providers downstream (the push services, the email relay, the SMS gateway), each with its own rate limits, outages and costs. So the design is: a notification API that checks preferences and per-user rate limits, then enqueues onto a separate queue per channel, with workers that call the provider, retry with backoff, and carry an idempotency key so a retry never sends twice.
Interview assumptions: 100 million users, 5 notifications per user per day, and a marketing blast to every user within an hour. Daily volume is 100,000,000 × 5 = 500 million, so 500,000,000 / 86,400 ≈ 6,000/s on average. The blast alone is 100,000,000 / 3,600 s ≈ 30,000/s, five times the everyday rate, which is why blasts go on their own lower-priority queue instead of sharing one with password resets. Components: an ingest API, a preferences and templates store, a scheduler for delayed and batched sends, one queue per channel (and per priority), channel workers, and a delivery-status log fed by provider callbacks. Replace the blast size first: it, not the daily average, sizes the workers and the provider contracts.
Follow-up Your SMS provider goes down for 20 minutes. What happens to the messages?
The SMS queue absorbs them: if SMS is 1% of the average 6,000/s, that is 60/s × 1,200 s ≈ 70,000 messages of backlog, which is nothing for a queue. Fail over to a second provider if you have one, and give every message a time-to-live, because a one-time login code that arrives 20 minutes late is worse than one that never arrives. When the provider returns, drain with a rate limit so you do not trip its throttling on the way back.
Learn more The order slip rail (queues & the utilization cliff) · The bouncer at the door (rate limiting) · Charged Twice (idempotency & exactly-once)
04Design a centralized logging pipeline for 10,000 containers, searchable by engineers during an incident.
The hinge is that collecting logs is cheap and indexing them is expensive, so you split the two: agents ship to a durable log buffer, and indexers consume from it at whatever pace the search cluster can take. Recent logs sit in a searchable index for a few days, and everything is also written compressed to object storage, where older logs are still retrievable, just slower.
Interview assumptions: 10,000 containers, 100 lines/s each, 200 bytes a line. That is 10,000 × 100 = 1 million lines/s, and 1,000,000 × 200 B = 200 MB/s, so 200 MB/s × 86,400 s ≈ 20 TB/day. Seven days hot is 20 TB × 7 ≈ 100 TB before index overhead, which is why the hot tier is days, not months, and why most of the history lives in object storage. Components: node agents that batch and compress, a log partitioned by service, indexers, a search cluster with time-based indices you drop whole, an archiver to object storage, and a query UI. Replace the lines-per-container number first: it varies by orders of magnitude between services, and one noisy service usually dominates.
Follow-up During an incident one service starts logging an error in a tight loop at ten times its normal volume, and the indexers fall 30 minutes behind. Now what?
The buffer is doing its job: 30 minutes of the whole 200 MB/s is 200 MB × 1,800 s ≈ 400 GB, so size retention to cover hours, not minutes. Put per-service quotas at the agent or the ingest edge, sample or drop DEBUG and repeated identical lines first, and give the indexers a priority lane so the incident's own services stay searchable. The worst outcome is the logging pipeline going dark exactly when the incident needs it, and quotas are what prevent that.
Learn more Kafka 02: The Log · The order slip rail (lab) · The Watchtower (observability)
05Design a web crawler that fetches 1 billion pages a month.
The hinge is the URL frontier: what to fetch next, under the constraint that you must be polite to every host. So you partition the frontier by host, one worker owns each host's queue and enforces a delay between requests to it, a priority score decides which hosts and pages come first, and a seen-set of URL hashes stops you fetching the same page twice.
Interview assumptions: 1 billion pages a month, 100 KB a page. That is 1,000,000,000 / 30 / 86,400 ≈ 400 pages/s, and 400 × 100 KB = 40 MB/s, about 300 Mbit/s of download, which is modest. Storage is 1,000,000,000 × 100 KB = 100 TB a month before compression. The seen-set is 1 billion × 8-byte hashes = 8 GB, which fits in memory sharded across a few machines, or smaller as a Bloom filter if you accept a few pages wrongly skipped. Throughput is not the hard part; politeness and choosing what to crawl are. Components: the frontier (per-host queues plus a priority layer), fetchers with DNS caching and robots.txt handling, a content store, a parser that extracts links back into the frontier, and a dedup step for URLs and near-duplicate content. Replace the page-size assumption first: it sets bandwidth and storage both.
Follow-up One site has 10 million pages, and another is a calendar that generates an infinite number of URLs. How does your crawler cope?
At one request a second, 10 million pages takes 10,000,000 s ≈ 100 days, so you give every host a crawl budget based on its priority and stop there rather than letting it pin a worker. The calendar is a spider trap: cap URL depth and length per host, and notice when new URLs keep producing near-identical content. Both answers are the same idea, a per-host budget, because the frontier is where you decide what is worth fetching at all.
Learn more Graphs (BFS, DFS, Dijkstra, topological sort) · Google Search - the case study · Splitting the ledger (sharding)

← Read-heavy systems Storage and consistency →