Modern System Design Interview

Prerequisites

Seniority is a poor readiness signal. Prerequisite mastery is a good one. Answer every Quick check in under a minute. Two or more misses means start with the linked material, not with Module 1.

Time budget

Learning objectives

Each objective is tested by exactly one numbered item in the mastery check, and the numbering matches: objective 1 is tested by mastery item 1.

1. Produce five clarifying questions from a one-line prompt in under four minutes, where every question has two plausible answers that lead to visibly different architectures, and no question survives that fails that fork test.
2. Convert a stated traffic figure and retention window into storage, bandwidth and machine-count estimates, showing each arithmetic step, and state for every number the design option it eliminates, deleting any number that eliminates nothing.
3. Draw a v0 design that satisfies the stated requirements using the fewest components the requirements permit, then name the specific metric and the specific threshold at which it fails, plus the one component you add next.
4. Choose between two replication or partitioning options for a given read-to-write ratio and data size, citing the number that decides it and the condition under which the rejected option would win.
5. Diagnose a described production symptom into exactly one named failure class from the failure library, and name the design change that removes it, in under two minutes and without proposing a retry.
6. Defend a single-node or single-region design against an interviewer pushing for distribution, citing the throughput or data-size threshold below which distribution costs more than it buys.
7. Rewrite a mid-level answer into a staff-level answer by adding blast radius, migration order, cost per unit of traffic and one operability commitment, annotating what each addition changed about the decision.

Warm-up retrieval

Three to five minutes. Answer out loud or in writing before opening anything. Retrieval beats rereading, and a visible answer converts one into the other.

Q1. From the hub's archetype router: name two signals inside the first minute that tell you this is a generalist distributed-systems round and not a product architecture round.

Q2. From the trade-off atlas on the hub, several pages back: state the decision rule for putting work behind a queue instead of doing it synchronously, and name the number that decides it.

Q3. The hub states plainly what a written course cannot do. What are the two things, and what does this section substitute for them?

Module map

+--------------------------------+
| FRAME              M1 to M4    |
| scope, clock, requirements,    |
| estimation that eliminates     |
+---------------+----------------+
                |
                v
+--------------------------------+
| PROPERTIES         M5          |
| availability math, durability, |
| consistency spectrum           |
+---------------+----------------+
                |
                v
+--------------------------------+    +---------------------+
| BUILDING BLOCKS    M6 to M9    |    | HARD PARTS          |
| request path, storage,         |==> | M10 multi-region    |
| movement, coordination         |    | M11 failure library |
+---------------+----------------+    +----------+----------+
                |                                |
                v                                v
+-------------------------------------------------------+
| JUDGEMENT                              M12 to M16     |
| operability, cost, simplicity, brownfield, levels      |
+-------------------------------------------------------+

Caption: the five clusters of this course and their reading order.
Frame feeds Properties, which feeds Building Blocks. Building Blocks
feeds Hard Parts. Both feed Judgement, which is where mid-level and
staff-level answers separate. Arrows point from prerequisite to
dependent; nothing points backwards.

Low ends total 720 minutes, high ends total 900. That is the 12 to 15 reading hours in the time budget above.

Method note for the Time column: it is reading only, computed per module at 120 to 150 words a minute. Each of Modules 1 to 8 repeats its reading range inline under its own heading, unchanged, and adds a practice range. The inline practice ranges for Modules 1 to 5 sum to 90 to 120 minutes, which is the 1.5 to 2.0 hours in the practice column above.

The delivery clock for this round

This page's position, and the reason the tables below exist: a design round is lost on time allocation far more often than on missing knowledge. Two failure shapes account for most of it. Running out of minutes with no design on the board leaves nothing to grade, and spreading 45 minutes evenly across every topic leaves nothing deep enough to grade either. The clock is a budget, not a script.

The 45 minute round

The phases sum to 45 by construction: 2 plus 6 plus 4 plus 8 plus 10 plus 10 plus 5.

Where these minute counts come from. They are this page's allocation, not a measured average of real rounds, and there is no dataset behind them.

One rule generates them: give each phase the smallest block in which its output can actually be produced, then spend whatever is left on depth, because depth is the only phase whose value keeps rising with time.

Treat the row boundaries as soft and the totals as hard. If your own timed mocks say you need eight minutes on requirements, take them from the deep dive and know that you did.

The 60 minute round

The extra 15 minutes buys one more deep dive and a real operability section. It does not buy more boxes. These sum to 60: 2 plus 8 plus 5 plus 9 plus 12 plus 16 plus 6 plus 2.

The first two minutes

Say four sentences, in this order. Rehearse them once so they are automatic and you can spend attention on listening.

1. "Let me restate it: you want X for roughly Y users, and the part that matters most is Z." (Restate, and put a number in it even if you invented the number.)
2. "I am going to spend about five minutes on requirements, then draw the simplest thing that works, then break it on purpose." (You just told them your clock, which reads as control.)
3. "Two things I will optimise for: A and B. I will trade away C to get them." (Naming what you sacrifice is the single cheapest senior signal available.)
4. "Stop me if you want depth somewhere specific rather than coverage." (It costs one sentence, and if it is taken you get to spend the round where the grading is.)

Declaring what you are skipping

Skipping silently reads as not knowing. Skipping out loud reads as prioritising. The pattern is: name it, say why it is out of scope, say what you would do if it were in scope, in one sentence.

• "I am not designing auth here. It is a solved shape and it does not interact with the write hot spot, which is the interesting part. If you want it, it is a token check at the gateway."
• "I am skipping the analytics path. It is asynchronous and it never blocks a user request, so it cannot change any decision we make in the next twenty minutes."
• "I will not size the CDN. Nothing in this design changes based on whether it is 2 or 20 terabytes a day."

Checking in without sounding unsure

The difference is that a weak check-in asks for approval and a strong one offers a fork.

How the split shifts by level

The rough shape, stated as this page's position rather than as a report on what happens in real loops: mid level spends its minutes proving the design is correct, senior spends them proving the trade-off was chosen, staff spends them proving the design is reachable, operable and affordable. It follows from what each level is accountable for once hired, which is the only defensible basis for a claim like this.

Module 1. Scope and honesty: which round is this

Time: 15 to 20 minutes reading, 10 to 15 minutes practice.

1a. Predict first

Read these five opening lines. Each is something an interviewer has plausibly said in the first minute of a round.

The archetypes, and what each one actually scores

A "system design interview" is not one round. At least six distinct rounds use the phrase, and they score different things. Preparing for the wrong one is the most expensive mistake available to you, because it costs weeks, not minutes.

How to tell which round you are in, inside 90 seconds. Listen for the noun in the prompt.

• A product name or user-facing verb ("shorten", "feed", "chat") means backend design.
• The words "interface", "endpoint", "partner", "SDK" mean contract design.
• The words "predict", "rank", "detect", "recommend" mean machine learning design.
• The words "class", "object", "implement" mean object design.
• A named running system plus a symptom means brownfield.

The rounds disagree with their titles more often than anyone admits. A recruiter's calendar invite says "System Design" and the interviewer runs a deep dive on their own team's queue. Your response is not to complain, it is to re-aim in one sentence.

On the name of this course category. Several paid products ship under nearly identical "Grokking" titles from different vendors, and at least one sits on a URL history that makes the two easy to confuse. That is a purchasing problem, not a learning problem. Check the vendor name on the checkout page. We mention it once and move on.

The routing diagram

        prompt heard in the first minute
                     |
       +-------------+-------------+
       |             |             |
  product noun   contract noun  model noun
       |             |             |
       v             v             v
 +-----------+  +-----------+  +-----------+
 | backend   |  | interface |  | ML system |
 | design    |  | design    |  | design    |
 +-----------+  +-----------+  +-----------+
       |
   is a running system named
       |
   +---+----+
   |        |
   v        v
+--------+ +----------+
| brown  | | green    |
| field  | | field    |
+--------+ +----------+

Figure routing the first minute of a round to one of five archetypes

Worked example: a recruiter email arrives

The email says: "You have a 60 minute System Design round with a senior engineer from the Payments Platform team. Please be ready to discuss scalability."

Block 1. PURPOSE: extract the scoring surface from the words I was actually given.

I have three signals: 60 minutes, Payments Platform, and the word scalability. The team name is the strongest of the three. A payments platform team spends its days on idempotency, ledger correctness and third-party failure, so a payments interviewer usually probes correctness under retry before they probe throughput. The word "scalability" is recruiter boilerplate and I weight it near zero.

The error most readers make here is treating recruiter adjectives as requirements. "Scalability" appears in almost every invite. It carries no information, so it cannot change my preparation.

Block 2. PURPOSE: decide what I will deliberately not prepare.

Given a payments team and 60 minutes, I drop object design drills and machine learning material entirely. I also drop deep video and media pipeline material. That frees roughly the whole of my preparation time for three things: exactly-once effects, ledger and reconciliation, and gateway failure handling.

The error most readers make here is additive preparation: adding payments material on top of everything else and running out of hours. Preparation is subtraction.

Block 3. PURPOSE: buy information cheaply before the round.

I send the recruiter one message: "Is the round greenfield design, or a walkthrough of an existing system? And is the interviewer looking for breadth or one deep dive?" This is a normal question, and recruiters usually answer it.

Block 4. PURPOSE: write the opening thirty seconds that steers a mislabelled round.

I write and rehearse two sentences. "I would like to spend the first five minutes on requirements and the numbers that matter, then draw the smallest thing that works, then break it deliberately. If you would rather go straight to depth on one area, tell me now and I will restructure."

The error most readers make here is improvising the opening. The opening is the only part of the round that is fully predictable, so it is the only part worth memorising.

Result. The reply came back: "Existing system, they want to see how you would extend it." That single sentence moved my preparation from greenfield drills to the brownfield checklist: dual writes, backfill, shadow traffic and rollback.

1e. Fade the scaffold

Faded example 1. The invite says: "45 minute technical design round, Search Infrastructure team, please be prepared to write on a shared document." Work through blocks 1 to 3, then produce block 4 yourself.

• Block 1: the signals are 45 minutes, Search Infrastructure, and a text-only tool.
• Block 2: a search infrastructure team probes index build cost, freshness lag, and query fan-out. I drop transactional and payments material.
• Block 3: I ask the recruiter whether the round is component depth or end-to-end.
• Block 4: you write the opening thirty seconds.

Faded example 2. The invite says: "60 minutes with a staff engineer, topic to be chosen on the call." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the signals are 60 minutes, staff level, and an unspecified topic.
• Block 2: an unspecified topic means I cannot subtract by domain, so I subtract by archetype instead and prepare the general arc rather than any one product.

1f. Check yourself

Q1. An invite says "System Design" and the first line of the round is "Design the retry and error semantics our mobile clients see." Which archetype are you in, and what is the first thing you change about your plan?

Q2. You are twenty minutes into a round you believed was greenfield, and the interviewer says "assume all of that already exists in production today". What just changed?

1g. When not to use this

Round triage is a preparation tool, not a ritual. It is over-applied more often than under-applied.

Module 2. The delivery clock

Time: 25 to 30 minutes reading, 15 to 20 minutes practice.

2a. Predict first

Here is a real-shaped timeline from a 45 minute round that did not pass.

minute 0    greeting and introductions
minute 2    interviewer states the prompt
minute 3    candidate draws client box balancer box service box
minute 9    interviewer asks what the write volume is
minute 11   candidate gives first requirement
minute 18   candidate adds a queue a cache and a wide column store
minute 26   interviewer asks how a top ten list is produced
minute 31   candidate says we could shard that
minute 38   interviewer stops the design and asks for questions
minute 45   end

Figure a minute by minute trace of a round that was lost early

The budget, and where it comes from

A 45 minute slot is not 45 minutes of design. Subtract the parts you do not control.

The same subtraction on a 60 minute slot leaves 52 working minutes (60 minus 2 minus 6, where the longer slot usually gets a slightly longer question block).

Allocate the working minutes to phases that produce score. Each figure below is a target with roughly plus or minus 2 minutes of slack, and each column sums to the working total.

Three checkpoints, and the sentence to say at each. Say these out loud. They buy you correction opportunities and they read as control, not doubt.

How the split shifts by level. The total is fixed, so every addition is a subtraction.

The clock as a diagram

45 MINUTE ROUND  each block is one minute

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|intro| clarify + numbers |  V0  | break to V1 |deep dive|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
 0    2                  13     18            25        32

+--+--+--+--+--+--+
| fail | wrap | Q |
+--+--+--+--+--+--+
32    35    36   45

checkpoints
  minute 8  = confirm the problem
  minute 20 = declare V0 done and break it
  minute 33 = offer the interviewer the last choice

Figure the 45 minute budget with three spoken checkpoints

Worked example: running the clock on a small prompt

The prompt at minute 2: "Design a service that stores and serves user avatar images."

Block 1. PURPOSE: spend the first six minutes buying constraints, not drawing.

I ask four questions and stop. How many users and how often do they change an avatar. What sizes do clients request. Is the image public or access controlled. Does an upload have to be visible immediately or is a few seconds acceptable. I get: 50 million users, an avatar changed on average once a year, four rendered sizes, public, and a few seconds is fine.

I say the derived shape out loud: this is an extremely read heavy, almost immutable, small object workload with a tiny write rate.

The error most readers make here is asking questions that cannot fork the design, such as "what is the tech stack". A question earns its 30 seconds only if two different answers produce two different drawings.

Block 2. PURPOSE: convert answers into two numbers that eliminate options.

Writes: 50 million changes per year. A year is about 3 times 10 to the 7 seconds, so that is under 2 uploads per second. That number eliminates every write-scaling mechanism in my vocabulary. No sharded write path, no ingestion queue for throughput reasons.

Reads: assume 20 avatar views per active user per day and 10 million daily active users, so 200 million reads per day. Divide by roughly 10 to the 5 seconds per day: about 2,000 reads per second. Both assumptions are labelled as assumptions and stated as such in the round.

That read number matters because 2,000 requests per second of small immutable public objects is exactly what a content delivery network exists for, and it eliminates the design where the application servers serve image bytes.

Block 3. PURPOSE: draw the smallest correct thing and say that it is deliberate.

Upload path: client to service, service writes the original to a blob store, enqueues nothing, and generates the four sizes inline because 2 uploads per second times four resizes is trivial work. Read path: client to content delivery network, cache miss to blob store origin, with the object key derived from user id and a version counter.

I say: "That is the whole design at these numbers. Adding a resize queue or a metadata cache now would be adding a box I cannot justify with a measurement."

The error most readers make here is adding a transcode queue reflexively because media plus queue is a memorised pairing. At two uploads per second, the queue's only real function is retry, and a retry column in the database does that more cheaply.

Block 4. PURPOSE: break my own design on the clock, before the interviewer does.

At minute 20 I break it: the version counter in the object key is what makes cache invalidation free, so what happens if a user changes their avatar twice in a second and the two resize jobs race? I add a compare-and-set on the version and let the later version win, and I note that the loser's objects are garbage that a lifecycle rule deletes after 30 days.

Then I spend minutes 25 to 32 on the one deep dive that this problem actually makes interesting: cache key design and the moderation hold, where an avatar must be pulled from the edge within seconds of a takedown decision. That is a purge path, and it is the only place where the cheap design genuinely strains.

2e. Fade the scaffold

Faded example 1. Same clock, 60 minute round. Prompt: "Design a service that lets employees search a company document store." You get blocks 1 to 3, then produce block 4.

• Block 1: I buy constraints. I learn there are 200,000 employees, 40 million documents, an average document of 40 kilobytes, permissions are per document, and freshness of five minutes is acceptable.
• Block 2: numbers. 40 million times 40 kilobytes is about 1.6 terabytes of raw text, which fits on one machine's disk and rules out a distributed index for size reasons alone. Assume 5 searches per employee per working day, so 1 million searches per day, about 12 per second at the daily average and perhaps 60 per second at peak if peak is 5 times average.
• Block 3: V0 is one search index, one crawler that pulls document changes every five minutes, and permission filtering applied at query time.
• Block 4: you break it and choose the deep dive.

Faded example 2. 45 minute round. Prompt: "Design a service that emails a receipt after every purchase." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: I learn there are 2 million purchases per day, receipts must never be sent twice, a delay of up to two minutes is acceptable, and a third-party email provider is used.
• Block 2: 2 million per day is about 20 per second on average. Assume a peak of 5 times average, so 100 per second. That eliminates any throughput-driven architecture and puts the entire difficulty on duplicate suppression and provider failure.

2f. Check yourself

Q1. You are at minute 24 of a 45 minute round and you have not drawn anything yet, because clarification kept opening new questions. What do you do?

Q2. Why does the budget put more staff-level minutes into framing than into the deep dive, when depth is what staff engineers are known for?

2g. When not to use this

The clock is scaffolding. Scaffolding that stays up forever becomes the building's problem.

Three situations where following the clock is the wrong move.

Module 3. Requirements that change the design

Time: 40 to 50 minutes reading, 20 to 25 minutes practice.

3a. Predict first

A candidate produced this requirements list in the first five minutes of a round on a social product.

The fork test

A clarifying question is worth asking only if two plausible answers produce two different drawings. Everything else is a checklist recital, and it consumes minutes that the design phase needs.

Ask this before every question: if the answer is A I draw X, if the answer is B I draw Y. If you cannot fill both halves, do not ask.

The question classes that actually fork designs. Five to seven of these, chosen for the prompt, is a full clarification phase.

Turning an adjective into a mechanism. Prompts arrive as adjectives. Your job is to convert.

Tag every requirement. Write GIVEN if the interviewer said it, ASSUMED if you invented it, DERIVED if it follows from the other two. This single habit prevents the most common credibility failure, which is quietly promoting your own guess into a fact and then defending it.

One question, drawn as a fork

       how stale may a follower feed be

       +---------------------------+
       |                           |
   under 1 second              5 seconds ok
       |                           |
       v                           v
+----------------+        +------------------+
| read time      |        | write time       |
| gather from    |        | fan out into per |
| followed users |        | user feed store  |
+----------------+        +------------------+
       |                           |
       v                           v
+----------------+        +------------------+
| cost moves to  |        | cost moves to    |
| every read     |        | every write and  |
| fan out is n   |        | storage grows    |
+----------------+        +------------------+

Figure one clarifying question forking a feed into two architectures

Worked example: turning a vague prompt into forks

The prompt: "Design a notification system."

Block 1. PURPOSE: find out what the system is actually for, because the phrase covers three different products.

I ask one question first: "Is this in-app notifications, push and email delivery to devices, or the alerting layer that decides what is worth notifying about?" The answer is push and email delivery for a consumer app.

That one question just removed two thirds of the possible designs. Notice it forked before any number was discussed.

The error most readers make here is starting with capacity questions. Volume clarifies size, but product scope clarifies which system you are drawing, and drawing the wrong system at the right scale scores zero.

Block 2. PURPOSE: ask the five questions that fork the delivery design, and state each fork out loud.

Block 3. PURPOSE: convert the fan-out answer into the one number that decides the topology.

A 20 million user campaign against a steady state I assume to be 50 million notifications per day is the whole story. 50 million per day divided by roughly 100,000 seconds per day is about 500 per second average. One campaign is 20 million messages. If the product wants it delivered in 10 minutes, that is 20 million divided by 600 seconds, about 33,000 per second, which is 66 times the average rate.

That ratio, not the absolute volume, is what forces a decoupled queue plus a rate-shaped sender. I say the ratio out loud, because the ratio is the argument.

Block 4. PURPOSE: state what I am deliberately leaving out, and why.

I say: "I am not designing the alerting or triggering logic, I am not designing the mobile client's display, and I am treating the push provider as an external dependency with its own failure modes. If you want any of those instead, say so now and I will trade one of them for the delivery path."

The error most readers make here is leaving scope implicit and then being surprised when the interviewer asks about the part they silently dropped. Stating the cut converts an omission into a decision.

3e. Fade the scaffold

Faded example 1. Prompt: "Design a system to store and query application logs." Blocks 1 to 3 given, produce block 4.

• Block 1: I ask whether this is for debugging by engineers, for compliance retention, or for alerting. The answer is engineer debugging with a 30 day window.
• Block 2: the forks. Query shape is text search and time range, not point lookup, which rules out a plain key-value store. Writes hugely outnumber reads. Loss of a small fraction of logs during an incident is acceptable, which is a very large permission.
• Block 3: numbers. Assume 2,000 hosts each emitting 100 lines per second at 200 bytes, so 200,000 lines per second and 40 megabytes per second, about 3.5 terabytes per day and roughly 100 terabytes for 30 days after assuming compression halves it.

Faded example 2. Prompt: "Design a coupon redemption service." Blocks 1 and 2 given. Produce blocks 3 and 4.

• Block 1: it is redemption at checkout, not coupon creation or targeting.
• Block 2: the forks I asked about are whether a coupon can be used twice (no), whether coupons are per user or global (both exist), and what happens if the service is down at checkout (checkout must proceed without the discount).

3f. Check yourself

Q1. An interviewer says "assume standard requirements". What do you do?

Q2. Give one clarifying question that fails the fork test, and rewrite it so it passes.

3g. When not to use this

Module 4. Estimation as elimination

Time: 40 to 50 minutes reading, 20 to 25 minutes practice.

4a. Predict first

Here is an estimation block of the kind that appears in most study material.

daily active users            100 million
requests per user per day     20
requests per day              2 billion
requests per second           23148
peak requests per second      46296
storage per record            1 kilobyte
daily storage                 2 terabytes
five year storage             3650 terabytes
bandwidth                     23 megabytes per second

Figure a typical estimation block with nine computed numbers

The arithmetic, done the way it survives a whiteboard

Work in powers of ten, and round aggressively. Precision you did not have in the input cannot appear in the output.

The last row is the single most useful fact in this module. One million events per day is ten per second. Estimation under pressure gets much easier once that one conversion is automatic.

The three quantities worth computing, and nothing else.

Peak factors are assumptions and must be labelled. A consumer product with one time zone might peak at 3 to 5 times its daily mean; a global product with flat traffic might peak at 1.5. State which you assume and why. If the design does not change between a factor of 2 and a factor of 5, say so and stop refining.

Hardware reference points are also assumptions. Never quote a benchmark you cannot source. Instead, state the shape of the machine you are assuming and invite correction.

Say the sentence: "I am assuming a machine with roughly 128 gigabytes of memory. If your machines are much larger, this next conclusion changes." That is honest, it is checkable, and it protects you from a follow-up you cannot win.

The graceful skip. Estimation is not always worth its minutes.

"I can size this if it will change what we build. My instinct is that both designs on the table work at any volume we would plausibly assume, so unless you want the numbers I will spend the time on the failure modes instead."

Interviewers who wanted estimation will ask for it. Interviewers who did not just got three minutes back, and you signalled that you know why the exercise exists.

The elimination funnel

+------------------------------------------+
| stated inputs   users rate size retention |
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| three quantities   qps   bytes   hot set  |
+------------------------------------------+
                    |
      +-------------+-------------+
      |                           |
      v                           v
+------------------+     +--------------------+
| option survives  |     | option eliminated  |
| keep it on board |     | say it out loud    |
+------------------+     +--------------------+
                    |
                    v
+------------------------------------------+
| if nothing was eliminated delete the math |
+------------------------------------------+

Figure estimation as a filter on options not as arithmetic

Worked example: sizing a link shortener so that it removes boxes

GIVEN by the interviewer: 500 million new links created per month, read to write ratio about 100 to 1, links must work for at least 5 years.

Block 1. PURPOSE: get the write rate, and use it to delete machinery.

500 million per month divided by 2.5 million seconds per month is 200 writes per second.

Eliminated by that number: a sharded write path, an ingestion queue justified by throughput, and any discussion of write batching. 200 small writes per second is inside what one primary database node handles, so if I later shard, the reason will be storage size, not write rate. I say exactly that sentence, because it pre-empts the follow-up.

The error most readers make here is computing the write rate and then adding a queue anyway, because writes plus scale is a memorised pairing. The number you just computed forbids it.

Block 2. PURPOSE: get the storage curve, and use it to choose the storage class.

Bytes per record: 7 bytes of short code, an assumed 100 bytes of average destination URL, 8 bytes of owner id, 8 bytes of creation timestamp, 8 bytes of expiry. That is 131 bytes of payload. I assume total on-disk cost is about 500 bytes per record once the primary key index, a secondary index on owner, and per-row overhead are counted, roughly four times payload.

500 million records per month times 500 bytes is 250 gigabytes per month, so 3 terabytes per year and 15 terabytes over the 5 year requirement.

Eliminated by that number: keeping the whole mapping in memory. At an assumed 128 gigabytes of usable memory per cache node, 15 terabytes is over 100 nodes of pure cache, which is an absurd bill for a redirect. Also eliminated: a single node that holds everything comfortably, not because 15 terabytes cannot fit on one machine, but because restoring 15 terabytes from a backup at an assumed 200 megabytes per second takes about 21 hours, and that is the number I would actually cite in the round.

Block 3. PURPOSE: get the read rate, and decide whether a cache is justified rather than assumed.

100 to 1 against 200 writes per second is 20,000 reads per second.

Now the elimination that matters. Reads are point lookups on a 15 terabyte dataset, so unless the working set fits in memory, each read is a random disk read. 20,000 random reads per second against a volume I assumed provides low thousands of operations per second means the disk is the wall. That justifies a cache, with a measurement rather than a reflex.

Block 4. PURPOSE: size the cache, and use the size to eliminate a whole tier.

Assume link traffic is heavily skewed and that 80 percent of reads hit links created in the last 30 days, and within those, 10 percent of the links serve most of the traffic. Links in 30 days: 500 million. Ten percent is 50 million.

A cached entry is not a stored row, and mixing the two is the arithmetic error to avoid here. The row was 500 bytes because it carried five fields plus two indexes plus per-row overhead. A cache entry carries only what the redirect needs: the 7 byte code as key and the 100 byte URL as value, which is 107 bytes, plus an assumed 43 bytes for the hash slot, the pointer and the expiry stamp. Call it 150 bytes. So 50 million entries times 150 bytes is 7.5 gigabytes.

Eliminated by that number: a distributed cache tier. 7.5 gigabytes fits in memory on a single cache node with room to spare, so I start with a small replicated cache, not a sharded one. I state the skew as an assumption and name the metric that would falsify it: cache hit ratio below about 80 percent in production means the skew assumption was wrong and the tier gets revisited.

Block 5. PURPOSE: check bandwidth, find that it eliminates nothing, and delete it.

20,000 reads per second times an assumed 500 byte redirect response including headers is 10 megabytes per second, about 80 megabits per second. That rules nothing out at all: it is inside a single machine's network capacity.

So I say one sentence and move on: "Bandwidth is not a constraint here, so I will not spend time on it." Deleting a number out loud is worth more than computing three more.

The error most readers make here is presenting every computed number regardless of consequence, which turns the estimation phase into a recital and burns four of the thirty eight working minutes.

4e. Fade the scaffold

Faded example 1. A photo product. GIVEN: 10 million photos uploaded per day, each stored at 4 sizes, average total 2 megabytes per photo across all sizes, kept forever. Reads are 30 times writes. Produce the last block.

• Block 1, write rate: 10 million per day divided by 100,000 is 100 uploads per second. That eliminates a sharded upload path and any throughput-driven ingestion queue.
• Block 2, storage: 10 million times 2 megabytes is 20 terabytes per day, so about 7 petabytes per year. That eliminates any block storage or database-resident design and forces an object store.
• Block 3, read rate: 100 times 30 is 3,000 reads per second of large immutable objects.
• Block 4: you decide what the read number eliminates, and size the edge cache.

Faded example 2. A chat product. GIVEN: 500 million messages per day, average 200 bytes of text, history retained forever, a user opens the app and reads the last 50 messages of each of 5 conversations. Produce the last two blocks.

• Block 1, write rate: 500 million divided by 100,000 is 5,000 messages per second. Assume a peak factor of 3 for a single dominant time zone, so 15,000 per second at peak. That still does not eliminate a single primary for raw insert rate, but it is close enough that I would not build on one.
• Block 2, storage: 500 million times an assumed 500 bytes stored per message once identifiers, timestamps and indexes are counted, is 250 gigabytes per day, about 90 terabytes per year.

4f. Check yourself

Q1. You compute 4,000 requests per second for a service. State two different designs this number eliminates, and one it does not.

Q2. An interviewer asks for capacity estimates on a design where both candidate options are a single service with a relational database, differing only in schema. What do you say?

4g. When not to use this

Module 5. Non-functional characteristics as mechanisms

Time: 55 to 70 minutes reading, 25 to 35 minutes practice.

5a. Predict first

A checkout service is promised at 99.99 percent availability. Its request path calls six internal services in sequence, each independently rated at 99.95 percent.

Availability, as arithmetic you can do on a whiteboard

The conversion table. A 30 day month is 43,200 minutes. Every figure below is that number times the unavailability fraction.

Two things fall out immediately. First, 99.99 percent means you cannot afford a single unplanned human intervention in a month, since paging a human and having them act takes longer than 4.32 minutes. That is why four nines is an automation claim, not a redundancy claim. Second, the difference between three and four nines is a factor of ten in cost and effort, so it must be justified by a stated business consequence.

Series and parallel.

Independence is the assumption that fails. Replicas share a control plane, a configuration push, a deployment pipeline, a certificate, a DNS zone and often a region. A shared dependency turns a parallel calculation back into a serial one. Treat the parallel formula as an upper bound and say so, because the gap between the formula and reality is the whole content of this section.

The design move is dependency reduction. For each dependency on the critical path, ask: can it be removed, cached, defaulted, or made asynchronous?

Durability is a different question from availability

A system can be highly durable and frequently unavailable, or highly available and lose data. Say which one the prompt cares about. Do not quote a vendor's durability figure you cannot source; instead state the mechanism: how many copies, in how many independent failure domains, and how you would detect that a copy has silently rotted.

The consistency spectrum, with the mechanism that buys each level

Strong versus eventual is not a spectrum, it is a two item list, and treating it as a spectrum is what makes the follow-up unanswerable. Here is the usable version, ordered from strongest.

The interview move. Say which level each operation needs, not which level the system has. In a payments product the balance decrement is linearizable and the transaction list is read your writes. Those are different mechanisms in the same service, and saying so is worth more than any single choice.

Latency, percentiles and fan-out amplification

Averages hide the failure. Use percentiles, and know what happens when a request fans out.

If one backend serves a request in under 100 milliseconds 99 percent of the time, and a request must contact n independent backends and wait for all of them, the chance that every one is fast is 0.99 to the power n.

At a fan-out of 100, the ninety ninth percentile of one backend has become the median experience of the user. This effect and the mitigations for it, including sending a duplicate request and cancelling the loser, are described in Dean and Barroso, "The Tail at Scale", Communications of the ACM, February 2013 (https://cacm.acm.org/research/the-tail-at-scale/).

Design consequences that follow directly:

• Reduce fan-out before optimising any single backend.
• Set per-dependency timeouts below the total budget, and make the budget explicit in the design.
• Prefer returning a partial answer over waiting for the slowest shard, when the product allows it.

Failure models, named

You cannot design against a failure you have not named. Use these five and say which ones you are handling.

Gray failure is the easiest class to leave out of a design and the one that causes the worst incidents, because every automated system believes the node is healthy while every user experiences it as broken.

The availability chain, drawn

+--------+   +--------+   +--------+   +--------+
| client |-->|  edge  |-->|checkout|-->|  auth  |
+--------+   +--------+   +--------+   +--------+
                              |
              +---------------+---------------+
              |               |               |
              v               v               v
        +----------+    +----------+    +----------+
        | pricing  |    | catalog  |    |  fraud   |
        +----------+    +----------+    +----------+
              |
              v
        +--------------+
        | card gateway |
        +--------------+

critical path is every box a failure of which fails checkout
each box on that path multiplies into the achievable number

Figure the checkout critical path where availability is multiplied

Worked example: turning "highly available and fast" into mechanisms

The prompt: a checkout service. The interviewer says it must be "highly available and fast".

Block 1. PURPOSE: convert two adjectives into numbers with consequences attached.

I ask what a minute of checkout downtime costs, and get "a lot, but we have never measured it". So I propose a target instead: 99.95 percent monthly, which is 21.6 minutes. I say why I did not propose 99.99: four nines allows 4.3 minutes a month, which is less than the time to page a human, so it is a commitment to fully automated recovery of every failure mode including the third-party card gateway, which we do not control.

For latency I propose p99 under 800 milliseconds for the checkout call, and I state the split I intend to defend: 200 milliseconds of my own work and 600 milliseconds of dependency budget.

The error most readers make here is accepting "highly available" and writing 99.99 percent because it sounds strong. A number you cannot mechanically achieve is worse than a lower number you can defend.

Block 2. PURPOSE: compute the achievable number from the dependency chain, before designing anything.

Assumed ratings, stated as assumptions: auth 99.99, pricing 99.95, catalog 99.9, fraud 99.9, card gateway 99.9. Serial unavailability is 0.01 plus 0.05 plus 0.1 plus 0.1 plus 0.1, which is 0.36 percent, so about 99.64 percent, or roughly 155 minutes per month.

The target was 21.6 minutes. I am off by a factor of 7 before I have drawn a single box, and I say this out loud, because it reframes the entire round: this is not a scaling problem, it is a dependency problem.

Block 3. PURPOSE: shorten the critical path rather than add nines to its members.

New serial unavailability: 0.01 percent from auth, plus the gateway leg. If both gateways are at 99.9 percent and their failures are independent, that leg contributes 0.001 times 0.001, which is negligible. Total is about 0.01 percent, so roughly 99.99 percent on paper.

I immediately discount it. Independence between two payment providers is plausible but not guaranteed, since both may depend on the same card networks, and my own service and its data store are not in the estimate at all. I claim 99.95 percent with a mechanism behind it, not 99.99 percent from arithmetic.

The error most readers make here is treating the parallel formula as a fact. Say the independence assumption out loud every single time you use it.

Block 4. PURPOSE: state consistency per operation and pick the failure direction.

Saying "linearizable here, eventual there, and here is why" is the difference between a recited consistency definition and a design.

5e. Fade the scaffold

Faded example 1. A media playback service. Target proposed at 99.9 percent monthly. Critical path: edge, playback session service, entitlement check, licence server, content delivery network. Assumed ratings: edge 99.99, session 99.95, entitlement 99.95, licence 99.9, delivery network 99.99. Produce the last block.

• Block 1: numbers. 0.01 plus 0.05 plus 0.05 plus 0.1 plus 0.01 is 0.22 percent, so about 99.78 percent, which is 95 minutes a month against a 43 minute budget.
• Block 2: the licence server dominates at 0.1 percent, contributing nearly half the total.
• Block 3: you shorten the path and state the consistency and failure direction per operation.

Faded example 2. A ride dispatch service, target 99.95 percent. Critical path: mobile edge, dispatch, driver location store, map routing provider, pricing, payments pre-authorisation. Produce the last two blocks.

• Block 1: assumed ratings are edge 99.99, dispatch 99.95, location store 99.95, routing provider 99.9, pricing 99.95, pre-authorisation 99.9. That sums to 0.01 plus 0.05 plus 0.05 plus 0.1 plus 0.05 plus 0.1, which is 0.36 percent, about 99.64 percent, or 155 minutes a month against a 21.6 minute budget.

5f. Check yourself

Q1. A candidate says "we will add a replica, so availability goes from 99 to 99.99 percent". What is the strongest objection?

Q2. Which is worse for a user: a service that is unavailable for 40 minutes once a month, or one that is unavailable for 80 seconds every day? Both compute to about 99.9 percent.

5g. When not to use this

Availability targets above three nines.

Multi-zone and multi-region redundancy.

Linearizable operations.

Module 6. Building blocks I, the request path

Time: 65 to 80 minutes reading, 30 to 45 minutes practice.

6a. Predict first

+--------+   +-----+   +---------+   +--------+   +--------+
| client |-->| DNS |-->| region  |-->|  L7    |-->| service|
|        |   | TTL |   | entry   |   | proxy  |   | pool   |
|        |   | 300 |   | address |   |        |   |        |
+--------+   +-----+   +---------+   +--------+   +--------+

health check runs every 10 seconds and needs 3 failures
DNS record is withdrawn when the region is marked unhealthy

Figure a DNS based failover path with a five minute record lifetime

The path, component by component

Every request crosses the same five layers. Each layer has one job, one characteristic failure, and one threshold below which it should not exist.

Name resolution and global steering. Two mechanisms, often confused.

A software load balancer designed to keep connections mapped to the same backend while the backend set changes is described in Eisenbud and others, "Maglev: A Fast and Reliable Software Network Load Balancer", USENIX NSDI 2016 (https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud). The reason that paper matters for interviews is the problem it names: when the set of servers changes, a naive hash reshuffles every connection, and long lived connections die.

Layer 4 versus layer 7 balancing.

Balancing algorithms, compared on the property that actually matters.

Consistent hashing distributes keys with minimal movement when the backend set changes, but it says nothing about load. Adding a per-backend capacity limit that forces overflow to the next backend keeps both properties, an approach set out in Mirrokni, Thorup and Zadimoghaddam, "Consistent Hashing with Bounded Loads", 2016 (https://arxiv.org/abs/1608.01350).

Health checks, and the way they cause outages.

The mandatory mitigation for deep checks is a fail-open rule: if more than an assumed 50 percent of backends are marked unhealthy simultaneously, ignore the checks and keep serving, because the far more likely explanation is that the check itself is wrong. Google's SRE material discusses the general shape of this problem under overload and cascading failure (https://sre.google/sre-book/handling-overload/, 2016).

Gateway. One place to apply authentication, rate limiting, request identifiers, routing and payload limits. Its real risk is not performance, it is ownership: gateways accumulate per-endpoint special cases until nobody can change them. State the rule out loud in the round: only policy that is genuinely common to all services lives here.

Edge caching: push versus pull.

Cache offload is arithmetic: origin requests equal total requests times one minus the hit ratio. At 20,000 requests per second and a 95 percent hit ratio, the origin sees 1,000 per second. Improving the ratio to 99 percent takes the origin to 200 per second, a five times reduction from a four point change. That non-linearity is why cache key design is worth interview minutes.

Cache key design, the three mistakes.

• Including a value that varies per user, such as a session identifier, which reduces the hit ratio to near zero.
• Excluding a value that changes the response, such as an accept-language header, which serves the wrong content.
• Using a mutable path with a short lifetime instead of an immutable versioned path, which forces you to build a purge system you did not need.

Rate limiting algorithms, compared on memory, burst and coordination. These numbers are derived from stated inputs, not measured.

Assume a limit of 1,000 requests per minute per key and 5 million active keys.

The sliding log row is the whole argument. Exactness costs three orders of magnitude more memory than the approximation, which is why almost nobody runs it above trivial limits. Stripe published a description of running token bucket and concurrency limiting in a shared cache in production (https://stripe.com/blog/rate-limiters, 2017).

Coordination cost is the second axis, and the one usually left out.

The request path, drawn

+--------+    +-----------+    +-------------+
| client |--->| name and  |--->| edge cache  |
|        |    | steering  |    | and TLS     |
+--------+    +-----------+    +-------------+
                                     |
                                 miss only
                                     v
                               +-------------+
                               |  gateway    |
                               | auth limit  |
                               +-------------+
                                     |
                                     v
                               +-------------+
                               | L7 balancer |
                               +-------------+
                               |      |      |
                               v      v      v
                          +------+ +------+ +------+
                          | svc  | | svc  | | svc  |
                          +------+ +------+ +------+

Figure the five layers a request crosses before reaching a service

Worked example: choosing a rate limiter with numbers, not taste

GIVEN: a public interface serving 200,000 requests per second at peak, 5 million active keys, 50 gateway nodes, a published limit of 1,000 requests per minute per key with a permitted burst of 50, and a p99 latency budget of 800 milliseconds end to end.

Block 1. PURPOSE: eliminate algorithms on memory before discussing anything else.

From the table above: the sliding log needs up to 8 kilobytes per key at this limit, so 5 million keys is up to 40 gigabytes of state. That exceeds an assumed 128 gigabyte cache node once replication and overhead are counted, and it buys exactness nobody asked for.

Token bucket needs 16 bytes of state plus key overhead. Assume 64 bytes of total overhead per entry in a real cache, so 80 bytes per key, giving 400 megabytes for 5 million keys. That is a rounding error on one node.

I choose token bucket, and I say why in one sentence: the published contract mentions a burst of 50, and token bucket is the only algorithm in the table that expresses a burst allowance as a first-class parameter rather than as an accident of window boundaries.

The error most readers make here is choosing the sliding log because it is described as accurate. Accuracy is not the requirement. The requirement is a published contract with a burst allowance, and 40 gigabytes is the price of an accuracy nobody can observe.

Block 2. PURPOSE: decide where the state lives, using the request rate rather than a preference.

Central placement means the limiter store handles 200,000 operations per second, one per request. Assume a single cache node comfortably serves around 100,000 simple operations per second, an assumption I would verify before building anything on it. Then a central design needs at least 2 shards, and I would run 4 for headroom and failure tolerance. Sharding is trivial here because the natural shard key is the limiter key itself, and there are no cross-key operations.

Local placement means each of the 50 nodes enforces 1,000 divided by 50, which is 20 requests per minute. That is only correct if each key's traffic is spread evenly across all 50 nodes. It is not: connections are long lived, and a single client with one connection hits one node and gets 20 requests per minute against a published 1,000. That is a contract violation, not an approximation, so local-only is eliminated.

Block 3. PURPOSE: check whether the network hop I just added is affordable.

Assume a same-zone round trip to the cache of 1 millisecond at the median and 3 milliseconds at the ninety ninth percentile, which is a figure I would measure rather than trust. Against an 800 millisecond p99 budget, 3 milliseconds is 0.4 percent. The hop is affordable, and I say the percentage rather than the milliseconds, because the percentage is the argument.

The real risk is not latency, it is coupling: if the limiter store is unavailable, does the interface stop? I choose fail open with a local fallback bucket, because rejecting all traffic to protect a limit is a worse outcome than briefly allowing more traffic than the contract. I state the cap on that decision: fail open only while fewer than an assumed 5 percent of requests are affected, and alert immediately.

Block 4. PURPOSE: reduce the central load by an order of magnitude if the numbers demand it.

If the interviewer raises traffic to 2 million requests per second, the central design needs 20 or more shards, and that is enough operational weight to reconsider. The lease design fixes it: each gateway node requests a block of 100 tokens for a key and spends them locally.

Central operations drop from one per request to one per 100 requests, so 2 million per second becomes 20,000 per second, which is a single shard again.

The price is bounded overshoot: in the worst case every one of the 50 nodes holds an unspent lease of 100 tokens, so up to 5,000 extra requests could be admitted for a key before the leases expire. Against a limit of 1,000 per minute, that is a real overshoot, so I shrink the lease to 10 tokens, which caps overshoot at 500 and still cuts central load by ten times.

The error most readers make here is presenting leases as strictly better. They trade exactness for load, and the trade must be quantified: overshoot equals node count times lease size, and that formula is the sentence to say out loud.

6e. Fade the scaffold

Faded example 1. Decide whether to put a content delivery network in front of an interface. GIVEN: 8,000 requests per second, responses average 40 kilobytes, 70 percent of responses are identical for all users and change at most daily, the rest are per user. Produce the last block.

• Block 1, bytes: 8,000 times 40 kilobytes is 320 megabytes per second, about 2.6 gigabits per second. That is more than a single machine's assumed network capacity, so serving all of it from origin machines means buying machines for bytes rather than for work.
• Block 2, cacheable share: 70 percent of requests are shareable, so a well designed cache key could take origin traffic to 2,400 requests per second and origin bytes to about 0.8 gigabits per second.
• Block 3: you decide, and state what would make you not do it.

Faded example 2. Choose a balancing algorithm. GIVEN: a service with an in-process cache where a hit costs 1 millisecond and a miss costs 40 milliseconds, 24 instances, and a key space where the top 20 keys account for an assumed 30 percent of traffic. Produce the last two blocks.

• Block 1: round robin gives every instance a full copy of the working set, so cache memory is multiplied by 24 and the hit ratio is set by per-instance memory rather than by total memory.

6f. Check yourself

Q1. Your deep health check calls the database. The database has a brief incident. What does the balancer do, and what should it have done?

Q2. Why does putting a rate limiter in the gateway not protect a downstream service from overload?

6g. When not to use this

Content delivery network.

Dedicated rate limiting service.

API gateway as a separate tier.

Global steering across regions.

Module 7. Building blocks II, storage

Time: 70 to 90 minutes reading, 35 to 50 minutes practice.

7a. Predict first

table  orders
  order id        primary key
  customer id     index 1
  status          index 2
  created at      index 3
  merchant id     index 4

write rate 2000 inserts per second
each order row is about 300 bytes

Figure a table with four secondary indexes and its insert rate

Replication, three topologies and the arithmetic of each

Synchronous, asynchronous and the middle.

The third row is the default worth defending in a round. It buys a recovery point objective of zero for single node failure at the cost of one round trip, and it avoids the availability trap where any slow replica blocks every write.

Quorum arithmetic. With N replicas, W nodes acknowledging a write and R nodes consulted on a read, overlapping quorums require W plus R to be greater than N.

The caveat that follows immediately: overlap guarantees that a read touches a node that saw the write, not that the read returns the newest value, unless the client resolves versions. Concurrent writes still need a conflict rule. The design that popularised this configuration space, along with virtual nodes and version vectors, is DeCandia and others, "Dynamo: Amazon's Highly Available Key-value Store", SOSP 2007 (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf).

Replication lag, as a number you can compute. If a replica falls behind during an incident, catch-up time is backlog divided by the surplus apply rate.

Worked: writes arrive at 2,000 per second, a replica can apply 5,000 per second, and it was disconnected for 10 minutes. Backlog is 2,000 times 600, which is 1.2 million writes. Surplus is 5,000 minus 2,000, which is 3,000 per second. Catch-up is 1.2 million divided by 3,000, which is 400 seconds.

If instead the apply rate were 2,500 per second, catch-up would be 2,400 seconds, six times longer for a replica that is only twice as slow. That non-linearity is why replica headroom matters more than replica count.

Partitioning, and the choice that is hard to reverse

Choosing the partition key is the highest leverage decision in the storage module. State the rule: the partition key should be the value present in the overwhelming majority of your queries. If your dominant query is "the last 50 messages in conversation X", the partition key is the conversation, and the sort key is the sequence number, so the answer is one contiguous read.

Secondary indexes across partitions, the two shapes.

The interview sentence: a global secondary index converts every insert into a multi-partition write, so either you accept that the index is eventually consistent with the row, or you buy a transaction. Most systems choose the former, and saying which one you chose is the whole point.

Rebalancing. Never partition by hash modulo the node count. Adding one node to N changes the destination of almost every key. Two workable options:

Hot keys. Partitioning spreads keys, not traffic. One key can exceed one partition's capacity by itself.

Detection first: you cannot fix a hot key you cannot see, so per-key traffic sampling or a top-k sketch belongs in the design, not in the postmortem.

Key-value internals, and the amplification that decides the choice

Two families. Define both before comparing.

• Page-oriented tree. Data lives in fixed-size pages arranged as a balanced tree. Updates modify pages in place, protected by a write-ahead log.
• Log-structured merge tree. Writes go to an in-memory table and a sequential log. When the memory table fills, it is written out as a sorted immutable file. Background compaction merges files into larger ones. This structure is described in Chang and others, "Bigtable: A Distributed Storage System for Structured Data", USENIX OSDI 2006 (https://www.usenix.org/legacy/event/osdi06/tech/chang.html).

Per-file summaries are what make the log-structured read path viable. A probabilistic filter answers "this file definitely does not contain this key" using an assumed 10 bits per key for roughly a 1 percent false positive rate. For 1 billion keys that is 10 billion bits, about 1.25 gigabytes of memory. That number is worth knowing, because it tells you when the filters stop fitting in memory, and when they stop fitting, point lookups start touching every level.

The honest interview answer. At the point rates most prompts describe, both families work. The deciding factors are the read pattern, whether the workload is update-heavy or append-heavy, and what your team can operate under pressure. Say that instead of reciting a preference, then name the one number that would change your mind: sustained write throughput per node beyond what random page updates can absorb.

Blob storage, and why the database is the wrong place

The failure mode that made this a research topic is small-file metadata cost: with billions of small objects, the metadata lookups needed to find a file can dominate the read itself. The response of packing many small objects into large append-only volumes with a compact in-memory index is described in Beaver and others, "Finding a Needle in Haystack: Facebook's Photo Storage", USENIX OSDI 2010 (https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf).

Partitions and replicas, drawn

        key space split into partitions

 +--------+  +--------+  +--------+  +--------+
 |  p 0   |  |  p 1   |  |  p 2   |  |  p 3   |
 +--------+  +--------+  +--------+  +--------+

        each partition has one leader and two followers

 +---------------+  +---------------+  +---------------+
 |   node A      |  |   node B      |  |   node C      |
 | p0 leader     |  | p1 leader     |  | p2 leader     |
 | p1 follower   |  | p2 follower   |  | p3 follower   |
 | p3 follower   |  | p0 follower   |  | p1 follower   |
 +---------------+  +---------------+  +---------------+

 losing node A promotes p0 on the node holding its follower
 adding node D moves whole partitions not individual keys

Figure partitions spread across nodes with leaders and followers

Worked example: choosing storage for the link shortener

Carried forward from module 4: 200 writes per second, 20,000 reads per second, 15 terabytes over 5 years, point lookups by short code, a 7.5 gigabyte hot set, and records that are written once and essentially never updated.

Block 1. PURPOSE: let the access pattern pick the shape before any product name is mentioned.

Every hot-path read is a point lookup by short code. There are no range scans, no joins and no multi-row transactions on the hot path. That shape is a key-value lookup, so the primary key is the code and the partition key is the code, hashed.

I explicitly refuse to add a secondary index on owner to the hot store. The owner-facing dashboard is a different access pattern with a different latency budget, so it gets its own read model fed asynchronously. Adding a global secondary index would make every creation a two-partition write to serve a query that runs a few times per user per month.

The error most readers make here is designing one store that serves every query. Two access patterns with different latency budgets and different volumes are two stores, and saying so is a senior move, not an extravagance.

Block 2. PURPOSE: size the nodes using the recovery window, not the capacity sheet.

15 terabytes fits on a small number of machines, so raw capacity does not force a node count. The operational constraint does. Assume restore runs at 200 megabytes per second, which is a figure I would verify. One hour of restore time is 720 gigabytes. If I want a node restorable in about 3 hours, nodes hold about 2 terabytes, so 15 terabytes needs 8 partitions of primary data.

With three copies of each partition, that is 45 terabytes of raw storage. I state the recovery time objective the choice implies: losing one node costs a promotion in seconds and a rebuild measured in hours, and the rebuild is the number that matters because a second failure during the rebuild is the scenario that loses data.

Block 3. PURPOSE: choose the engine family using the workload's actual shape.

The workload is append-only with no updates and no deletes until expiry. That is close to the ideal case for a log-structured engine: compaction has almost nothing to reclaim, so the usual write amplification argument barely applies, and point lookups are one file check per level with per-file filters in memory.

Filter memory check: 30 billion records over 5 years would be a problem, but this dataset is 30 billion records only if you keep every link forever at this rate. Recompute: 500 million per month times 60 months is 30 billion records, so at an assumed 10 bits per key the filters alone are 37.5 gigabytes spread over 8 nodes, about 4.7 gigabytes per node. That fits, but it is not free, and it is the number I would monitor.

Block 4. PURPOSE: state the caching layer as a consequence of the earlier numbers rather than as a habit.

From module 4, the hot set is 7.5 gigabytes. That fits on one cache node, so I use a small replicated cache in front of the store rather than a sharded tier, and I set the entry lifetime to an assumed 24 hours because records are immutable, which makes invalidation a non-problem.

The one invalidation case is deletion or a takedown, which is rare and can afford an explicit purge. I name the metric that governs the tier: cache hit ratio. Below about 80 percent the skew assumption from module 4 was wrong, and the correct response is to re-derive the hot set rather than to add nodes.

7e. Fade the scaffold

Faded example 1. Choose the storage layout for a user timeline read as "the newest 30 items for user X, then page backwards". GIVEN: 200 million users, an assumed average of 500 timeline items per user, an item reference of 40 bytes, and 40,000 timeline reads per second. Produce the last block.

• Block 1, shape: the query is a range scan inside one user, newest first, so partition by user and sort by descending sequence.
• Block 2, size: 200 million times 500 times 40 bytes is 4 terabytes of references, which is small enough that references, not full items, is clearly the right unit to store.
• Block 3: you decide the engine family and the caching, and name what breaks.

Faded example 2. Choose storage for an inventory service. GIVEN: 5 million distinct items, 3,000 stock decrements per second at peak, and a single flash-sale item that receives an assumed 40 percent of all decrements during a sale. Produce the last two blocks.

• Block 1, shape: the operation is a conditional decrement on one key, which must be linearizable per item, so the partition key is the item and every decrement must be routed to that partition's leader.

7f. Check yourself

Q1. A candidate proposes a global secondary index so a rare admin query is fast. What do you say?

Q2. Your replicas are asynchronous and a user reports that their own edit disappeared after saving. What happened, and what is the smallest fix?

7g. When not to use this

Sharding a database.

Read replicas.

Leaderless quorum store.

A separate object store for files.

Module 8. Building blocks III, movement

Time: 65 to 85 minutes reading, 30 to 45 minutes practice.

8a. Predict first

producers  20000 events per second steady

+--------------------------------------------+
|  log topic   24 partitions                 |
+--------------------------------------------+
        |            |             |
        v            v             v
   +---------+  +---------+   +---------+
   | consumer|  | consumer|   | consumer|
   | group   |  | group   |   | group   |
   +---------+  +---------+   +---------+

one consumer instance processes 500 events per second
the team plans to run 60 instances

Figure a log topic with 24 partitions and a plan to run 60 consumers

Queue or log: they are not the same thing

Choose a queue when the message is a task to be done once by someone. Choose a log when the message is a fact that several systems care about and someone will want to reprocess it. Saying which of those two the payload is, before naming any technology, is the move that separates a designed answer from a named one.

Delivery semantics, stated precisely

Where exactly-once is a lie. A broker can make delivery and offset commitment atomic inside its own system. Apache Kafka's transactional producer and consumer offsets do this, as described in Confluent's write-up of the design (https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/, 2017). What no broker can do is make an external side effect atomic with a message acknowledgement. If your handler charges a card, sends an email or calls a partner, then a crash between the side effect and the acknowledgement is indistinguishable from a crash before the side effect.

The honest formulation to say in a round: at-least-once delivery plus an idempotent effect equals exactly-once outcomes, and that is the only kind you can actually build.

Sizing a deduplication store, because "we will deduplicate" is not a design. Deduplication needs a key, a window and memory.

Worked: 500 transactional messages per second, a 24 hour window, and an assumed 32 bytes per key plus 32 bytes of entry overhead. Keys in the window are 500 times 86,400, which is 43.2 million. At 64 bytes each that is about 2.8 gigabytes, which fits on one node.

Now change the input to 20,000 messages per second: 1.73 billion keys, about 110 gigabytes, which does not. At that point you either shorten the window to one hour, which cuts it to 4.6 gigabytes, or you move deduplication into the sink by making the write itself conditional on the key.

That derivation is the answer. The window length is a design parameter you choose from a memory budget, not a detail.

Ordering, and the requirement people over-state

Global ordering across a distributed stream costs a single serialisation point, which is a throughput ceiling. Almost no product needs it.

The cost of ordering by entity is a hot partition. If one entity produces a disproportionate share of events, its partition is a single consumer's problem. The remedy is the same as the hot key remedy in the storage module, and the trade is identical: split the key and give up strict ordering for that entity, or keep ordering and accept the ceiling.

Backpressure, and the failure it prevents

Little's Law. For a stable system, the number of items in flight equals arrival rate times the average time each spends in the system. Written as L equals lambda times W.

Use it in two directions.

• Forward: at 20,000 arrivals per second and 50 milliseconds of processing, in-flight work is 20,000 times 0.05, which is 1,000 items. That is your concurrency requirement.
• Backward: if the latency budget for time spent in the queue is 2 seconds and arrivals are 20,000 per second, the maximum tolerable queue depth is 40,000. A queue longer than that is not a buffer, it is a latency violation you have not noticed yet.

The three responses to overload, and when each is right.

The response that is never right is an unbounded buffer. It converts a throughput problem into a memory exhaustion problem and delays the failure until it is unrecoverable.

Retry storms and metastable failure. When a system slows, clients retry, which raises load, which slows it further. The system can stay stuck in the degraded state even after the original trigger is removed, because retries alone now sustain the overload. This class is named and analysed in Bronson and others, "Metastable Failures in Distributed Systems", HotOS 2021 (https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11.pdf).

Three mechanisms, all cheap, all expected in a senior answer.

Dead letter handling. A message that fails repeatedly must leave the main path, or it blocks a partition forever. State three things in the design: the attempt count that triggers diversion, where diverted messages go, and who looks at them. A dead letter queue nobody reads is a silent data loss mechanism with extra steps.

Event time, watermarks and the completeness knob

Two clocks exist. Processing time is when your system saw the event. Event time is when it happened. They diverge because of mobile clients, retries, network delays and offline buffering.

A watermark is an assertion: "I believe I have now seen all events with an event time before T." Everything downstream of that assertion is a trade.

The design question is not "how do I handle late data", it is "who consumes the correction". If the downstream is a dashboard, corrections are fine. If the downstream is a billing invoice already sent to a customer, they are not, and the design must close the window with a hard cutoff and a documented policy for what happens to events arriving afterwards.

Windowing types, in one line each.

Movement, drawn

+----------+     +---------------------------+
| producer |---->| durable log               |
+----------+     | partition by entity id    |
                 +---------------------------+
                    |          |          |
                    v          v          v
              +---------+ +---------+ +---------+
              |consumer | |consumer | |consumer |
              +---------+ +---------+ +---------+
                    |          |          |
                    +----------+----------+
                               |
                     +---------+---------+
                     |                   |
                     v                   v
              +-------------+     +-------------+
              | sink with   |     | dead letter |
              | idempotent  |     | after n     |
              | write       |     | attempts    |
              +-------------+     +-------------+

Figure a log with per entity partitioning an idempotent sink and a
dead letter path

Worked example: the notification campaign, sized properly

Carried forward from module 3. GIVEN: 50 million notifications per day steady, about 500 per second; one marketing campaign of 20 million messages that the product wants delivered in 10 minutes; transactional notifications must arrive within 30 seconds; duplicates are unacceptable for transactional and tolerable for marketing.

ASSUMED, and stated as such: the push provider accepts 5,000 messages per second for this account; each send is about 40 milliseconds of mostly waiting on the network; each message is about 200 bytes on the wire.

Block 1. PURPOSE: test the stated requirement against the dependency before designing anything.

The campaign wants 20 million messages in 600 seconds, which is 33,333 per second. The provider accepts 5,000 per second. At that rate 20 million messages take 4,000 seconds, which is about 67 minutes.

So the requirement is not achievable, and no amount of internal architecture changes that. I say this immediately, because it is the single most valuable sentence available in this problem: "the 10 minute target requires roughly seven times the provider throughput we have, so before I design anything I want to know whether we can buy more provider capacity, spread across several sender accounts, or relax the target."

The error most readers make here is designing a beautifully scalable internal pipeline that ends at a dependency doing 5,000 per second. Always compute the slowest link first.

Block 2. PURPOSE: size the buffer from the mismatch, which decides the buffer's technology.

Suppose the campaign is enqueued as fast as we can generate it and drained at 5,000 per second. Peak backlog is roughly 20 million minus what drains during generation. If generation takes 600 seconds, drain removes 3 million in that time, so the peak backlog is about 17 million messages.

At 200 bytes each, that is 3.4 gigabytes. Eliminated by that number: an in-memory queue, and any broker configuration that keeps the backlog in memory. Required: a durable, disk-backed log with retention comfortably above the drain time, which here needs to cover at least 67 minutes and which I would set to hours for safety.

Block 3. PURPOSE: prove that one shared pipe violates the transactional requirement, using arithmetic rather than intuition.

If transactional and marketing messages share one first-in-first-out pipe, a transactional message enqueued at the peak sits behind 17 million marketing messages. At a drain rate of 5,000 per second, its wait is 17 million divided by 5,000, which is 3,400 seconds, about 57 minutes. The requirement was 30 seconds. The violation is a factor of over 100.

Therefore the two classes get separate topics and separate consumer groups with separate provider quota, not a priority field on one queue. I say why explicitly: a priority field inside one partitioned log does not help, because a consumer reads its partition in order and cannot skip ahead to a high priority message sitting behind a million low priority ones.

Block 4. PURPOSE: make duplicates impossible where they are unacceptable, and size the machinery that does it.

Transactional messages carry a deduplication key equal to the triggering event identifier. Before sending, the worker performs a conditional insert of that key with a 24 hour lifetime, and only sends if the insert succeeded. Sizing, from the section above: 500 per second times 86,400 seconds is 43.2 million keys, and at an assumed 64 bytes per entry that is about 2.8 gigabytes, which fits on a single node with replication.

Marketing messages get no deduplication store, because the volume would be 40 times larger for a class where duplicates are tolerated. Choosing different guarantees for different classes of the same message type, and justifying it with a memory figure, is the depth this problem rewards.

Block 5. PURPOSE: state the overload policy and the failure handling before being asked.

The error most readers make here is treating the dead letter path as a formality. State the trigger count, the destination and the human process, or it is a silent loss channel.

8e. Fade the scaffold

Faded example 1. An event pipeline counts video views per minute. GIVEN: 40,000 events per second at peak, each event 250 bytes, a consumer instance handles an assumed 2,000 events per second, and the current topic has 16 partitions. Produce the last block.

• Block 1, capacity: 16 partitions times 2,000 is 32,000 per second, against 40,000 arriving. Lag grows at 8,000 per second.
• Block 2, the backlog if it runs for 30 minutes: 8,000 times 1,800 is 14.4 million events, about 3.6 gigabytes at 250 bytes each.
• Block 3: you fix it, and state the recovery time and the risk of the fix.

Faded example 2. Ad click accounting. GIVEN: 30,000 clicks per second, each click must be billed exactly once, clients can be offline so events arrive with event times up to 12 hours old, and an invoice is generated daily. Produce the last two blocks.

• Block 1, the semantic: billing means the effect must happen once, so the design is at-least-once delivery plus an idempotent write keyed on the click identifier, not a broker feature.

8f. Check yourself

Q1. A design says "we use a broker with exactly-once delivery, so we do not need idempotency". What is wrong?

Q2. Your consumer lag is flat at 4 million events and not growing. Is the system healthy?

8g. When not to use this

A message broker at all.

A stream processing framework.

Exactly-once machinery.

Priority lanes and separate topics.

Module 9. Building blocks IV: coordination

Coordination means getting more than one machine to agree on one fact when machines die and messages arrive late. It is the most over-reached-for building block in the interview and the easiest one to describe vaguely.

9a. Pre-question

Read this design fragment and commit to an answer before continuing.

+--------------------------------------------------------------+
| Three app servers, one Postgres, one Redis.                   |
| Every 60s each server runs the same job sweep:                |
|   1. SET sweeplock <id> NX PX 30000     (Redis lock, 30s TTL) |
|   2. SELECT jobs WHERE due_at < now() AND state = 'pending'   |
|   3. call the email provider for each row                     |
|   4. UPDATE jobs SET state = 'sent'                           |
| Measured: the lock is held by exactly one server 99.9% of     |
| sweeps. Volume: 10,000 due jobs per hour.                     |
+--------------------------------------------------------------+
Caption: a lock-based job sweep across three servers, with its
measured mutual-exclusion rate and its job volume.

Question. Roughly how many duplicate-eligible emails per day does the 0.1% figure expose you to, and does adding a second Redis node reduce it?

9b. Explanation in small steps

The one question that routes everything. Before you name Raft, ask: does this decision have to survive the death of the machine that made it, and is there any repair possible afterwards if two machines make it at once?

Most interview prompts land in row two. Candidates answer as if they were all in row three.

Terms, defined before use.

Quorum arithmetic you can do at the whiteboard. With N replicas, a majority quorum is floor(N/2)+1, and the group survives N minus quorum failures.

Convert the right column to time: 0.99970 leaves 2.6 hours of expected downtime per year (0.0003 times 8,760), and 0.99999 leaves about 5 minutes. That is the argument for five nodes and the argument against seven in one table.

Raft at the depth an interview actually rewards. You do not need the paper's proofs. You need these five facts and the consequences.

Paxos differs mainly in shape: single-decree Paxos agrees on one value and Multi-Paxos adds a stable leader to skip a phase. Raft folds the leader into the protocol from the start. For an interview, the honest sentence is that Raft is easier to reason about and both give you the same guarantee, and the interesting question is not which one but whether you should be running either.

The latency floor, derived. Assume three replicas across three availability zones in one region with a 1 ms zone-to-zone round trip, and an NVMe fsync of about 0.5 ms. Both assumptions are conservative and you should measure yours.

• One commit needs the leader's fsync plus one follower's ack: about 1.5 ms.
• A chain of writes that each depend on the previous one therefore tops out near 1,000 / 1.5, roughly 650 per second.
• Batching 100 entries per commit gives about 65,000 entries per second at the same 1.5 ms latency. Throughput comes from batching. Latency does not improve.

Now move the replicas to three regions with a 70 ms round trip. One commit is about 70 ms, so a dependent chain runs at 14 per second. This single derivation kills most "just make it globally consistent" answers.

Leases and the safe action window. A lease of length L is not safe for L seconds of work. If your clock skew bound is e and the maximum delay on the renewal response is d, the holder may only act for L minus e minus d.

• L = 15 s, e = 200 ms, d = 2 s gives a 12.8 s safe window.
• Renew at L/3, so 5 s, which buys two renewal attempts before expiry.

Fencing tokens close the gap leases leave. A paused holder that wakes after expiry still believes it holds the lease. Have the lock service hand out an increasing integer with every grant, attach it to every write, and have the storage layer refuse any write whose token is below the highest it has seen.

This is the difference between "the lock is probably right" and "a wrong holder cannot corrupt anything". Martin Kleppmann's 2016 write-up on distributed locking is the clearest public treatment of the failure this prevents: how to do distributed locking (accessed August 2026).

Clocks. Three kinds, and only one is safe to order events with.

A hybrid logical clock stores physical milliseconds in the high bits and a counter in the low bits, bumping the counter when two events share a millisecond. It guarantees that if A happened before B then hlc(A) is less than hlc(B), which a raw wall clock does not. The original description is Kulkarni et al., Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases, 2014 (paper, accessed August 2026).

Skew, quantified. Suppose your platform team guarantees clocks within plus or minus 10 ms and your cluster takes 50,000 writes per second. Then 10 ms times 50,000, that is 500 writes, sit inside the ambiguity window at any moment. Any rule of the form "the later timestamp wins" silently picks a winner among those 500. If your platform team cannot state a bound at all, timestamp ordering is not a design, it is a coin flip.

Idempotency keys, the pattern that replaces most coordination.

Sizing, derived: 1,000 writes per second times 86,400 seconds is 86.4 M rows per day. A 7 day retention gives 605 M rows. At 200 bytes per row that is about 121 GB plus index, which fits comfortably on one node. If your write rate is 20x that, revisit, because 2.4 TB of hot key lookups is a different design.

Stripe's public API documentation is a good reference for the client-facing contract this produces: idempotent requests (accessed August 2026).

9c. ASCII diagram

       +------------------+        +------------------+
       |  CLIENT          |        |  LOCK SERVICE    |
       |  key = intent id |        |  grants token N  |
       |  retries reuse   |        |  N increases     |
       +---------+--------+        +---------+--------+
                 |                           |
                 v                           v
       +--------------------------------------------------+
       |  SERVICE                                          |
       |  1. insert key (unique index) -> NEW or DONE      |
       |  2. attach token N to every write                 |
       +---------------------+----------------------------+
                             |
                             v
       +--------------------------------------------------+
       |  STORAGE                                          |
       |  rejects any write with token < highest seen      |
       +--------------------------------------------------+

Caption: two independent safety mechanisms. The unique index on the
idempotency key stops a repeated intent; the fencing token stops a
stale lock holder. Neither one substitutes for the other.

9d. Worked example: charge a card exactly once

The prompt. "Users check out. We call an external payment gateway. Make sure nobody is ever charged twice, and nobody is ever silently not charged."

Inputs I am given or will assume, stated up front:

Block A. Purpose: decide whether this needs agreement at all

I start by asking whether two concurrent writers could produce something no repair can fix. They can: two captures against the same card. So this is not row one of my routing table.

Then I ask whether repair is possible. It is: a refund exists, and the gateway exposes a lookup by reference. So this is row two, not row three. That single observation removes leader election, Raft, and any distributed lock from my answer, and I say so out loud rather than letting the interviewer wonder if I forgot them.

The error most readers make here. Reaching for a distributed lock on the user id. It does not help. The lock protects the window between two of your processes, but the duplicate charge you actually see in production comes from one process retrying after a timeout, and a lock the same process re-acquires is no barrier at all.

Block B. Purpose: make a client retry cost nothing

The client generates an idempotency key when the user presses Pay, and reuses it for every retry of that press. Pressing Pay again generates a new key, because that is a new intent.

Server side, in one transaction:

+-------------------------------------------------------------+
| INSERT INTO idem(key, fingerprint, state) VALUES (k, f, NEW) |
|   conflict on key ->                                         |
|      state = DONE        -> return stored response           |
|      state = IN_PROGRESS -> return 409, Retry-After: 2       |
|      fingerprint differs -> return 422                       |
|   no conflict ->                                             |
|      proceed, then write state = DONE with the response      |
+-------------------------------------------------------------+
Caption: the four outcomes of an idempotency key insert, and the
response each one produces.

Sizing this table with the numbers above: 100 per second average times 86,400 is 8.64 M rows per day. Payments need a long horizon because a human may retry days later, so I keep 30 days: 259 M rows at roughly 300 bytes is about 78 GB. That fits one node with room, so I do not shard it, and I say that explicitly because restraint is a scored behaviour.

The error most readers make here. Letting the server mint the key. Then every retry looks like a new intent and the table becomes an audit log with no safety property.

Block C. Purpose: survive the ambiguous timeout

This is the block that separates a real answer from a recited one. Our timeout fires and we do not know whether the gateway charged the card.

I set the timeout at 10 s rather than close to the 4 s p99, because a timeout below the gateway's own tail converts slow-but-successful calls into ambiguous ones, and ambiguity is the expensive state. At 0.2% ambiguity and 8.64 M calls per day, that is 17,280 ambiguous charges per day. Nobody resolves 17,280 cases by hand, so automation is forced by the number, not by taste.

The resolution path:

1. Before calling the gateway, write an intent row with our own reference id and state PENDING. This must be durable before the network call, otherwise a crash loses the fact that we tried.
2. Send the reference id to the gateway as its own idempotency key, so the gateway deduplicates on our behalf.
3. On timeout, leave the row PENDING and return "processing" to the user.
4. A reconciler polls the gateway by reference id, on a backoff, and resolves PENDING to CAPTURED or FAILED.
5. Anything still PENDING after the gateway's own settlement horizon goes to a human queue with the reference id attached.

The error most readers make here. Retrying the gateway immediately on timeout without sending the same gateway-side key. That is the mechanism that produces the double charge you were hired to prevent.

Block D. Purpose: name what still breaks, and price the fix

I close by naming what I deliberately did not build: no distributed lock, no consensus group, no leader election. One unique index and one reconciler carry the whole correctness argument, and the operational surface is two objects instead of a quorum.

9e. Faded examples

Fade 1: exactly one order-confirmation email. Same shape, last block blank.

• Block A, decide whether agreement is needed: two sends produce a duplicate email. Repair is a social apology, not a system action, but the cost is low. Row two of the routing table.
• Block B, make retries free: unique index on (order_id, template_version) in an outbox table, inserted in the same transaction as the order.
• Block C, survive the ambiguous send: the mail provider accepts a caller message id, so send it and treat a timeout as PENDING; the provider's events webhook resolves it.
• Block D, what still breaks and what it costs: you write this block.

Fade 2: one nightly report per tenant across twelve app servers. Last two blocks blank.

• Block A, decide whether agreement is needed: two reports for one tenant means two emails and double compute. Repair is possible and cheap.
• Block B, make retries free: you write this block.
• Block C, survive the ambiguous run: you write this block.

9f. Check yourself

Q1. Your Raft group has five members across three availability zones with a 1 ms inter-zone round trip. A product manager asks for a globally consistent counter readable from Sydney with a 40 ms p99. What do you tell them?

Q2. A candidate proposes fencing tokens but the storage layer is a plain S3 compatible object store with no compare-and-set. What breaks, and what is the cheapest repair?

9g. When not to use this

Consensus (Raft, Paxos, or a hosted equivalent).

Distributed locks.

Hybrid logical clocks.

Module 10. Multi-region and geo-replication

10a. Pre-question

+-------------------------------------------------------------+
| Current: one region, us-east. All users, all data.           |
| Measured, last 30 days:                                      |
|   traffic from Europe          18%                           |
|   p99 page latency, US users   210 ms                        |
|   p99 page latency, EU users   540 ms                        |
|   requests per page load       4 sequential API calls        |
|   TLS + TCP handshakes per new connection   2 round trips    |
| Target: p99 under 300 ms everywhere.                         |
+-------------------------------------------------------------+
Caption: current single-region latency by geography, with the
per-page request structure that determines how the gap arises.

Question. Before adding a region, how much of the 330 ms EU gap can you recover without moving any data, and what does that imply about the decision?

10b. Explanation in small steps

Only three things force a second region. Say which one you are serving, out loud, in the first minute.

Candidates who conflate these produce active-active designs for a latency problem and pay conflict resolution costs forever.

The distance table you can derive at the whiteboard. Light in fibre travels at about 200,000 km per second, so 5 microseconds per km one way and 10 per km round trip. Multiply the great-circle distance, then add 30 to 50% for real routing and switching.

Consequence, stated as a rule: no synchronous cross-region dependency belongs in a user-facing request path with a p99 budget under about 400 ms, because one crossing plus your own processing already consumes most of it.

The three topologies, and the one most people mean.

The middle row is what most production "active-active" systems actually are, and naming it precisely is a senior signal. Each user, tenant or account has a home region recorded in a small global routing table. Reads are served locally from a replica. Writes are proxied to the home region.

Conflict resolution, compared honestly.

The pattern worth internalising: pick single-writer-per-entity first, and only move down this table for the specific fields that genuinely need concurrent writes from multiple regions.

What cannot be partitioned by region. Every design has a small set of globally unique things. Name them early.

• Global uniqueness constraints, for example usernames or short codes.
• Cross-tenant aggregates, for example a global leaderboard or a total count.
• Anything a regulator requires to be a single authoritative record.

Standard treatments: allocate ranges of ids per region so uniqueness is local (a region owns codes ending in its digit), or accept a single global service for the tiny slice that needs it and give it a generous latency budget because it is off the hot path.

Residency changes the data model, not just the deployment. If EU personal data must not leave the EU, then a global secondary index containing that data is illegal, a global cache of user objects is illegal, and cross-region lookups must return opaque references that the owning region resolves. Interviewers rarely raise this; raising it yourself is a strong signal.

Cost, derived. Assume 20,000 writes per second globally, 500 bytes per replicated record including framing, and an inter-region transfer price of $0.02 per GB. The price is an assumption: cloud list prices for inter-region transfer sat in the low cents per GB range through 2026, and you should substitute today's figure from your provider's page.

• Bytes: 20,000 times 500 B is 10 MB per second, 864 GB per day, about 26 TB per month.
• Active-passive, one direction: 26,000 GB times $0.02 is about $520 per month.
• Three-region mesh: each region generates a third of the writes and ships to two others, so 2 times 26 TB total, about $1,040 per month.

That is not the expensive part, and saying so is the point. The expensive parts are the extra full copies of storage (a 50 TB dataset at $0.08 per GB-month is $4,000 per copy per month, so two extra copies is $8,000), and the headroom: each region must be able to absorb the others' traffic, so a two-region pair running at 50% utilisation costs twice a single region running at 100%.

10c. ASCII diagram

     EU USER                            US USER
        |                                   |
        v                                   v
  +-------------+                     +-------------+
  | EDGE POP EU |                     | EDGE POP US |
  | TLS ends    |                     | TLS ends    |
  +------+------+                     +------+------+
         |                                   |
         v                                   v
  +---------------------+          +---------------------+
  | REGION eu-west      |          | REGION us-east      |
  | read replica        |<========>| read replica        |
  | writes for EU-homed |  async   | writes for US-homed |
  +----------+----------+          +----------+----------+
             |                                |
             +--------------+-----------------+
                            v
                  +-------------------+
                  | HOME REGION TABLE |
                  | user -> region    |
                  | tiny, cached      |
                  +-------------------+

Caption: home-region routing. Edges terminate TLS near the user, each
region owns writes for the users homed there, replication is
asynchronous, and one small global table records ownership.

10d. Worked example: take a photo-sharing app into Europe

The prompt. "We run in us-east only. Leadership says we are going global. Design it."

Block A. Purpose: find out which requirement is actually forcing this

I do not start drawing. I ask which of the three forcing requirements applies, because they produce different architectures.

The interviewer says: EU is 18% of traffic, EU users complain about speed, and legal has raised residency for EU accounts. That is two forcing requirements (latency and residency) and not the third (there is no stated requirement to survive losing a whole region). So I am building home-region partitioning, not full active-active, and I say that now so the rest of the conversation has a spine.

I also state what I am ruling out and why: full active-active would let an EU user write to us-east, which defeats the residency requirement and buys nothing, since a user's writes come from one place at a time.

The error most readers make here. Accepting "we are going global" as a requirement. It is a business statement. The design forks completely on which of the three it means, and asking costs eight seconds.

Block B. Purpose: choose the partition axis, and name what will not partition

The axis is the account. Every account gets a home_region on creation, chosen by signup IP with a manual override. Photos, comments, likes and the account record all live in the home region.

What does not partition, and my answer for each:

The error most readers make here. Partitioning by data type instead of by entity owner. "Photos in EU, comments in US" creates a cross-region join on every page load and violates residency at the same time.

Block C. Purpose: design the cross-region read with a stated staleness budget

An EU user opens a US user's profile. Options, and the numbers that choose:

I choose replicated public profiles plus CDN fragments, with a stated staleness budget of 60 seconds for profile fields and 5 seconds for follower counts. I put the number in the API contract, because "eventually consistent" without a number is not a design, it is a shrug.

The residency line is drawn here: only public fields replicate. Email, phone, precise location and private albums never leave the home region, so the EU replica of a US profile carries display name, avatar URL and counts, nothing else.

The error most readers make here. Replicating the whole row because it is easier, then discovering during the legal review that the design is illegal and the schema has to be split anyway.

Block D. Purpose: cost the move and rehearse the failure

Costs, using the assumption prices above and a stated dataset of 50 TB:

Total marginal storage and transfer is under $1,000 per month, which reframes the conversation: the cost of this project is engineering time and operational complexity, not infrastructure. Saying that out loud is a senior move.

Failure rehearsal, which is the block that gets skipped when time runs short:

• eu-west loses its database primary. EU users cannot write. US users are unaffected because they are homed elsewhere. Blast radius is 18% of write traffic, not 100%, and that containment is the main non-latency benefit.
• The home-region routing table is unavailable. Every request now cannot find its home. This is the new single point of failure I created, so it gets a read-only local cache in every region with a long TTL, and it is small enough (one row per account, tens of bytes) to hold entirely in memory.
• Replication lag to the EU public-profile replica exceeds 60 seconds. Serve stale and emit a metric, do not fail the request, because a stale display name is not an incident.

10e. Faded examples

Fade 1: add a second region to a payments ledger. Last block blank.

• Block A, forcing requirement: availability. The business states a target of surviving a full region loss with an RPO under 5 seconds.
• Block B, partition axis: none. A ledger has a global invariant (debits equal credits per account) and accounts are not geographically owned, so this is active-passive, not partitioned. Writes go to one region.
• Block C, cross-region read path: read replicas serve balance queries with a stated staleness of 5 seconds; anything that decides a transaction reads from the primary region.
• Block D, cost and failure rehearsal: you write this block.

Fade 2: add a second region for a session store. Last two blocks blank.

• Block A, forcing requirement: latency. Session validation happens on every request, and a cross-region hop on every request is unaffordable.
• Block B, partition axis: you write this block.
• Block C, cross-region read path: you write this block.

10f. Check yourself

Q1. Your interviewer says "make it active-active so we get better availability". What is the one number you ask for, and how does the answer change your design?

Q2. An EU user and a US user both edit the same shared album title within 50 ms of each other, and your regions are 80 ms apart. What does each conflict-resolution strategy produce, and which do you ship?

10g. When not to use this

A second region.

Full active-active with conflict resolution.

Cross-region synchronous replication.

Module 11. The failure library

Every named failure below is a shape, not a story. Learn the shape, the amplification factor, the design decision that prevents it, and the signal that proves it is happening. Naming the shape in an interview is worth more than describing the symptom.

11a. Pre-question

+-------------------------------------------------------------+
| Service A calls Service B. Both are healthy at t=0.          |
|   A: 5,000 rps, timeout on B = 2 s, retries = 3, no jitter   |
|   B: capacity 6,000 rps, p99 latency 40 ms                   |
| At t=1 min, B's database gets slow. B's p99 goes to 3 s.     |
| At t=6 min, the database is fixed. B's p99 would be 40 ms    |
|   again if the offered load were 5,000 rps.                  |
+-------------------------------------------------------------+
Caption: a two-service call chain with its retry policy, capacity
and the timeline of a transient dependency slowdown.

Question. What is the offered load on B at t=2 min, and does B recover on its own at t=6 min?

11b. Explanation in small steps

The taxonomy that matters: does it self-heal when the trigger goes away?

Only the third class needs a kill switch, and it is the class that is hardest to name under pressure. The formal treatment is Bronson et al., Metastable Failures in Distributed Systems, HotOS 2021 (paper, accessed August 2026).

The library. Each row gives the amplification factor you can compute.

Derivation 1: the retry budget. A per-request retry count of 3 sets a worst case of 4x offered load, and your fleet is usually provisioned at 50 to 70% utilisation, so 4x guarantees overload.

Instead, cap retries as a fraction of the request rate with a client-side token bucket: allow retries up to 10% of the current request rate. Worst case is then 1.1x, and healthy-state retries still work because the bucket is nearly always full. AWS publishes a clear treatment of this and of jitter: timeouts, retries, and backoff with jitter (accessed August 2026).

Derivation 2: cache stampede concurrency. One key, 10,000 requests per second, a 50 ms backend query. When the key expires, every request in the next 50 ms misses: 10,000 times 0.05 is 500 concurrent backend queries for one key. If the backend allows 100 concurrent queries, you have just built an outage from a cache expiry. With single-flight, the number is 1. With serve-stale-while- revalidate, the number is 1 and no request waits.

Derivation 3: jitter. One million mobile clients poll every 300 seconds. If they synchronise (a push restart, a midnight boundary, a deploy), the peak is 1,000,000 requests in roughly one second. With full jitter, meaning the next interval is drawn uniformly from 0 to 300 s, the arrivals spread across 300 s and the peak is about 3,300 per second. That is a 300x reduction from one line of client code, which is why jitter is the highest-leverage item in this module.

Derivation 4: hot key. 1,000 shards, uniform traffic means each shard sees 0.1% of load. One celebrity account draws 5% of reads. At 200,000 reads per second globally, that key's shard sees 10,000 reads per second against a shard capacity of, assume, 2,000. It is 5x over while 999 shards idle. Fixes, in increasing cost: cache the key at the edge with a short TTL; replicate the key across N shards and read from a random one; split the key by suffix and merge on read.

Derivation 5: timeout ordering. If the gateway times out at 1 s and the service behind it times out at 2 s against the database, then between 1 s and 2 s the service is doing work no one will read, holding a connection and a thread. Under load this is pure waste that deepens the overload. The rule: timeouts strictly decrease along the call chain, and each hop passes its remaining deadline down so the callee can abandon early.

Little's law, the one formula worth memorising. L = lambda times W. Items in the system equals arrival rate times time in system.

• 5,000 requests per second at 40 ms means 200 requests in flight. A thread pool of 100 is already the bottleneck.
• The same 5,000 per second at 2 s means 10,000 in flight. Nothing you own has 10,000 threads, so the queue is what actually absorbs it, and an unbounded queue converts an overload into an unbounded latency.

The one rule that prevents most of this. Every queue has a bound and a shed policy. An unbounded queue does not prevent failure, it converts a fast failure into a slow one and hides it from your latency alerts until the memory runs out.

11c. ASCII diagram

   OFFERED LOAD                       CAPACITY LINE
        |                                   |
  4x  --+-------- retry storm --------------+---- overload
        |            |                      |
  1.1x--+---- retry budget caps here -------+---- safe
        |
        +--> sustaining loop? --> yes --> METASTABLE
        |                                (no self-heal)
        +--> sustaining loop? --> no  --> TRANSIENT
                                         (waits it out)

   ESCAPE FROM METASTABLE:
   +------------+   +--------------+   +----------------+
   | shed load  |-->| below recov. |-->| restore slowly |
   | (drop 80%) |   | threshold    |   | (ramp 10%/min) |
   +------------+   +--------------+   +----------------+

Caption: retry amplification against capacity, the test that decides
whether a failure self-heals, and the three-step escape from a
metastable state.

11d. Worked example: the feed service that would not come back

The prompt. "Your feed service p99 went from 80 ms to 30 s during a database incident. The database was repaired 40 minutes ago. The feed service is still at 30 s. Diagnose and fix it, live."

Facts I have on the dashboard:

Block A. Purpose: decide which class of failure this is, before touching anything

The trigger is gone and the system has not recovered. That single fact moves this out of "wait it out" and into metastable, and it changes what I do: no amount of patience helps, and adding capacity may not help either, because capacity that gets consumed by the loop just feeds the loop.

The proof is in the two rate numbers: 8,000 requests in, 31,000 queries out. Roughly 3.9 queries per request against a baseline I expect to be near 1. That ratio is the amplifier, and finding the amplifier is the whole first move.

The error most readers make here. Scaling up. Doubling pods doubles the connection demand on a database that is already the constraint, which deepens the loop. Under metastable conditions, scale-up is often the wrong direction.

Block B. Purpose: identify the sustaining loop, not just the amplifier

The 3.9x could be retries or it could be a cache that stopped working. I check cache hit rate: it fell from 94% to 11% during the incident and has not recovered.

That is two loops stacked, and I name both:

1. Retry loop. Timeouts fire, clients retry 3 times, load quadruples.
2. Cache loop. Slow backends mean fills time out, so nothing repopulates, so every request goes to the database, so the database stays slow, so fills keep timing out. The cache cannot refill under load, which is the classic cold-cache metastable trap.

The second loop is the one that survives a retry fix, so I attack it too.

The error most readers make here. Finding the retry storm, fixing it, and declaring victory. The cache loop alone keeps a system down.

Block C. Purpose: escape the state, in the order that actually works

Three actions, in this sequence, and the sequence matters:

Shedding 80% of traffic during an incident feels wrong and is correct. Serving 20% of users well beats serving 100% of users a 30 s timeout, and the ramp is what stops the loop restarting the moment the gate opens.

The error most readers make here. Shedding 20% instead of 80%. The shed must take offered load below the recovery threshold, and the recovery threshold is well below normal capacity because a cold cache multiplies per-request cost.

Block D. Purpose: make the state unreachable, and price each fix

I close by naming the SLI that would have caught this in the first two minutes: queries-per-request as a ratio, alerted when it exceeds 1.5. That single derived metric distinguishes "the database is slow" from "we are amplifying", and it is the thing I would add before any of the fixes above.

11e. Faded examples

Fade 1: the midnight spike. Last block blank.

• Block A, classify: every night at 00:00 UTC the API sees a 40x spike lasting 90 seconds, then normal. It self-heals, so it is not metastable. The shape is a thundering herd from clients whose daily sync is scheduled on a wall-clock boundary.
• Block B, find the amplifier: 2 M devices, all firing a daily sync at 00:00. Compressed into 90 s, that is roughly 22,000 rps against a normal 550 rps.
• Block C, escape now: raise the rate limit temporarily and let the queue absorb it, because it self-heals and the spike is short.
• Block D, make it unreachable and price each fix: you write this block.

Fade 2: the checkout that dies on Black Friday. Last two blocks blank.

• Block A, classify: at 3x normal load, checkout latency goes from 200 ms to 9 s and stays there until the fleet is restarted. It does not self-heal, so it is metastable.
• Block B, find the amplifier and the loop: you write this block.
• Block C, escape now: you write this block.

11f. Check yourself

Q1. A service has a 99.99% success rate and its client retries once on failure. A colleague argues that retries make the effective success rate 99.999999%. What is wrong with that reasoning?

Q2. Name the leading indicator you would alert on for a queue backlog spiral, and say why the obvious latency alert is too late.

11g. When not to use this

Automatic load shedding.

Circuit breakers.

Fallback paths.

Module 12. Observability and operability as design output

A design you cannot operate is not a design. This module treats the SLI set, the alert set, the label schema and the debug path as artefacts you draw on the whiteboard, in the same pass as the boxes.

12a. Pre-question

+--------------------------------------------------------------+
| Proposed metric, one per HTTP request:                        |
|   http_request_duration_seconds                               |
| Labels:                                                       |
|   route          (200 distinct values)                        |
|   status_code    (8 distinct values)                          |
|   pod            (30 pods)                                    |
|   region         (3 regions)                                  |
|   customer_id    (proposed by a stakeholder, 1,000,000 values)|
| Scrape interval: 15 s.  Histogram: 12 buckets.                |
+--------------------------------------------------------------+
Caption: a proposed request-duration metric with its label
cardinality, scrape interval and bucket count.

Question. How many time series does this produce with and without customer_id, and what does that do to storage per day?

12b. Explanation in small steps

Four definitions, in dependency order.

Error budget as time, derived. A 30 day month is 43,200 minutes.

Use this table to end arguments. When someone asks for five nines, show them 26 seconds per month and ask which deploy fits in it.

Burn-rate alerting, derived. Alerting on "error rate above 1%" pages you for things that do not threaten the objective and misses slow bleeds. Alert on the fraction of budget consumed instead.

• A one hour window is 1/720 of a 30 day month.
• Burning at 14.4 times the even rate for one hour consumes 14.4/720, that is 2%, of the month's budget.
• Burning at 6 times for six hours consumes 6 times 6/720, that is 5%.
• Burning at 1 time for the whole month consumes exactly 100%, by definition.

So: page on a fast burn (14.4x over 1 hour, confirmed by a 5 minute window to avoid single-scrape noise), and open a ticket on a slow burn (3x over 24 hours). The canonical public treatment is the Google SRE Workbook chapter on alerting on SLOs (accessed August 2026).

Two method families, and when each applies. RED covers request-driven services: Rate, Errors, Duration. USE covers resources: Utilisation, Saturation, Errors. Brendan Gregg's published description of the USE method is at usemethod (accessed August 2026).

Saturation is the item people omit and the one that pages earliest. Connection pool wait time, queue depth and thread pool queue length are all saturation signals that rise before latency does.

Choose SLIs from the user journey, not from the architecture. For each journey, one availability SLI and one latency SLI. Two rules:

• Measure as close to the user as you can afford. A server-side 200 that the client never received is not a good event.
• Count valid events carefully. Requests from a load test, from a scanner, or with a malformed body are not valid events, and including them either hides real failures or manufactures fake ones.

Cardinality budget, as a design constraint. Give yourself a number before you design the labels, the same way you give yourself a latency budget.

Tracing, and the two sampling strategies.

A pragmatic default: head sample at a low rate for the baseline, and force-sample every trace that errors or exceeds a latency threshold, using a sampling decision propagated in the trace context so the whole trace is kept.

Trace propagation is a design constraint, not a library setting. Every hop that drops the context breaks the trace: a queue that does not carry headers, a thread pool that does not propagate context, a third party that strips unknown headers. Draw the propagation path on the diagram and mark where it breaks.

The operability block every design should end with. Four items, in order:

1. Three SLIs with their measurement point.
2. Two alerts: one fast-burn page, one slow-burn ticket.
3. One dashboard that answers "is it us or a dependency" in under 30 seconds.
4. One debug path: given a user complaint with a timestamp, the exact steps to find their request.

12c. ASCII diagram

  USER JOURNEY                 SIGNALS PRODUCED
  +---------------+
  | open feed     |---> SLI: good = 200 in < 300 ms
  +-------+-------+           valid = authenticated GET /feed
          |
          v
  +---------------+
  | GATEWAY       |---> RED: rate, errors, duration (low card)
  |               |---> trace starts here, id propagates down
  +-------+-------+
          |
          v
  +---------------+     +----------------+
  | FEED SERVICE  |---->| CACHE          |--> USE: hit rate,
  |               |     |                |    eviction, memory
  +-------+-------+     +----------------+
          |
          v
  +---------------+
  | DATABASE      |---> USE: pool wait, saturation, replica lag
  +---------------+

  ALERTS: fast burn 14.4x/1h -> page | slow burn 3x/24h -> ticket

Caption: signals mapped onto one user journey. RED at request
boundaries, USE at resources, one SLI defined at the user-visible
edge, and the two burn-rate alerts derived from it.

12d. Worked example: make the URL shortener operable

The prompt. "You designed a URL shortener. Now tell me how you run it." Given: 10,000 redirects per second, 100 creates per second, redirect p99 target 50 ms, business impact is that a failed redirect is a lost click.

Block A. Purpose: pick SLIs from what the user does, not from what we built

Two journeys, so two SLI pairs. I refuse to write an SLI for the cache or the database, because those are causes, not user experiences.

Valid events exclude requests for codes that never existed (those correctly return 404 and are not our failure) and exclude synthetic load-test traffic by user agent. I say the exclusions out loud, because a candidate who defines valid events precisely is signalling operational experience.

The error most readers make here. Defining the SLI on the server's own response code. If the edge returns 502 because our origin was slow, our origin logs a success and our SLI stays green while every user fails.

Block B. Purpose: set targets the business can defend, then derive the alerts

Redirects lose money per failure and are read-only, so the target is high: 99.95%, which is 21.6 minutes per 30 days. Creates are retried by a human and happen 100x less often, so 99.9% (43.2 minutes) is honest and cheaper.

Alerts derived from 99.95% on redirects:

Notice that the fast-burn page fires at 0.72% errors and the slow-burn ticket at 0.15%. Both are derived from one SLO, not chosen by feel, and that derivation is the thing to show.

The error most readers make here. Setting a static "alert if errors above 1%" threshold. At 10,000 rps, 1% is 100 failures per second, and a 99.95% SLO is blown in under four minutes at that rate. The static threshold is far too loose for the fast case and silent for the slow bleed.

Block C. Purpose: design the label schema inside a stated cardinality budget

Budget: 50,000 series for this service. Now I spend it.

Total is about 2,000 series, comfortably inside 50,000, which leaves room for the metrics I have not thought of yet. Explicitly banned as labels: the short code (unbounded), the destination URL (unbounded), the user id (unbounded). All three go on traces and on sampled logs instead, where a 1% sample makes them free.

The error most readers make here. Adding the short code as a label because hot-key detection sounds like a metrics problem. It is a top-K problem: solve it with a sampled top-K stream or a periodic log aggregation, not with a time series per code.

Block D. Purpose: write the debug path before the incident

Given a user saying "my link was broken around 14:32 UTC", the on-call steps:

1. Dashboard panel one: SLI for redirects, last 6 hours, split by region and edge point of presence. Answers "was it everyone or one place".
2. Dashboard panel two: dependency panel, cache hit rate and database pool wait, same window. Answers "is it us or a dependency" in under 30 seconds, which is the stated requirement.
3. If it was localised, query sampled logs for the short code in the window. Codes are on logs, not metrics, which is exactly why the sampling rate for error traces is 100%.
4. Pull the trace by id from the log line, and read the span durations.

I finish by naming the one derived metric I would add beyond the standard set: database queries per redirect. Its healthy value is near 0.06 (a 94% cache hit rate), and an alert at 0.3 catches a cache degradation before latency moves.

12e. Faded examples

Fade 1: operability for a queue-based image worker. Last block blank. Given: 500 images per second arriving, target is 95% processed within 60 s.

• Block A, SLIs from the journey: freshness, meaning the fraction of images whose end-to-end age at completion is under 60 s, measured from the enqueue timestamp, not the dequeue timestamp. Plus a success ratio on processing.
• Block B, targets and alerts: 95% within 60 s over 28 days. Fast burn at 14.4x over one hour pages; oldest-unprocessed-message age over 120 s pages immediately, because that is the leading signal.
• Block C, label schema inside budget: image_type (6), status (4), region (3), plus a duration histogram, staying near 300 series. No image id, no user id, no filename as labels.
• Block D, the debug path: you write this block.

Fade 2: operability for a shared cache tier. Last two blocks blank. Given: 200,000 operations per second, 40 nodes, used by 6 services.

• Block A, SLIs: the cache has no direct user journey, so the honest position is that it gets no SLO of its own; it gets USE signals and its effect appears in the SLOs of the six services that depend on it.
• Block B, targets and alerts: you write this block.
• Block C, label schema inside budget: you write this block.

12f. Check yourself

Q1. Your service reports 99.98% availability from its own metrics, but users report frequent failures. Name two structural reasons the number can be right and useless.

Q2. A stakeholder asks for 99.999% availability. Give the two-sentence reply that moves the conversation forward rather than refusing.

12g. When not to use this

Distributed tracing.

Tail sampling.

A dedicated metrics platform per team.

Module 13. Cost as a first-class constraint

13a. Pre-question

+--------------------------------------------------------------+
| Service, steady state:                                        |
|   10,000 requests per second                                  |
|   5 ms of CPU per request                                     |
|   2 KB response body, served to the public internet           |
|   target CPU utilisation 60%                                  |
| Assumed list prices (substitute your provider's current page):|
|   compute        $0.04 per vCPU-hour                          |
|   egress         $0.08 per GB to the internet                 |
+--------------------------------------------------------------+
Caption: a steady-state service with its CPU cost per request,
response size and the two unit prices needed to cost it.

Question. Which is larger, the compute bill or the egress bill, and by how much?

13b. Explanation in small steps

Assumed unit prices used throughout this module. These are order-of-magnitude list prices in the range cloud providers published through 2026. They are labelled assumptions, and the point is the method, not the digits. Substitute your provider's current page before quoting any of this in a real design review.

Two structural facts worth more than the numbers: moving bytes usually costs more than storing them for a month, and every price above is per unit, so the only way to argue about cost is to know your units per request.

Express everything as a unit cost. Three units cover almost every prompt.

From the pre-question: $6,600 per month over 25.92 million thousands of requests is about $0.00025 per 1,000 requests. That number is more useful than the total because it survives a traffic forecast.

The four cost shapes, and what each one implies.

Derivation: the cross-zone tax. Three availability zones, callers pick targets uniformly, so 2 out of 3 calls leave the zone. Take the pre-question service: 25.92 billion requests per month, 3 internal calls each, 4 KB of request plus response per internal call.

• Internal bytes: 25.92e9 times 3 times 4 KB is about 311 TB per month.
• Crossing a zone boundary: two thirds, so about 207 TB.
• At $0.01 per GB: 207,000 GB times $0.01 is about $2,070 per month.

That is 84% of the compute bill, generated by nothing the architecture diagram shows. Zone-aware routing (prefer a same-zone replica, fall back across zones) removes most of it and costs one load-balancer setting. This is the highest value-per-word cost point you can make in an interview.

Derivation: when a cache pays for itself. A cache is worth its cost when the avoided backend work exceeds the cache's own bill.

• Backend read cost: assume a database read consumes 2 ms of CPU. At $0.04 per vCPU-hour, one CPU-second is $0.0000111, so one read is about $0.0000000222.
• 10,000 reads per second for a month is 25.92 billion reads, costing about $576 in database CPU.
• A 64 GB cache node costs, at the memory price, 64 times $0.005 times 730, that is about $234 per month, and you need at least two for redundancy, so $468.

So a 94% hit rate saves about $541 of database CPU and costs $468. On money alone it is roughly break-even, and the honest conclusion is that caches are almost never justified by cost. They are justified by latency and by protecting a backend that cannot be scaled horizontally. Saying this correctly is what separates a cache you can defend from a cache added reflexively, or one claimed to save money it does not save.

Derivation: retention as the dominant storage lever. 500 GB of new data per day, 3 replicas, block storage at $0.08 per GB-month.

Now tier: keep 30 days hot on block storage and the rest on object storage at $0.021 with a single replica handled by the service.

• Hot: 45 TB at $0.08 is $3,600.
• Cold, 335 days at 500 GB per day is 167 TB, at $0.021 is about $3,507.
• Total about $7,100 against $43,800, a 6x reduction, for the cost of one lifecycle policy and a slower read path for old data.

How to argue for a cheaper design without sounding cheap. Four moves, in order of effectiveness:

The failure mode to avoid is proposing the cheap design without the number, which reads as timidity, or proposing the expensive one without the number, which reads as inexperience.

13c. ASCII diagram

  COST OF ONE REQUEST, DECOMPOSED
  +--------------------------------------------------------+
  | CPU        5 ms  x $0.04/vCPU-hr / 3600 = $0.000000056  |
  | EGRESS     2 KB  x $0.08/GB             = $0.00000016   |
  | INTERNAL   3 calls x 4 KB x 2/3 x $0.01 = $0.00000008   |
  | STORAGE IO amortised per request        = small         |
  +--------------------------------------------------------+
        |            |               |
        v            v               v
  +-----------+ +-----------+ +---------------+
  | fix by    | | fix by    | | fix by        |
  | less work | | fewer     | | zone-aware    |
  | per req   | | bytes     | | routing       |
  +-----------+ +-----------+ +---------------+

Caption: one request's cost split into three components with the
lever that reduces each. Egress dominates CPU by roughly 3x here, and
invisible internal chatter is comparable to CPU.

13d. Worked example: cost the news feed fan-out decision

The prompt. "Push or pull for the feed? Justify it with numbers."

Given and assumed inputs, stated before any arithmetic:

Block A. Purpose: cost the push design at the median, where it looks good

Fan-out on write. Every post is copied into each follower's feed list.

• Feed inserts: 1 M posts times 200 followers is 200 M inserts per day.
• Average insert rate: 200 M divided by 86,400 is about 2,315 per second. Peak at 3x is about 7,000 per second, which one well-provisioned cluster handles.
• Storage: 30 M active users times 500 entries times 200 bytes is 3 TB. At $0.08 per GB-month with 3 replicas, that is 3,000 times $0.08 times 3, about $720 per month.
• Read cost: a feed open is one range scan of a contiguous list. 30 M DAU times 4 opens is 120 M reads per day, about 1,390 per second.

The push design's read path is nearly free, which is why it wins by default.

The error most readers make here. Stopping here. The median is not where this design fails, and an interviewer who hears only the median number concludes you have not built one of these.

Block B. Purpose: find the input that eliminates push, and quantify it

The 50 M follower account. Assume it posts 5 times a day.

• Inserts from that one account: 50 M times 5 is 250 M per day.
• The entire rest of the system generates 200 M per day.

One user generates more write load than every other user combined. Worse, the insert burst is not spread: a post fans out to 50 M rows, and at 7,000 inserts per second of spare capacity that single post takes about two hours to deliver, which is a product failure, not just a cost one.

This number eliminates pure push. It does not eliminate push for everyone, which is the distinction that matters.

The error most readers make here. Concluding "so use pull". Pull for everyone costs a fan-in query across 200 followees on every feed open: 120 M opens per day times 200 lookups is 24 billion lookups per day, about 278,000 per second, which is roughly 200x the push design's read rate. Pull fixes the tail and breaks the median.

Block C. Purpose: cost the hybrid and find the threshold that defines it

Hybrid: push for normal accounts, pull for accounts above a follower threshold T. Merge at read time.

Choose T by cost. Pushing one post to F followers costs F inserts. Pulling that author at read time costs one extra lookup per feed open by each of the F followers, and each follower opens 4 times a day.

• Push cost per post: F inserts.
• Pull cost per post-day: F followers times 4 opens is 4F lookups.

At first glance pull looks worse, but the pull lookup is one query against the author's own timeline, cached and shared, whereas the push insert is F distinct row writes to F distinct partitions. Assume a cached author-timeline read is 20 times cheaper than a partitioned insert (an assumption you should replace with a measurement, but the direction is robust because one is a cache hit and the other is a durable write).

• Push: F insert-units.
• Pull: 4F divided by 20, that is 0.2F insert-units, per day of posting.

So pull is cheaper per follower once the account posts often enough that its posts are re-read many times, and push is cheaper for accounts whose posts are read few times. The practical threshold is where the burst matters rather than the average: set T so that a single post's fan-out completes inside the freshness budget. At 7,000 inserts per second of headroom and a 30 second freshness target, T is 7,000 times 30, that is 210,000 followers. Round to 100,000 for safety.

Result: accounts under 100,000 followers are pushed, which covers the overwhelming majority of posts; accounts over it are pulled and merged at read time. The threshold was derived from a latency budget and a measured insert rate, not chosen because it sounded round.

The error most readers make here. Presenting the hybrid without a threshold. "Push for normal users, pull for celebrities" is a shape, not a design. The number, and where the number came from, is the answer.

Block D. Purpose: state the total, the unit cost, and what would change it

Total is roughly $3,000 per month, or $0.0001 per active user per month. The striking result is that egress on the response bodies dominates, so the highest leverage cost optimisation is not the fan-out strategy at all: it is shrinking the 25 KB feed response, for example by returning ids and letting the client fetch cached post bodies from the CDN.

I would say exactly that in the interview, because arriving at "the thing we spent twenty minutes arguing about is not the expensive part" is a stronger finish than defending the hybrid.

13e. Faded examples

Fade 1: cost a video transcode pipeline. Last block blank. Given: 10,000 uploads per day, average 8 minutes, 5 output renditions, transcoding runs at roughly 2x realtime per vCPU (assumption, replace with a measurement for your codec and preset).

• Block A, cost the obvious component: CPU-minutes per upload is 8 minutes times 5 renditions divided by 2x realtime, that is 20 CPU-minutes. Times 10,000 uploads is 200,000 CPU-minutes per day, about 3,333 CPU-hours per day, or 100,000 per month. At $0.04 that is about $4,000 per month.
• Block B, find the input that changes the answer: the 5 renditions. Dropping to 3 by using per-title adaptive ladders cuts transcode cost by 40% and affects only a minority of devices.
• Block C, cost storage and delivery: 10,000 videos per day at, assume, 300 MB across all renditions is 3 TB per day, 90 TB per month accumulating. Delivery at, assume, 20 views per video times 300 MB of one rendition is far larger than storage.
• Block D, total, unit cost and what would change it: you write this block.

Fade 2: cost an idempotency table at payments scale. Last two blocks blank. Given: 100 writes per second average, 300 peak, 30 day retention, 300 byte rows, plus one index on the key.

• Block A, cost the obvious component: 100 times 86,400 is 8.64 M rows per day, 30 days is 259 M rows, at 300 bytes is 78 GB, plus an index at roughly 30% is about 101 GB. With 3 replicas at $0.08 per GB-month, that is about $24 per month. The storage is free in any practical sense.
• Block B, find the input that changes the answer: you write this block.
• Block C, cost the thing that is not storage: you write this block.

13f. Check yourself

Q1. A colleague proposes moving from three replicas to two to save a third of the storage bill on a 200 TB dataset. Compute the saving and give the counter-argument in the same breath.

Q2. Your design serves 1 KB JSON responses at 50,000 requests per second to the public internet. Name the single change with the largest cost effect and estimate it.

13g. When not to use this

Detailed cost modelling in an interview.

Reserved capacity and commitment discounts.

Chargeback and per-team cost attribution.

Module 14. Simplicity and scale thresholds

Removing a box is the move this page rates most highly, because it is the one that cannot be faked without the arithmetic. No vendor teaches restraint, because complexity fills pages. This module gives you the thresholds that make restraint defensible rather than lazy.

14a. Pre-question

+--------------------------------------------------------------+
| Proposed design for an internal job board:                    |
|   React app -> API gateway -> 4 microservices                 |
|   -> Kafka -> stream processor -> Elasticsearch               |
|   -> sharded Postgres (8 shards) -> Redis cluster (6 nodes)   |
| Stated requirements:                                          |
|   500 new job posts per day                                   |
|   200,000 searches per day                                    |
|   3,000 companies, 400,000 candidate accounts                 |
|   search results may be up to 60 seconds stale                |
+--------------------------------------------------------------+
Caption: a proposed 8-component architecture next to the four
requirement numbers it is meant to satisfy.

Question. Convert the two traffic numbers into per-second rates, then say how many of the eight components the requirements actually justify.

14b. Explanation in small steps

What one machine actually does. These are assumptions, deliberately conservative, and you should replace them with a measurement from your own hardware. They exist so you can do arithmetic rather than gesture.

Two numbers to keep: 50,000 reads per second and 10,000 writes per second on a single node. Almost every interview prompt is under one of those.

The threshold table. For each component, the measurement that justifies it and the threshold below which it is over-engineering.

The interaction cost of every box you add. N components have N times N minus 1, all over 2, pairwise interfaces that can fail. This is a combinatorial heuristic, not a measurement, and it is honest as a heuristic:

Going from 3 boxes to 8 multiplies the interaction surface by roughly 9. Not all pairs actually talk, but every stateful box independently adds a deploy, a version, a backup, a restore rehearsal, a capacity model and an on-call runbook. That list is the real cost, and reciting it is how you defend removing a box without sounding like you cannot build one.

Vertical before horizontal, stated as a rule. The largest single instances available on major clouds carry hundreds of vCPUs and a terabyte or more of memory. Doubling an instance takes a maintenance window. Sharding takes a quarter and is permanent. Exhaust the reversible option first.

How to defend a single-node answer when the interviewer expects a cluster. The script has four beats, and the order matters:

1. State the number. "At 23 searches per second, one Postgres is at under 1% of its capacity."
2. State the trigger for the next step. "I would add a read replica when reads push the primary past 60% utilisation, which at this query mix is about 20,000 per second, so roughly 900 times current traffic."
3. State the reversibility. "A read replica is a day of work and is reversible. Sharding is a quarter and is not, so it comes much later."
4. Offer the harder version explicitly. "If you would like, I can design the version at 100 times this traffic, which is where the shape genuinely changes."

Beat four is the one that converts a restraint answer from a risk into a strength, because it proves you chose simplicity rather than defaulting to it.

The two failure modes of restraint. Be honest about both.

14c. ASCII diagram

  TRAFFIC ---------------------------------------------->
  1/s        100/s      5k/s      20k/s     10k w/s    +

  +---------+
  | 1 app   |
  | 1 db    |
  +----+----+
       |
       +----------> +-----------+
                    | + replica |
                    | (reads    |
                    |  > 60%)   |
                    +-----+-----+
                          |
                          +--------> +------------+
                                     | + cache    |
                                     | (p99 miss) |
                                     +-----+------+
                                           |
                                           +-----> +---------+
                                                   | + shard |
                                                   | (RAM or |
                                                   |  fsync) |
                                                   +---------+

Caption: the escalation ladder. Each rung names the measurement that
promotes you to it, and nothing is added without crossing the
threshold on the axis above.

14d. Worked example: the internal job board from the pre-question

The prompt. "Design an internal job board." The interviewer has drawn nothing and is waiting.

Block A. Purpose: convert requirements into rates before drawing anything

I write four numbers on the board before a single box:

Every one of those numbers eliminates something. That is the standard from the estimation module and it applies here with more force, because the elimination is the answer.

The error most readers make here. Drawing the boxes first and sizing after. Once a Kafka box is on the whiteboard, removing it feels like a retreat, so you defend it. Size first and the box never gets drawn.

Block B. Purpose: build the smallest design that satisfies every stated requirement

  +---------+     +-------------+     +----------------+
  | BROWSER |---->| APP SERVER  |---->| POSTGRES       |
  |         |     | (2 procs,   |     | primary        |
  |         |     |  1 vCPU ea) |     | + text index   |
  +---------+     +------+------+     +--------+-------+
                         |                     |
                         |                     v
                         |            +----------------+
                         |            | READ REPLICA   |
                         +----------->| (failover, not |
                                      |  load)         |
                                      +----------------+
       +-------------------------------------------+
       | OBJECT STORE: resumes and company logos    |
       +-------------------------------------------+

Caption: the smallest design meeting all four requirements. Two app
processes for redundancy, one Postgres with a text index, one replica
present for failover rather than for read capacity.

Justification, requirement by requirement:

• 0.006 writes per second against roughly 10,000 available: 0.00006% utilised.
• 23 searches per second against roughly 50,000 cached reads: under 0.05%.
• Under 1 GB of data on a machine with hundreds of gigabytes of RAM: the entire dataset is resident, so every query is a memory hit.
• Full text search over a corpus this small is a built-in index, and the 60 second staleness allowance is not even needed, because the index is synchronous.

The replica is present for a reason I state explicitly: not for read load, which is negligible, but so that a lost primary is a promotion rather than a restore from backup. That is a redundancy requirement, not a scale one, and separating those two justifications is the point of the block.

The error most readers make here. Adding the replica and then claiming it helps with read scale. It does not here, and claiming it does tells the interviewer you attach reasons to components after choosing them.

Block C. Purpose: name the trigger for every component I did not add

This is the block that converts restraint into expertise. For each thing the interviewer might have expected:

Notice that one trigger, Elasticsearch, is a product decision rather than a scale one. Saying so is important: not every component is justified by traffic, and pretending otherwise is its own kind of dishonesty.

The error most readers make here. Listing what was not added without the trigger numbers. "We do not need Kafka yet" is an opinion. "We need Kafka above roughly 1,000 events per second and we are at 5" is an engineering position.

Block D. Purpose: check the simple design against every non-functional requirement

This is the guard against under-designing, and I run it out loud:

The security row is deliberately the one that fails. Small designs are not automatically complete designs, and finding the requirement your simplicity did not address is what stops this module becoming an excuse.

14e. Faded examples

Fade 1: an internal analytics dashboard. Last block blank. Given: 40 internal users, 6 dashboards, each running 12 queries, refreshed every 5 minutes during business hours, over a 2 TB event table.

• Block A, convert to rates: 40 users times 6 dashboards times 12 queries every 300 seconds is at most 9.6 queries per second, and in practice far less because users do not all watch all dashboards. The 2 TB table is the only interesting number.
• Block B, smallest design: a columnar store or a partitioned table with pre-aggregated rollups computed by a 5 minute cron. No streaming, no cache, no separate serving layer, because the refresh cadence already permits batch.
• Block C, triggers for what was not added: streaming if the freshness requirement drops under about 1 minute; a serving cache if concurrent query count pushes the warehouse past its slot limit; a separate warehouse if analytics queries start affecting production.
• Block D, check against non-functional requirements: you write this block.

Fade 2: a scheduling service for internal meeting rooms. Last two blocks blank. Given: 5,000 employees, 300 rooms, roughly 4,000 bookings per day, availability checks roughly 40,000 per day.

• Block A, convert to rates: 4,000 bookings per day is 0.05 writes per second; 40,000 checks per day is 0.5 reads per second, with a peak near the top of each hour, so assume 20x concentration giving 10 per second. 300 rooms times a year of slots is a small table.
• Block B, smallest design: you write this block.
• Block C, triggers for what was not added: you write this block.

14f. Check yourself

Q1. An interviewer says "assume this needs to handle a million users". What do you ask before changing your design, and why does the question matter more than the number?

Q2. You have proposed a single-node design. The interviewer looks unconvinced and says "what if it goes down?" Give the answer that neither capitulates nor dismisses.

14g. When not to use this

Restraint as a default posture.

The single-node capacity assumptions in this module.

Removing a component from a running system just because it is under threshold.

Module 15. Brownfield design

Every paid case study starts from a blank page. Several real staff-plus rounds hand you a running system with users on it and ask what you would do next. The skills barely overlap: greenfield rewards the best end state, brownfield rewards the safest ordered path to it.

15a. Pre-question

+--------------------------------------------------------------+
| Running system, 3 years old:                                  |
|   one Postgres primary, 4 TB, 8,000 writes/s peak             |
|   one table, events, is 3.1 TB of the 4 TB                    |
|   p99 write latency rose from 12 ms to 210 ms over 6 months   |
|   replica lag peaks at 45 s during the nightly batch          |
|   14 services write to this database directly                 |
|   no downtime window; the product is used 24/7 globally       |
+--------------------------------------------------------------+
Caption: a running single-database system with its size, write rate,
latency trend, replica lag and the number of direct writers.

Question. Which single fact in that box most constrains the plan, and what does it rule out?

15b. Explanation in small steps

The ordering rule that governs everything. Make the new system correct before it is authoritative, and make it authoritative before it is exclusive. Every migration below is that rule expanded.

The migration ladder. Six rungs, in order. Skipping a rung is where outages come from.

Dual write is not atomic, and this is the fact candidates miss. Writing to two systems in one request is two independent failures. Three honest options:

The outbox works when you own the writers. CDC works when you do not. Naming which constraint you are under, and picking accordingly, is the senior move.

Backfill arithmetic. Backfill is bounded by the write capacity you are willing to spend on it, not by how fast you can read.

• Suppose the target can absorb 100,000 writes per second and you allocate 20% to backfill, so 20,000 per second.
• 6 billion rows divided by 20,000 is 300,000 seconds, that is 3.5 days of continuous running.
• If you may only backfill in a 6 hour nightly window: 6 times 3,600 times 20,000 is 432 M rows per night, so about 14 nights.

Two rules that fall out. First, backfill in key order with a checkpoint so a crash resumes rather than restarts. Second, the backfill must never overwrite a newer live write: either write with an "insert if not exists" semantic, or compare versions, or backfill only rows whose id is below the watermark you recorded when dual write started.

Shadow traffic and the acceptance gate. Shadow comparison without a gate is a science project. Define the gate before you start.

Blast radius, made explicit. For every step, write down three numbers before you take it.

The third one is the one people forget, and it is the reason cutover of writes is scheduled for a Tuesday morning and not a Friday evening.

Deprecation, and the one-client problem. Removing the old path is the step that never finishes. What actually works:

1. Instrument the old path per caller, so you know exactly who is left.
2. Publish the removal date and the migration guide at the same time.
3. Introduce scheduled brownouts: return errors for 5 minutes at a known time, then 30 minutes, then an hour. A brownout finds the callers who ignored the email and produces a list, not a surprise.
4. When one caller remains, go and do their migration for them. It is cheaper than another quarter of carrying the old path.

Buying time is a legitimate first move. Separating the cheap reversible relief from the expensive terminal change is the whole skill here.

15c. ASCII diagram

  STEP 1 SHADOW          STEP 2 DUAL WRITE      STEP 3 BACKFILL
  +--------+             +--------+             +--------+
  | CLIENT |             | CLIENT |             | CLIENT |
  +---+----+             +---+----+             +---+----+
      |                      |                      |
      v                      v                      v
  +--------+  copy    +--------------+       +--------------+
  | OLD    |------>   | OLD (auth.)  |       | OLD (auth.)  |
  | (auth.)|  compare |   + outbox   |       |   + outbox   |
  +--------+   async  +------+-------+       +------+-------+
      |                      | relay                | relay
      v                      v                      v
  +--------+             +--------+     +---------------------+
  | NEW    |             | NEW    |<----| BACKFILL, id < mark |
  | (dark) |             | (dark) |     | 20k rows/s, resumes |
  +--------+             +--------+     +---------------------+

  STEP 4 READ CUTOVER: 1% -> 10% -> 50% -> 100%, flag-controlled
  STEP 5 WRITE CUTOVER: new authoritative, old still written
  STEP 6 DECOMMISSION: brownouts, then freeze, then delete

Caption: the migration ladder drawn as three states plus three
scripted steps. The old system stays authoritative through step 4, so
everything up to that point is reversible by a flag.

15d. Worked example: the write hot spot, next two quarters

The prompt. "Here is the system in the pre-question box. You have two quarters. What do you do?"

Block A. Purpose: measure before moving, and refuse to plan without the breakdown

I do not accept "the database is slow" as a diagnosis. Three measurements first, and I name what each would change:

Suppose the answers come back: the events table is 78% of write time, index maintenance is over half of that (there are 6 indexes on events), and one telemetry service is 62% of the write rate.

That reshapes the whole plan. This is not a "we outgrew Postgres" problem; it is a "one service is writing telemetry into our transactional database with six indexes on it" problem.

The error most readers make here. Jumping straight to sharding or to a new data store. Sharding a table whose load comes 62% from one misplaced caller carries all the cost and fixes the wrong thing.

Block B. Purpose: buy time with the cheapest reversible change

Quarter one, first six weeks. Nothing here is a migration.

Expected result, stated as a prediction I can be held to: write p99 returns toward the 12 ms baseline and the system has roughly 2.5x headroom on statement rate. That buys the time to do the real work carefully instead of under pressure.

The error most readers make here. Treating quick wins as beneath a design answer. In a staff round, sequencing the reversible relief before the irreversible migration is the answer, and skipping it reads as an inability to prioritise.

Block C. Purpose: design the terminal state and pick the migration mechanism

Terminal state: events leaves the transactional database entirely and lands in a store designed for append-heavy, time-ordered data with a retention policy. The transactional database keeps the roughly 0.9 TB of actual business entities and returns to being small.

The mechanism is decided by one constraint from the pre-question: 14 direct writers, which I do not control.

CDC is the answer here specifically because of the coupling fact, and I say the causal link out loud, since the same problem with 2 writers I own would get the outbox.

Backfill sizing, using the module's method: 3.1 TB of events at, assume, 400 bytes per row is about 7.75 billion rows. If the new store absorbs 100,000 writes per second and I allocate 20% to backfill, that is 20,000 per second, so about 387,500 seconds, roughly 4.5 days of continuous backfill. I record the maximum event id at the moment CDC starts and backfill only below it, so live CDC writes and the backfill can never fight.

The error most readers make here. Choosing the mechanism from a preference about elegance rather than from the constraint. The outbox is the better pattern in the abstract and the wrong one here.

Block D. Purpose: sequence the two quarters with explicit gates

Two things I flag as risks with owners, because a plan without named risks is a wish:

• The events table's schema is used by ad hoc analytics queries nobody has inventoried. Mitigation: instrument reads for four weeks before week 18, and publish the caller list.
• The 4.5 day backfill will not run cleanly on the first attempt. Mitigation: the checkpointed, resumable design, plus a rehearsal on one day-partition in week 10 rather than discovering the problems in week 11.

Finally, what I am explicitly not doing in these two quarters: not sharding, not splitting the monolith, not introducing a second region. Each is defensible later, none is on the critical path of the stated problem, and saying which attractive work you are declining is a staff-level behaviour.

15e. Faded examples

Fade 1: move sessions from sticky in-memory state to a shared store. Last block blank.

• Block A, measure first: what fraction of requests currently benefit from stickiness, what is the session size distribution, and how often does a node restart evict live sessions. Suppose sessions average 4 KB, there are 2 M concurrent, and node restarts log out roughly 30,000 users per deploy.
• Block B, buy time cheaply: reduce deploy-time impact with connection draining and a rolling deploy, which cuts the 30,000 without changing architecture.
• Block C, terminal state and mechanism: a signed self-contained token with a small replicated revocation list, or a shared store holding 2 M times 4 KB, which is 8 GB and fits in memory on one node pair. Dual-read from both during transition: check the shared store first, fall back to the local session, and write only to the shared store.
• Block D, sequence with gates: you write this block.

Fade 2: split a shared orders table that three teams write to. Last two blocks blank.

• Block A, measure first: per-team write rate and query patterns, which columns each team actually reads, and how many queries join across the columns the teams would like to separate. Suppose 40% of queries join across the proposed boundary, which is the number that decides whether the split is viable at all.
• Block B, buy time cheaply: add the missing composite indexes and move reporting reads to a replica, buying enough headroom to plan properly.
• Block C, terminal state and mechanism: you write this block.
• Block D, sequence with gates: you write this block.

15f. Check yourself

Q1. Why is dual write from the application usually worse than change data capture, and name the one situation where the reverse is true.

Q2. You are at week 20 of the plan above, 100% of reads on the new store, and a defect appears that has been silently corrupting 0.3% of rows since the backfill. What do you do, and what does the plan's design give you?

15g. When not to use this

A full migration ladder.

Change data capture.

Shadow traffic comparison.

Module 16. Level calibration

What a staff answer adds is the question this page gets asked most, and the thing paid courses leave implicit. This module answers it as observable behaviour rather than as vocabulary, then makes you produce the three versions yourself.

16a. Pre-question

+--------------------------------------------------------------+
| Three answers to "how do we keep the feed fast?"              |
|                                                               |
| A: "Add a Redis cache in front of the feed query, with a      |
|     5 minute TTL, and shard by user id."                      |
|                                                               |
| B: "Feed p99 is 900 ms and 700 ms of that is the fan-in       |
|     query. A cache fixes it only if the hit rate exceeds      |
|     about 85%, which needs a 5 minute TTL. That makes the     |
|     feed 5 minutes stale, so I would confirm the freshness    |
|     requirement before choosing between the cache and         |
|     precomputing the feed on write."                          |
|                                                               |
| C: "Feed latency has three causes and only one is worth       |
|     fixing this half. I would spend the quarter on the        |
|     fan-in query, defer the cache, and put a freshness SLO    |
|     in place so the next person has a number to design        |
|     against instead of an opinion."                           |
+--------------------------------------------------------------+
Caption: three answers to the same question, differing in what they
do with the measurement rather than in which components they name.

Question. Rank A, B and C by level, and name the specific behaviour that separates each from the one below it.

16b. Explanation in small steps

What levels are not. They are not vocabulary, not the number of components, and not the obscurity of the technology named. A candidate who says "Raft" and cannot derive the write latency has shown less than one who says "one primary with failover" and states the RTO.

The behaviour table. Read this as a ladder where each column contains the one before it.

Two failure modes, symmetric and equally costly.

The rule: perform your level plus one, and only in the areas where you actually have depth. Signal the higher level by volunteering a trade-off and a threshold, not by widening the topic list.

The observable proxies you can self-score. Record yourself, then count.

These ranges are a self-calibration heuristic derived from the behaviour table above, not measured data from real loops. Use them to notice a pattern in your own recordings, not as a score.

What the level delta is not allowed to be. Adding boxes. If your staff version of an answer has more components than your senior version, you have probably confused seniority with complexity. In practice, staff answers often have fewer components and more constraints.

16c. ASCII diagram

                        +-------------------------+
   STAFF                | sequence over time,     |
                        | cost, org, what NOT to  |
                        | do, decision artefacts  |
                        +-----------+-------------+
                                    | contains
                        +-----------v-------------+
   SENIOR               | chooses between designs |
                        | volunteers trade-offs   |
                        | numbers that eliminate  |
                        +-----------+-------------+
                                    | contains
                        +-----------v-------------+
   MID                  | correct design from     |
                        | given requirements      |
                        +-------------------------+

   FAILURE: reaching the top box without the two below it reads as
   hand-waving, and is scored lower than a clean MID answer.

Caption: levels as containment, not as separate skills. Each level
includes everything below it, and skipping a layer is penalised.

16d. Worked example: "design a notification service", three times

The prompt. "Design a service that sends notifications to our users across push, email and SMS." Given: 20 M users, 50 M notifications per day, three channels, and a stated requirement that a user must never receive the same notification twice.

Block A. Purpose: fix the level signature before answering

I decide, before speaking, which behaviours I intend to show. For a senior target that means at least three unprompted trade-offs with thresholds, at least four eliminating numbers, and one deliberate descope. Naming the target to myself stops the answer drifting into a topic tour.

The base arithmetic, which every version shares:

• 50 M per day is 579 per second average. Assume a 5x peak concentration because notification traffic follows waking hours, giving about 2,900 per second.
• 2,900 per second is comfortably inside one queue and a modest worker fleet, so scale is not the interesting part of this problem. Deduplication and channel failure are.

That last sentence is the hinge, and every level uses it differently.

The error most readers make here. Treating 50 M per day as a big number. It is 579 per second. Candidates who fail to convert spend the round designing for scale that does not exist and never reach the actual difficulty.

Block B. Purpose: write the mid answer, and mark its ceiling

"Notifications are produced by services publishing events. I would put them on a queue, with a worker fleet consuming and dispatching to a channel adapter for push, email and SMS. Each notification gets an id, and I store delivered ids in a table with a unique constraint so a retry cannot deliver twice.

User channel preferences live in a preferences table the worker reads. Failed sends are retried with exponential backoff, and after five failures they go to a dead letter queue. I would shard the queue by user id so ordering per user is preserved."

This is a correct design. It satisfies the stated requirements, it names the deduplication mechanism, and the components are appropriate. Nothing in it is wrong.

Its ceiling: every number in the prompt was accepted without being used. It does not say why five retries rather than three, what the per-channel failure profile is, whether ordering per user is actually required, or what happens when the SMS provider is down for an hour. It answers the question asked.

Block C. Purpose: add the senior delta, and mark its ceiling

The senior version keeps everything above and adds four things. I mark each with the behaviour it demonstrates.

"At 579 per second average and 2,900 at peak, this is one queue and roughly 20 workers, so I will not spend time on scaling and will spend it on delivery semantics instead. [eliminating number, deliberate descope]

The three channels have different failure profiles, and that changes the design. Push is cheap and fails fast; SMS costs real money per message and its providers have minute-scale outages; email has delayed bounces that arrive hours later. So I would not have one worker pool: I would have one per channel, with its own concurrency limit and its own retry budget, so an SMS provider outage cannot consume the workers that deliver push. [unprompted trade-off]

On the never-twice requirement, the unique constraint stops our duplicates but not the provider's. If we time out on an SMS send, we do not know whether it went.

I would pass our notification id to the provider as its idempotency key where supported, and where it is not, decide per channel: for push, resend, because a duplicate is nearly free; for SMS, do not resend, because a duplicate costs money and annoys the user more than a miss. That is a product decision and I would confirm it. [trade-off with a threshold, and a question for the requirement owner]

Ordering per user is worth questioning. Sharding the queue by user id costs us head-of-line blocking: one user's stuck SMS delays their push. If ordering is not actually required, I would drop the sharding and gain isolation. [challenging a choice with the cost that decides it]"

Its ceiling: it is still answering "what is the right design". It does not say what gets built first, what it costs, who operates it, or what happens in the second quarter.

The error most readers make here. Adding the trade-offs as a list at the end rather than at the decision points. A trade-off stated where the decision is made reads as reasoning; the same trade-off recited at the end reads as a checklist.

Block D. Purpose: add the staff delta, and name what it costs

The staff version contains the senior one and adds sequence, cost, operational contract and refusal.

"Before the design, one reframe. The requirement is never send the same notification twice, but the failure users complain about is usually notification volume, not duplication. I would want to know whether the actual pain is duplicates or a lack of batching and rate limiting per user, because those are different systems and one of them is much cheaper. Assuming duplication is genuinely the requirement, here is the plan. [reframing with evidence, then proceeding rather than stalling]

Cost first, because it decides the shape. At 50 M per day, if 5% go to SMS at, assume, $0.005 per message, that is 2.5 M messages, so $12,500 per day, roughly $375,000 a month. Push and email together are a rounding error against that.

So the expensive component in this system is not any box on my diagram, it is the SMS channel, and the highest-value engineering is anything that reduces unnecessary SMS: per-user batching, channel fallback ordering, and a preference default that prefers push when the app is installed. I would build that in quarter one. [priced trade-off, and it changes what gets built]

Operational contract. One SLI per channel: fraction of notifications delivered within their channel's target, which is 30 seconds for push, 5 minutes for email, 60 seconds for SMS. Fast-burn page at 14.4 times the budget over an hour. The one alert that actually matters is per-channel queue age, because that predicts a breach before it happens. On call is one rotation, and a provider outage must degrade to another channel automatically rather than paging a human. [operational design as an output]

Sequence. Quarter one: single worker pool, push and email only, the deduplication table, and the cost controls. SMS is deliberately last because it is the expensive channel and I want the batching working before it is switched on. Quarter two: per-channel pools, provider fallback, delayed bounce handling. [sequence with a reason]

What I am not doing: no multi-region, no separate template service, no user-facing notification centre. Each is defensible and none is on the critical path of the stated problem. If the notification centre matters more than deduplication, tell me and I will re-plan. [refusal with an offer]"

Notice what the staff version did not do: it added no components. It removed one (the per-channel pools moved to quarter two), added constraints and a sequence, and priced the thing nobody had priced.

The error most readers make here. Performing the staff answer without the mid answer underneath. If asked how deduplication works and you cannot describe the unique constraint, the reframing and the cost model read as evasion.

16e. Faded examples

Fade 1: level up an answer to "design a URL shortener". Last block blank.

• Block A, level signature and base arithmetic: 100 M new links per year is 3.2 per second, and 10,000 redirects per second is the read side. Reads exceed writes by roughly 3,000 to 1, which is the number the whole design hangs on.
• Block B, mid answer and its ceiling: base62 encoding of a counter or a hash, a key-value store mapping code to URL, a cache in front, a 301 redirect. Correct. Ceiling: does not say why base62 over a hash, does not size the keyspace, does not mention that 301 is cached by browsers forever and therefore breaks analytics and link editing.
• Block C, senior delta and its ceiling: 7 characters of base62 is 62 to the 7^th, about 3.5 trillion codes, which at 100 M per year lasts far longer than the product will. Choose 302 over 301 specifically because the analytics requirement needs each click to reach us, and name the cost, which is one extra round trip per click forever. Volunteer the custom-alias collision path. Ceiling: still a design, not a plan.
• Block D, staff delta and what it costs: you write this block.

Fade 2: level up an answer to "how would you monitor this system?" Last two blocks blank.

• Block A, level signature and base arithmetic: the honest observation is that a tool list is the path of least resistance here, so the differentiator is deriving the alerts from an SLO rather than naming dashboards.
• Block B, mid answer and its ceiling: metrics for rate, errors and duration; logs; traces; dashboards per service; alert on error rate and on high latency. Correct and complete as a list. Ceiling: no thresholds derived from anything, and no user journey named.
• Block C, senior delta and its ceiling: you write this block.
• Block D, staff delta and what it costs: you write this block.

16f. Check yourself

Q1. A candidate's answer contains Raft, CRDTs, multi-region active-active and a service mesh. The interviewer scores it below a candidate who used one database and one queue. Give two reasons that is a correct scoring decision.

Q2. You are interviewing at senior level and you finish your design with 12 minutes left. Name the highest-value thing to do with those minutes, and the low-value thing that fills them by default.

16g. When not to use this

Performing above your interviewing level.

The self-scoring proxy counts.

Level deltas as a template to recite.

Case study 1: URL shortener

Most published treatments give this prompt one lesson, produce a base62 encoder, and stop. The interesting part is the key space, and that is exactly the part the short version skips. This case study spends its depth there.

The prompt (shortener)

"Design a service that turns a long URL into a short link, and sends anyone who visits the short link to the original."

Clarifiers that fork the design (shortener)

Ask these because each answer changes a box on the whiteboard. Do not recite them.

Requirements (shortener)

Functional:

• Create a short code for a submitted URL, and return it.
• Resolve a short code to its URL and redirect.
• Record a click event per resolution.
• Support optional user-chosen aliases.

Non-functional:

Why the distribution is a stated requirement and not a footnote. Cache sizing is the only decision here that depends on the shape of the traffic rather than its volume, and the shape changes the answer by a factor of three.

Writing the exponent down as an assumption makes the later arithmetic checkable, and gives you one sentence to say in the room: "I am assuming s equals 1, which is the pessimistic end; if your measured hit ratio beats my derived number, s is above 1 and everything downstream gets cheaper."

Numbers that eliminate options (shortener)

Every line below ends with the option it kills. If a number kills nothing, it does not belong in the answer.

Two byte sizes, stated once so the rest of the page agrees with itself. They are different quantities and the difference is the whole reason a cache is cheaper than a database.

The 100 byte average URL is the assumption to challenge first, because it moves both numbers proportionally. It is reasonable because a shortener's input is overwhelmingly campaign links with query parameters, not bare domains, and you can measure it from one day of create traffic.

The hit-ratio formula, derived once and reused everywhere below. Under Zipf with exponent s = 1 over N items, the probability of rank i is proportional to 1/i, so the share of traffic taken by the top K items is the ratio of two harmonic numbers, H(K) over H(N), and H(n) is approximately ln n plus 0.577. That gives a cache of K entries a hit ratio of (ln K + 0.577) divided by (ln N + 0.577). Three consequences worth saying out loud:

• The hit ratio grows with the logarithm of cache size, so a tenfold larger cache buys ln 10 divided by H(N), which is 2.303 divided by 24.2, or 9.5 percentage points. Not more.
• Reaching 90 percent needs K such that ln K + 0.577 is 0.9 times 24.2, which is K of about 1.6 billion entries, or 240 GB at 150 bytes each. That is a real option, but it is a decision with a price, not a rounding assumption.
• This formula describes a perfect cache holding exactly the K most popular codes. LRU under the same distribution lands a few points below it, so treat every hit ratio derived here as an optimistic bound and hold budget back accordingly.

Why s = 1 and not something friendlier. s = 1 is the pessimistic end of the plausible range for link popularity, and pessimistic is the right default when the number decides how much hardware you buy. The falsifier is direct: instrument the production cache, plot hit ratio against cache size on a log x-axis, and read the slope. A slope steeper than 9.5 points per decade means s is above 1 and every number below improves.

V0: the smallest design that meets the requirements (shortener)

+-----------+      +------------------+      +----------------+
| Browser   | ---> | App server       | ---> | Postgres       |
|           | <--- | in-process LRU   | <--- | links table    |
+-----------+      | 200k entries     |      | code PK        |
                   +------------------+      | url longtext   |
                        |                    +----------------+
                        v
                   +------------------+
                   | Click log file   |
                   | appended locally |
                   +------------------+

Caption. V0 has three components. The browser calls a stateless app server that keeps a small in-process cache; on a miss it does a primary-key lookup in one Postgres instance; click events are appended to a local log file and shipped out of band.

Why this is correct at the stated scale:

• Writes are 350 per second at peak. A single Postgres primary with one unique index does this without effort.
• Reads are 35,000 per second at peak. Apply the formula above to the 200,000-entry in-process cache: (ln 200,000 + 0.577) divided by (ln 1.825e10 + 0.577) is 12.78 divided by 24.20, which is 52.8 percent. So the cache absorbs about half the reads, not most of them, and the primary sees 35,000 times 0.472, which is 16,500 primary-key lookups per second.
• Whether one primary survives 16,500 point lookups per second is the load-bearing check, and it needs two ceilings rather than one.
• Storage ceiling: ASSUME the code index is a four-level B-tree whose upper levels stay resident, so a miss costs one random read, and ASSUME the NVMe volume sustains 60,000 random reads per second. 16,500 is 28 percent of that, so storage is not the binding resource.
• CPU ceiling: ASSUME 0.5 ms of CPU per lookup across connection handling, parse, plan, execute and result encoding, which is deliberately pessimistic for a primary-key read. A 16 core primary held at 60 percent utilisation gives 9.6 core-seconds per second, so the ceiling is 9.6 divided by 0.0005, about 19,200 lookups per second. CPU binds first.
• So V0 meets the requirement with 16,500 against a ceiling of 19,200, which is 14 percent of headroom. It fits, and it fits narrowly. Both ceilings are load-test questions rather than facts, and I would say that out loud before drawing the next box.
• Short codes come from a Postgres sequence, passed through a keyed pseudorandom permutation so the issue order carries no information. Details in the first deep dive.

Breaking points, and what each version adds (shortener)

Breaking point one: the deploy cliff, at any traffic above roughly 19,000 reads per second.

The threshold is the primary's derived ceiling, because a cold cache turns every read into a lookup. Above 19,200 reads per second the primary is over its CPU ceiling for as long as the caches stay cold, and the stated peak of 35,000 is 1.8 times over it.

What fails: the in-process cache is per instance and per process lifetime. A rolling deploy replaces every instance, every cache starts empty, and for the next thirty seconds the full 35,000 reads per second lands on the primary. Connection pool wait time goes vertical, and redirect p99 goes from 8 ms to multiple seconds.

How you observe it: cache hit ratio plotted against deploy markers, and database connection pool wait time at p99. If those two lines are correlated with your release timeline, you have found it. This is a thundering herd, not a capacity problem, and adding database CPU does not fix it.

+---------+   +-------------+   +-----------+   +-------------+
| Browser |-->| App server  |-->| Shared    |-->| Postgres    |
|         |   | local LRU   |   | cache     |   | primary     |
+---------+   | 1 sec TTL   |   | 15 GB     |   +-------------+
              +-------------+   +-----------+          |
                     |                                 v
                     v                          +-------------+
              +-------------+                   | Read        |
              | Click       |                   | replicas    |
              | event queue |                   +-------------+
              +-------------+

Caption. V1 keeps the tiny local cache but puts a shared cache tier behind it, moves reads to replicas, and moves click events onto a queue instead of a local file.

What the shared tier is worth, derived. It holds the hottest 100 million codes at 150 bytes each, so 15 GB. Because it is a superset of the local caches, the combined hit ratio is just its own: (ln 1e8 + 0.577) divided by 24.20, which is 19.0 divided by 24.20, or 78.5 percent.

The primary then sees 35,000 times 0.215, which is 7,500 lookups per second against the 19,200 ceiling, so headroom goes from 14 percent to 61 percent. That is the argument for the tier, and it is a better one than the deploy cliff alone.

If you want more, the same formula prices it: 1 billion entries is 150 GB and buys 88 percent, leaving 4,200 lookups per second. Each decade of cache costs ten times the memory and returns 9.5 points. Say that ratio out loud and the question of how big the cache should be stops being a matter of taste.

V1 changes, each with the observation that justified it:

Breaking point two: 18 billion rows and 9.1 TB, reached in year five.

What fails: index maintenance and operational recovery, not query latency. A 9 TB single-instance database has a restore time measured in many hours, and your 99.99 percent monthly availability budget is 4.3 minutes. One restore blows a year of budget.

How you observe it: track time-to-restore from your most recent backup as a monthly drill, and plot it against your error budget. Also watch write latency p99 creeping upward as the index depth grows.

V2 splits the store by responsibility:

• The code-to-URL mapping moves to a hash-partitioned key-value store keyed by short code. It is a pure point-lookup workload, it never needs a join, and hash partitioning by code gives even distribution for free.
• The relational database keeps only user-owned metadata: who created a link, campaign tags, custom alias ownership. That is a small fraction of the rows.
• Click events land in a columnar analytics store via the queue, partitioned by day, with a retention policy.

Deep dive one: generating the key

This is the dive that the one-lesson version skips, and it is where a senior answer separates from a recited one.

The candidate strategies, compared on what actually matters.

Why hash-and-truncate fails, with the arithmetic. With a key space of 62 to the 7 equals 3.52e12 and 1.825e10 keys in use, the expected number of colliding pairs is roughly n squared divided by 2N, which is (1.825e10 squared) divided by (2 times 3.52e12), or about 47 million collisions. Truncated hashing is not collision-free at this scale, so you need a uniqueness check on insert regardless. Once you have that check, the hash bought you nothing.

The design I would defend. A single allocator row hands out blocks of 100,000 consecutive counter values. Each app instance holds a block in memory and consumes it locally. Fleet-wide at 350 writes per second, the allocator sees one allocation every 285 seconds, so it is nowhere near a bottleneck and a brief allocator outage is survivable because instances still hold unconsumed blocks.

The raw counter is then passed through a permutation of the 7-character space before anyone sees it, so that the issue order carries no information. Choosing that permutation is where this dive earns its place, because the version people reach for first does not work.

The wrong answer, and the exact reason it is wrong. The common suggestion is to multiply the counter by a fixed odd constant modulo 62 to the 7. Write it as code(n) = a times n, modulo M, with M equal to 62 to the 7 and a chosen coprime to M. It is a bijection, so it never collides, and to a human two consecutive codes do look unrelated. It hides nothing from a machine. The map is affine, so:

• code(n + 1) minus code(n) equals a, modulo M, for every n without exception. The difference is a constant, not a scramble.
• An attacker creates two links back to back, subtracts one code from the other, and now holds a. From any known code, repeatedly adding a walks the entire issued range in order.
• Adding an offset does not help. For code(n) = a times n plus b, the difference is still a. Two consecutive codes recover a, and a third recovers b.
• Inverting the map needs only the modular inverse of a, which is one line of arithmetic. Reversible was never the hard part.

The general statement is the one to carry into any interview: an affine map preserves differences and is trivially invertible, so it obscures the output only from an observer who is not looking. Unpredictability is a property of keyed functions, not of bijections.

The property you actually need. The map must be a keyed pseudorandom permutation over the code space: given any number of counter-to-code pairs, an attacker without the key must not be able to predict the code for a counter they have not seen. Three ways to get that, priced honestly.

The caveat that survives the fix. A correct pseudorandom permutation raises the cost of discovery. It is not access control. A short code is a bearer token, so anyone who obtains it can use it, and confidential destinations need an authorisation check on resolve regardless of how the code was made.

Four practical notes follow:

• Because the issued code is stored in the row, you never need to invert the permutation, so you can rotate the key whenever you like and only future codes change.
• The space is dense, which brute force exploits. 1.825e10 issued codes in 3.52e12 slots is 0.52 percent occupancy, so a random 7-character probe lands on a live link about once every 192 tries. A keyed permutation stops arithmetic enumeration. It does nothing about guessing.
• If unguessability is a real requirement rather than a preference, buy it with length, because each extra character divides the hit rate by 62. At 8 characters the same 1.825e10 codes occupy 0.0084 percent of the space, so a probe succeeds about once every 11,900 tries. That is the trade the clarifier table flagged: unguessable costs a character.
• Rate limiting and negative caching are needed in all three options above, which means the third option is not as far from the first as it looks.

What each property buys you:

Keep the unique index even though the counter guarantees uniqueness. If the allocator ever splits its brain and hands the same block to two instances, the index turns a silent data-corruption incident into a visible insert error. The index costs you nothing on a workload of 350 inserts per second, and it is the difference between a page and a postmortem.

The blocklist. A 7-character base62 space will generate offensive strings. Filter generated codes against a blocklist before issuing them, and skip forward when one hits. This is a real product requirement, and it is the kind of detail that only shows up once you have shipped one.

Deep dive two: the read path under a Zipf distribution

The dives on this page vary by problem. Here the read path is interesting because a single link can go viral, which is a hot-key problem that does not arise in the crawler or the chat system.

What the distribution actually buys, at each tier. All three lines use the same formula from the sizing section, hit ratio equals (ln K + 0.577) divided by (ln 1.825e10 + 0.577), with the denominator 24.20.

The shape of that table is the point, and it is the sentence to say in the room: under s = 1 the hit ratio is logarithmic in cache size, so memory buys 9.5 points per decade and no affordable cache reaches 95 percent.

If your production hit ratio is far above the derived line, your traffic is more skewed than s = 1, and you should re-derive with the measured exponent rather than keep a lucky number.

Sensitivity runs the other way too. At s = 1.2 the same 200,000-entry local cache would take about 92 percent, so one tenth on the exponent is worth more than a hundredfold of memory.

Hot key. One link in a widely shared post can take most of your 35,000 reads per second. In a hash-partitioned cache, one key lives on one node, so that node absorbs the whole spike while its peers idle.

The fix is the small local cache with a short time to live, and the number is easy to derive. With 200 app instances and a 1 second local time to live, the shared cache sees at most 200 requests per second for that key regardless of how popular the link becomes, because each instance can only miss once per second. User traffic can grow by a factor of a thousand and the shared cache load for that key does not move.

Cache stampede. When the shared cache entry expires, every instance misses at the same moment and they all query the database together. Two mitigations, and you should name the trade-off:

Facebook's team described the lease approach for exactly this problem in Scaling Memcache at Facebook (Nishtala and others, USENIX NSDI 2013, accessed August 2026), which is worth reading as the primary source rather than a summary.

Negative lookups. Scanners probe random 7-character codes. Every one is a cache miss followed by a database read that finds nothing. At even a modest probe rate this can exceed your legitimate traffic. Two answers:

• Cache the negative result with a short time to live, say 30 seconds. Cheap, effective, and bounded.
• Put a Bloom filter of all issued codes in front. With 1.825e10 codes at a 1 percent false positive rate, the filter needs about 9.6 bits per element, which is 21.9 GB. That is larger than your hot cache, for a problem that negative caching already solves. Bloom filters (Bloom, Communications of the ACM, 1970) are the right tool when the negative set is unbounded and the positive set is small, which is the opposite of this situation.

301 versus 302, which is a traffic decision and not a trivia question. A 301 is a permanent redirect and browsers cache it, so repeat clicks never reach you. A 302 or 307 keeps every click on your servers. The semantics are defined in RFC 9110 (June 2022).

If analytics is a product feature, you must use 302 or 307, and that decision is what produced your 35,000 reads per second in the first place. If it is not, 301 can remove most of your read traffic, and it also removes your ability to ever disable a malicious link.

Failure modes and the operating contract (shortener)

Service level indicators to publish:

• Redirect success rate, measured as non-5xx responses on the resolve path.
• Redirect server-side latency at p50 and p99.
• Shared cache hit ratio, and local cache hit ratio, plotted separately.
• Create success rate, and key block allocation latency.
• Click event pipeline lag.

Alert on error budget burn rate for redirect availability, not on raw error count. A 4.3 minute monthly budget means a 2 percent error rate for 10 minutes is an incident and a 0.001 percent rate all month is not.

Re-checking against the non-functional requirements:

Where this answer is a simplification (shortener)

• Abuse is the actual hard problem. A shortener is a phishing tool by default. Real services check destinations against reputation feeds at create time and re-check them later, because a link can be created benign and repointed. That means the mapping is mutable, which invalidates the "write-once" simplification the whole design leans on.
• Custom domains multiply everything. Customers want links on their own domain, which means per-domain certificate issuance and renewal, and it turns your single hostname into thousands. The ACME protocol (RFC 8555, March 2019) is the mechanism.
• Bots inflate analytics. Link preview crawlers from chat and social apps fetch every link that gets pasted, often several times. Your click counts are wrong unless you filter them, and filtering them is an ongoing arms race.
• Deletion is a legal requirement, not a feature. Click logs contain IP addresses. Retention policy and deletion requests are design inputs.
• Further reading with dates. Instagram's engineering post on sharding and ID generation (2012) is the clearest short primary source on ID design trade-offs; search for it by title, since the post has moved hosts. Twitter's archived Snowflake repository (first released 2010, repository archived September 2021, accessed August 2026) documents the time-ordered alternative. RFC 9110 (June 2022) is the authority on redirect semantics.

Level delta: the same prompt at three levels (shortener)

What each level added, in one line each:

• Mid to senior: every number now eliminates something, and every box now has a measurement behind it.
• Senior to staff: the operational consequences enter the design (restore time against error budget, corruption tripwires, load shedding), and at least one component gets removed on evidence.

Case study 2: Web crawler

The prompt (crawler)

"Design a web crawler that fetches a large portion of the public web and stores the pages for a search index."

Clarifiers that fork the design (crawler)

Requirements (crawler)

Functional:

• Fetch pages starting from a seed list, and discover new URLs from fetched pages.
• Respect robots exclusion rules and per-host rate limits.
• Store fetched content and its metadata.
• Avoid fetching the same content twice, whether the duplication is by URL or by content.

Non-functional:

Numbers that eliminate options (crawler)

V0: the smallest design that meets the requirements (crawler)

+-----------+     +----------------------+     +--------------+
| Seed list | --> | Crawler process      | --> | Object store |
+-----------+     | frontier by host     |     | page bodies  |
                  | seen-set in memory   |     +--------------+
                  | robots cache         |
                  | fetch parse extract  |
                  | discovered URLs feed |
                  | back into own queue  |
                  +----------------------+

Caption. V0 is a single process holding a per-host frontier, an in-memory seen-set and a robots cache; it fetches, parses, writes bodies to an object store, and feeds discovered URLs back into its own frontier.

Why this is correct at the milestone:

• 16.5 pages per second times 0.5 second latency is about 9 requests in flight. A single process handles that without breaking a sweat.
• The seen-set at 1e8 URLs times 8 bytes is 800 MB, which fits in memory.
• Storage at 1e7 pages times 25 KB is 250 GB, which is one bucket and no partitioning decision.
• Politeness is trivially correct because one process owns every host, so there is no coordination question at all.

That last point is the one to say out loud. Politeness is a global constraint, and V0 satisfies it for free by having a single owner. Every later version has to work to keep that property.

Breaking points, and what each version adds (crawler)

Breaking point one: state size, at roughly 100 million pages crawled.

What fails: the in-memory seen-set crosses available RAM. You observe it as resident set size climbing linearly with pages crawled, then the page fault rate spiking and the crawl rate collapsing. A restart at this point loses all frontier state, and re-deriving it means re-crawling.

V1 shards by registrable domain:

+--------------------+   +--------------------+   +------------+
| Router             |-->| Fetcher shard 1    |-->| Object     |
| hashes registrable |   | owns those hosts   |   | store      |
| domain to a shard  |   | frontier on disk   |   | page       |
| takes back every   |   | bloom plus KV seen |   | bodies     |
| discovered URL the |   +--------------------+   +------------+
| shards emit        |   +--------------------+
+--------------------+-->| Fetcher shard N    |
                         | same structure     |
                         +--------------------+

Caption. V1 routes every discovered URL to the shard that owns its registrable domain; each shard keeps its own disk-backed frontier and seen-set, and all shards write bodies to a shared object store.

The critical design choice: shard by registrable domain, not by hostname and not by full URL.

That property, keeping politeness node-local, is what makes the shard boundary correct. The IRLbot work ("IRLbot: Scaling to 6 Billion Pages and Beyond", WWW 2008) documents the subdomain explosion problem in detail and is the primary source worth citing here.

Breaking point two: the frontier stops being about capacity and starts being about quality, at roughly 1 billion discovered URLs.

What fails: a small number of hosts consume a large fraction of the frontier. Infinite calendars, session identifiers in query strings, and faceted-search parameter combinations generate unbounded URL spaces. You observe it as a pages-per-host distribution where the top host holds a double-digit percentage of pending URLs, and a rising content-duplicate rate.

V2 adds three things:

• A URL canonicaliser applied before the seen-set check, so parameter noise collapses.
• A per-domain budget: a cap on pending URLs and on pages per crawl cycle, so no single domain can starve the frontier.
• A near-duplicate detector, covered in deep dive two.

Deep dive one: politeness and frontier scheduling

Two constraints pull against each other. Politeness says wait between requests to the same host. Throughput says never leave a fetch slot idle. Frontier design is the resolution.

The scheduling invariant, with the number. To keep F fetch slots busy with a per-host delay of d seconds and a mean fetch latency of L seconds, the number of hosts you must be actively tracking is at least F times d divided by L. With F equal to 200, d equal to 2 seconds and L equal to 0.5 seconds, that is 800 hosts. If your frontier holds URLs from fewer than 800 ready hosts, you cannot hit 386 pages per second no matter how much hardware you add.

The structure that satisfies it.

+--------------------+       +---------------------------+
| Priority intake    | ====> | Per host queues           |
| new and recrawl    |       | one FIFO per active host  |
+--------------------+       +---------------------------+
                                        |
                                        v
                             +---------------------------+
                             | Ready heap keyed by       |
                             | next allowed fetch time   |
                             +---------------------------+
                                        |
                                        v
                             +---------------------------+
                             | Fetcher slot pool         |
                             | 200 concurrent            |
                             +---------------------------+

Caption. URLs enter a priority intake, are placed into a queue per active host, and each host queue registers in a heap ordered by the earliest time that host may next be fetched; fetchers pull whichever host is due.

This two-level shape (a priority-ordered intake feeding per-host queues) is the structure described in the Mercator paper (Heydon and Najork, 1999). It is worth naming the source rather than presenting it as novel, and it is worth noting the critique: Mercator's fixed queue count assumes a roughly stable set of active hosts, which is a poor fit for a crawl where host popularity shifts hourly. A dynamically sized host-queue map with eviction of idle hosts is the modern adaptation.

Adaptive politeness beats a fixed delay. A fixed 2 second delay is simultaneously too fast for a struggling shared-hosting server and too slow for a large site that would happily serve you fifty requests per second. Set the delay as a multiple of the observed response time:

The multiplier is a single tunable, and the mechanism is self-correcting: a host you are hurting slows down, which slows you down further. That feedback loop is the whole point.

robots.txt handling, precisely. RFC 9309 (September 2022) standardised the exclusion protocol. Two operational points that are easy to miss:

• The 24 hour cache ceiling means robots fetches are themselves a traffic source. At 1 million active domains, that is 1 million fetches per day, which is 11.6 per second, about 3 percent of your crawl budget. Derivable, and worth stating.
• A robots fetch that fails with a server error is not permission to crawl. Treat 5xx as "disallow, retry later"; treat 404 as "allow". Getting this backwards is how crawlers earn bans.
• Crawl-delay is not part of RFC 9309. If you honour it, say that you are honouring a non-standard extension.

DNS is a real bottleneck. At 386 pages per second across hundreds of distinct hosts, uncached DNS resolution becomes a per-request dependency with variable latency. Run a caching resolver inside the crawler, honour record TTLs, and floor very short TTLs so a misconfigured domain cannot force a lookup per fetch.

Deep dive two: deduplication at three levels

The dives differ from the URL shortener's deliberately: there the interesting read-path property was a hot key, here it is that most of your discovered work is work you have already done. Duplicate detection is the crawler's dominant cost.

Level one: URL identity. Canonicalise before hashing. Lowercase the scheme and host, drop the default port, remove the fragment, resolve dot segments, sort query parameters, and strip known tracking parameters. Then hash the result. Skipping canonicalisation multiplies your seen-set and your frontier by the number of parameter permutations a site emits.

The seen-set decision, and the counterintuitive part. The instinct is a Bloom filter. Size it and see:

Going from 1 percent to 0.1 percent costs 6 GB, which is cheap. But look at what a false positive means here: the filter says "seen" when the URL is new, so you silently never crawl that page. At 1 percent over 2 billion genuinely new URLs, that is 20 million pages you will never index and never notice.

Now the part that is usually stated backwards. A Bloom filter saves disk lookups only for items that are absent. On the web, most discovered links point to pages you have already seen, so most checks return "maybe present" and still require the exact lookup. If 80 percent of your 1e10 checks are for already-seen URLs, the filter eliminates lookups for the other 20 percent and leaves 8 billion disk reads intact. The usual pitch for a Bloom filter does not apply to this workload.

The design that actually works: batch and merge. Buffer discovered URL hashes, and once per interval sort them and merge against the sorted on-disk seen-set.

The merge turns a random-I/O problem into a sequential one, and the cost of that is latency: a discovered URL is not known to be new until the next merge runs. For a crawler that is a perfectly acceptable trade, and saying so is the answer.

Level two: exact content identity. Hash the normalised body after stripping whitespace and boilerplate. At 1e9 pages times 8 bytes, the content-hash set is 8 GB and fits in memory. This catches mirrors, and it catches the duplicate URLs that canonicalisation missed, which is most of them.

Level three: near duplicates. Templated pages differ in a byline or a timestamp and are otherwise identical. Exact hashing misses all of them. The standard method is a 64-bit simhash with a Hamming distance threshold, described in "Detecting Near-Duplicates for Web Crawling" (Manku, Jain and Das Sarma, WWW 2007).

The retrieval trick is a pigeonhole argument you can derive in the room. Split the 64-bit fingerprint into 4 blocks of 16 bits. If two fingerprints differ in at most 3 bits, those 3 bits cannot cover all 4 blocks, so at least one block matches exactly. Build 4 tables, each keyed on a different block, and a candidate lookup is 4 exact lookups.

Cost: 4 tables times 1e9 entries times 16 bytes per entry equals 64 GB. That is affordable, and the derivation is what makes it a senior answer rather than a recalled fact.

Failure modes and the operating contract (crawler)

Service level indicators:

• Pages fetched per second, against the target rate.
• Fetch outcome distribution by status class, watched for shifts rather than absolute values.
• Politeness violations per host per minute. This is the indicator that protects your ability to keep crawling at all, and its target is zero.
• Fetcher slot idle time while the frontier is non-empty, which is the direct signal that host diversity is too low.
• Frontier depth and its growth rate versus drain rate.
• Duplicate rate at each of the three levels, tracked separately.

Alert on any non-zero politeness violation, on frontier growth exceeding drain for a sustained window, and on fetch success rate dropping by more than a set margin from its trailing baseline.

Re-check against the requirements:

Where this answer is a simplification (crawler)

• JavaScript rendering changes the cost model completely. A headless browser render costs roughly two orders of magnitude more CPU than an HTTP fetch and parse. If a meaningful share of your target corpus requires rendering, the capacity plan is a rendering plan, and the fetch tier stops being interesting.
• Sitemaps are a better discovery channel than link following for large sites, and they carry change hints. The sitemaps protocol at sitemaps.org (0.90, in use since 2005) is the reference.
• Crawl budget is a published concept on the receiving side. Google documents crawl capacity limit and crawl demand for large sites in Optimize your crawl budget (Google Search Central, page dated July 2026, accessed August 2026), which is useful for understanding what site operators expect from a crawler.
• The legal surface is real. Terms of service, jurisdiction, and the wave of AI-crawler restrictions mean "the public web" is a shrinking and contested set. Cloudflare wrote on 1 July 2025 that it was changing its default to block AI crawlers that do not pay for content (Content Independence Day: no AI crawl without compensation, accessed August 2026), which materially changes what a new crawler can reach.
• You may not need to crawl at all. Common Crawl publishes monthly archives of billions of pages under an open licence. For many products, consuming that corpus is the correct engineering answer, and proposing it shows judgement rather than laziness.
• Primary sources worth reading: the Mercator paper (1999) for frontier architecture, IRLbot (WWW 2008) for domain-level scaling and trap resistance, Manku et al. (WWW 2007) for near-duplicate detection, and RFC 9309 (September 2022) for exclusion semantics.

Level delta: the same prompt at three levels (crawler)

What each level added:

• Mid to senior: state size replaces throughput as the reason for every component, and politeness becomes a derived constraint rather than a policy footnote.
• Senior to staff: the design is evaluated for its effect on systems it does not own, and the build-versus-consume question is asked before the architecture.

Case study 3: News feed

The prompt (feed)

"Design the home timeline for a social network, so that a user opening the app sees recent posts from the accounts they follow."

Clarifiers that fork the design (feed)

Requirements (feed)

Functional:

• Return a page of recent posts from accounts the user follows.
• Support paging backward through the feed without duplicates or gaps.
• Reflect a new post from a followed account within a stated freshness window.
• Stop showing posts from an account after an unfollow.

Non-functional:

Numbers that eliminate options (feed)

V0: the smallest design that meets the requirements (feed)

+---------+     +--------------+     +----------------------+
| Client  | --> | Feed service | --> | Posts table          |
+---------+     |              |     | index author id time |
                |              |     +----------------------+
                |              |     +----------------------+
                |              | --> | Follows table        |
                +--------------+     | index follower id    |
                                     +----------------------+

Caption. V0 reads the caller's followee list, then runs one indexed range query per followee merged by time, with no precomputed feed state anywhere.

Why this is correct at the stated smaller scale:

• No materialised state means no fan-out lag, no repair jobs, and no divergence between the feed and the truth.
• An unfollow takes effect instantly and for free.
• A deleted post disappears everywhere immediately, satisfying the strict-privacy branch of the clarifier table without any extra machinery.
• Storage is one copy of each post.

Breaking points, and what each version adds (feed)

Breaking point one, then V1 (feed)

What fails: 17,400 reads per second times 200 followees is 3.48 million index probes per second.

How you observe it: plot feed read latency at p99 against the user's followee-count percentile. Pull degrades linearly with followee count, so the curve is a straight line upward, and the users at the ninetieth percentile of followee count are the ones filing complaints. That shape is the signature of a fan-in problem and it distinguishes this from a general capacity shortage.

V1 pushes at write time:

+---------+  +--------------+  +-----------------+
| Author  |->| Post service |->| Posts store     |
+---------+  +--------------+  +-----------------+
                    |
                    v
             +--------------+     +-----------------+
             | Fanout queue |---->| Fanout workers  |
             +--------------+     +-----------------+
                                          |
                                          v
                                  +-----------------+
+---------+   +--------------+    | Feed lists      |
| Reader  |-->| Feed service |--->| one per user    |
+---------+   | hydrate ids  |    +-----------------+
              +--------------+

Caption. V1 writes each post once to the posts store, then a queue of fan-out workers appends the post identifier to one materialised list per active follower; readers do one range read on their own list and hydrate identifiers into content.

A feed read is now one range read plus a batched hydration, independent of followee count. That is the whole point: the cost moved from a variable the reader controls to a variable the author controls.

Breaking point two, then V2 (feed)

What fails: the account with 50 million followers, at 42 minutes of fan-out.

How you observe it: fan-out job duration is bimodal, with a tight cluster under a second and a sparse tail in the thousands of seconds. Feed freshness measured at the follower level shows a systematic bias where followers written late in the fan-out see the post minutes after followers written early. Meanwhile, ordinary authors' fan-out jobs queue behind the celebrity's and their freshness degrades too, which is the part that makes this urgent.

V2 is the hybrid:

The threshold of 600,000 is derived, not chosen: freshness budget of 30 seconds multiplied by the 20,000 inserts per second a single author's fan-out may consume without starving everyone else. If the interviewer relaxes freshness to 5 minutes, the threshold moves to 6 million and the hybrid may become unnecessary. Say that out loud; it demonstrates that the number is a lever and not a fact.

The read-side merge cost is bounded because a user follows few very large accounts. Assume a ceiling of 50 celebrity followees per user (ASSUMED; it is bounded by how many accounts above 600,000 followers exist and how many any one person follows). Fifty range reads issued as one batched call adds a single round trip.

Deep dive one: the fan-out cost model as follower count grows

The crossover is a property of an edge, not of a user. This is the framing that makes the whole trade-off collapse into one comparison. Consider a single follow edge from author A to follower B:

• Push costs one insert every time A posts.
• Pull costs one range read every time B opens the feed.

So push is cheaper on that edge exactly when A's posts per day is less than B's feed opens per day. Everything else in this section is a consequence.

Checking the celebrity with the same rule: an account posting 20 times a day to 50 million followers generates 1 billion inserts per day under push. Those 50 million followers collectively open the feed 250 million times per day, so pull costs 250 million reads. Pull is 4 times cheaper on steady-state volume, and it is unboundedly better on freshness, because there is no fan-out to wait for.

Total cost as a function of follower count. Hold posts per day at 2 and follower opens at 5, and vary F:

Read the table carefully: push stays cheaper on raw operation count at every row, including the last. Push does not lose on cost, it loses on latency and on burst isolation. That distinction is the senior-level point, and most treatments get it wrong by claiming push is too expensive for celebrities.

The three fan-out variants, side by side.

Fan out to active followers only. With 60 percent of accounts inactive for 30 days, skipping them removes 8,400 inserts per second and 60 percent of the 4.8 TB. The returning-user path costs almost nothing: at an assumed 100,000 returns per day times 200 followees, that is 20 million range reads per day, which is 231 per second. Two orders of magnitude cheaper than what you saved. This is the highest-return optimisation on the page and it takes one predicate.

Deep dive two: pagination and duplicate suppression on a mutating list

The dives vary by problem on purpose. The crawler's second dive was deduplication of content; here it is deduplication of what the reader has already seen, which is a different problem with a different economy.

Chronological feeds paginate cleanly. A cursor of (timestamp, post identifier) is stable because new items arrive at the head, never in the middle. Page two is "everything older than this cursor". Offset pagination is wrong here for the usual reason: an insert at the head shifts every offset, so the reader sees an item twice.

Ranked feeds break that guarantee. If the ranking changes between the request for page one and the request for page two, items move across the page boundary. The reader sees duplicates, gaps, or both, and neither is recoverable at the client.

The fix is to freeze the ranking for the session:

The 20 GB figure is what makes this feasible; if it had come out at 20 TB, the answer would have to be an approximation instead.

Suppressing already-seen items across sessions. Exact tracking is too expensive: 300 million users times 1,000 recent post identifiers times 8 bytes is 2.4 TB. A per-user Bloom filter of 1,000 items at a 1 percent false positive rate needs about 9,600 bits, which is 1.2 KB per user, or 360 GB in total. That fits.

The trade-off is worth stating precisely, because it is the reason a Bloom filter is right here and wrong in the crawler's seen-set. A false positive suppresses a post the user has not seen. The user never learns that the post existed, so the error is invisible and costs one item in a hundred.

In the crawler, the same error silently drops a page from the index. It is equally invisible, but there it accumulates permanently into a measurable coverage gap. Same mechanism, different acceptability, decided by whether the loss is recoverable.

Unfollow and delete correctness on a pushed feed. Materialised identifiers can outlive the reason they were written. Two options:

Filtering at read is usually right, because you are already hydrating each post and the visibility check rides along on data you fetched anyway.

Failure modes and the operating contract (feed)

Service level indicators:

• Feed read latency at p50 and p99, segmented by followee-count percentile. The segmentation is the point; an unsegmented p99 hides the fan-in problem entirely.
• Freshness, measured as publish time to visible time at the p99 follower.
• Fan-out queue lag, at p50 and p99, with the celebrity path excluded so it does not mask ordinary regressions.
• Empty feed rate, which catches materialisation failures that latency metrics miss.
• Duplicate item rate per session.

Alert on fan-out lag p99 crossing the freshness objective, on empty feed rate deviating from its baseline, and on feed read p99 for the top followee-count decile rather than for the whole population.

Re-check against the requirements:

Where this answer is a simplification (feed)

• The ranking is the product. Everything above is the delivery substrate. A real home timeline is a candidate generation and ranking system with integrity filters, diversity constraints and advertising insertion, and those decide the user experience far more than fan-out does.
• "Feeding Frenzy: Selectively Materializing Users' Event Feeds" (Silberstein and colleagues, SIGMOD 2010) is the primary academic treatment of the push, pull and hybrid decision, and it formalises the per-edge crossover argument used above. It is the single best thing to read after this page.
• Facebook's TAO paper, TAO: Facebook's Distributed Data Store for the Social Graph (Bronson and others, USENIX ATC 2013, accessed August 2026), describes the graph store behind this class of read path and is the clearest published source on hydration at scale.
• LinkedIn published a description of FollowFeed on its engineering blog in 2016 that argues explicitly for the pull side of the trade-off, which is a useful counterweight to the assumption that push always wins. Search for it by title; the post has moved between blog paths since publication.
• Twitter's timeline architecture talk, Timelines at Scale (Raffi Krikorian, QCon San Francisco 2012, published on InfoQ April 2013, accessed August 2026), is the widely cited primary account of a production hybrid.
• Multi-surface reality. Home feed, profile, search and notifications all need similar materialisation, and the cost of building four of these separately is the actual architectural problem inside a large company.

Level delta: the same prompt at three levels (feed)

What each level added:

• Mid to senior: the design is derived from a read-to-write ratio and a measured degradation curve instead of from a remembered pattern.
• Senior to staff: thresholds become negotiable levers tied to stated objectives, a widely repeated claim gets corrected with arithmetic, and the metric definitions are designed alongside the system.

Case study 4: Chat and presence

The prompt (chat)

"Design a messaging system supporting one-to-one and group conversations, with delivery receipts and online presence, across a user's multiple devices."

Clarifiers that fork the design (chat)

Requirements (chat)

Functional:

• Send a message to a one-to-one or group conversation, and deliver it to every participant's devices.
• Report sent, delivered and read states to the sender.
• Show whether a contact is online.
• Let a device that has been offline catch up without losing or duplicating messages.

Non-functional:

Numbers that eliminate options (chat)

V0: the smallest design that meets the requirements (chat)

+----------+     +----------------------+     +----------------+
| Device   | <=> | Gateway plus chat    | --> | Messages store |
| websocket|     | connections in RAM   |     | key conv id    |
+----------+     | assigns seq per conv |     | plus seq       |
                 +----------------------+     +----------------+
                          |
                          v
                 +----------------------+
                 | Push provider        |
                 | for offline devices  |
                 +----------------------+

Caption. V0 holds websocket connections in one process, assigns a per-conversation sequence number on write, appends to a messages store keyed by conversation and sequence, and asks a push provider to wake devices that are not connected.

Correct at: 50,000 concurrent connections (about 500 MB of connection state at the assumed 10 KB each) and 500 messages per second, which is a trivial insert rate.

Three properties V0 already has, which is why the later versions are additions rather than rewrites:

• Ordering is solved. One partition owner assigns the sequence number, so there is one writer per conversation and therefore a total order, with no consensus protocol involved.
• The offline queue is solved. A device that reconnects sends its last known sequence number and asks for everything after it. There is nothing to queue.
• Duplicates are solved at the client. The client generates a message identifier before sending; the server enforces uniqueness of that identifier within the conversation, and the client discards anything it already has.

Breaking points, and what each version adds (chat)

Breaking point one, then V1 (chat)

What fails: 30 million connections cannot live on one node, and once they are spread, the sender's gateway does not know which gateway holds the recipient.

How you observe it: connection count per node against its memory ceiling, then, after a naive horizontal split, a delivery success rate that falls to roughly one over N as messages land on gateways that do not hold the recipient.

+---------+   +-------------+     +------------------+
| Device  |<=>| Gateway     | --> | Chat service     |
+---------+   | holds 500k  |     | assigns seq      |
              | connections |     | per conversation |
              +-------------+     | routes delivery  |
                    |             | back to gateways |
                    |             +------------------+
                    v                 |          |
                    |                 v          v
             +---------------------------+  +--------------+
             | Session registry          |  | Messages     |
             | user to device to gateway |  | store        |
             | written by the gateways   |  | partitioned  |
             | read by the chat service  |  | by conv id   |
             +---------------------------+  +--------------+

Caption. V1 separates a stateful gateway tier holding connections from a stateless chat service that assigns sequence numbers, writes to a partitioned message store, and routes delivery back through gateways; a session registry, written by gateways and read by the chat service, maps each user and device to the gateway currently holding it.

Breaking point two, then V2 (chat)

What fails: the second device.

How you observe it: unread counts that differ between a user's phone and laptop, messages that stay bold on one device after being read on another, and support reports of "my desktop is missing messages from yesterday". None of these show up in latency or error metrics, which is why this failure is usually found by users rather than by dashboards.

V2 makes the device, not the user, the unit of state:

• The session registry maps a user to a set of (device, gateway) pairs.
• Read state becomes a cursor per (device, conversation), with the per-user state derived from the set.
• Delivery targets every currently connected device, and every disconnected device catches up by cursor on reconnect.

Deep dive one: delivery ordering

Chat makes ordering interesting in a way the other three problems on this page do not: three different orderings exist and users notice when they disagree.

The mechanism: one writer per conversation. The conversation identifier is the partition key, the partition owner assigns a strictly increasing sequence number, and that gives a total order within the conversation without a consensus protocol. At 4,096 partitions and 46,000 messages per second, each partition sees 11 messages per second, so the single-writer constraint costs nothing.

Global ordering across conversations is both expensive and useless, and saying so is a better answer than designing it. No user can observe the relative order of two messages in two different conversations well enough for it to matter.

Push is a hint; pull by cursor is the authority. This is the simplification that removes an entire class of problems:

Client-side ordering rules, stated exactly:

• Sort by server sequence number, never by any clock. A client wall clock is attacker-controlled and routinely wrong.
• Show the sender's own message optimistically at the tail, marked pending, then re-sort when the server returns its sequence number. Users tolerate a message moving; they do not tolerate a delay before it appears.
• Deduplicate on the client-generated message identifier. Both a network retry and a session-registry split brain produce a second copy, and both are handled by the same three lines of client code.

Where this model does not stretch. Causal ordering across conversations, the case where someone says "check the group chat" and the referenced message has not arrived yet, is not solved by per-conversation sequence numbers. The Matrix specification models room history as a directed acyclic graph of events precisely to handle ordering under partition, and Lamport's "Time, Clocks, and the Ordering of Events in a Distributed System" (Communications of the ACM, 1978) is the foundational reading.

Deep dive two: multi-device sync and offline catch-up

The second dive differs from the news feed's pagination dive because the constraint is different: there the list was mutating under the reader, here the list is stable and the reader is intermittent across several devices.

The wrong model and the right one.

The cursor model stores messages once and makes every device's state a small pointer. The 360 GB figure is what makes it obviously correct compared with duplicating the message body per device.

Bounding catch-up. A device offline for three days must not replay three days. Naive catch-up across 50 conversations at 500 messages each is 25,000 messages, which at 500 bytes is 12.5 MB on a mobile connection at app start. Instead:

That is 2,000 messages, about 1 MB, which is a reasonable cold start. The remaining history stays reachable by cursor.

Receipt aggregation, stated as rules because ambiguity here ships bugs.

Push notifications are a separate, lossy channel. The mobile push services are best effort by design and support collapse keys and time-to-live, both documented by their owners: Apple in Sending notification requests to APNs (Apple Developer Documentation, accessed August 2026) and Google in Firebase Cloud Messaging (accessed August 2026), which documents collapsible messages and message lifespan. A push receipt is not a delivery receipt. Treating it as one produces the bug where a message is marked delivered while the phone is in a tunnel with the app killed.

The new-device problem, which encryption makes unsolvable rather than hard. With server-readable content, a new device fetches history by cursor and it just works. With end-to-end encryption the server holds ciphertext addressed to key material the new device does not have, so history cannot be handed over by the server.

Signal's Sesame specification (revision 2, April 2017, accessed August 2026) describes the multi-device session model, and it is the reason several encrypted products simply start a new device with no history. Say this in the room if the encryption clarifier came back as end-to-end; it demonstrates that you traced a requirement through to a product consequence.

Failure modes and the operating contract (chat)

Service level indicators:

• Send to delivered latency at p50 and p99, for online recipients only, since offline recipients measure the push provider rather than your system.
• Connection establishment success rate, and reconnect rate as a leading indicator of gateway trouble.
• Catch-up query latency at p99, which is the metric that goes bad first during a reconnect storm.
• Presence staleness at p99.
• Duplicate delivery rate, which should be small and non-zero. A reported zero means the metric is broken, because at-least-once delivery is the stated design.
• Message durability violations, target zero, measured by client-side gap detection in sequence numbers.

Alert on send to delivered p99, on gateway reconnect rate exceeding its baseline, on catch-up p99, and on any detected sequence gap.

Re-check against the requirements:

Where this answer is a simplification (chat)

• End-to-end encryption changes the system, not a component. The server cannot search, cannot moderate content, cannot deduplicate by content, and cannot serve history to a new device. Signal's Double Ratchet specification and Sesame (both accessed August 2026) are the primary sources, and RFC 9420 (July 2023) standardises Messaging Layer Security for group encryption at scale.
• Storage engines matter more than the diagram suggests. Discord published two detailed accounts of exactly this workload, How Discord Stores Billions of Messages (January 2017) and How Discord Stores Trillions of Messages (March 2023), both accessed August 2026, and together they are the best public description of message-store evolution under real growth.
• The connection tier is its own specialty. Very high connection counts per machine are achievable, but treat any specific figure you have heard as evidence that a heavily tuned stack reached it once, not as a number you can assume. Size the connection tier from your own measured memory per idle connection.
• Mobile networks are the real adversary. Connections drop constantly, radio wake-ups cost battery, and the client's retry policy affects your server load more than any server-side choice.
• Interoperability is now a regulatory input. The EU Digital Markets Act obligations that took effect for designated gatekeepers in March 2024 require messaging interoperability, which changes the protocol surface from an internal choice to an external contract.
• Matrix publishes its full specification at spec.matrix.org, including the event graph and state resolution, and it is the most detailed openly readable design for federated chat ordering.

Level delta: the same prompt at three levels (chat)

What each level added:

• Mid to senior: components get removed rather than added, and the delivery guarantee is stated as something achievable rather than as a wish.
• Senior to staff: one estimate (presence volume) reorders the whole design, ambiguous product semantics get pinned down as explicit rules, and the weakest part of the answer is volunteered rather than defended.

Case study 5: A video platform

The prompt (video)

"Design a video platform: people upload videos, and other people watch them."

Clarifiers that fork the design (video)

Ask these to change the architecture, not to fill two minutes. Each row has a different answer on each side.

Requirements (video)

Functional:

• Resumable upload of a source file.
• Transcode to a multi-rung bitrate ladder plus a manifest.
• Publish, then adaptive-bitrate playback with seek.
• Delete a video, including from the edge.

Non-functional:

Numbers that eliminate options (video)

Every number below is followed by what it removes from the table.

1. Upload arrivals: 500,000 divided by 86,400 seconds is 5.8 per second; at an assumed 3x diurnal peak that is 17 per second. Eliminates sharding the upload metadata database and putting a queue in front of the upload API. Seventeen inserts per second is one modest database.
2. Source minutes per day: 500,000 uploads times 10 minutes is 5,000,000 source-minutes.
3. Transcode compute: ASSUME 4 CPU-minutes to produce the whole five-rung ladder from one source-minute, because each rung encodes faster than real time on a single core at these resolutions and five rungs sum to a small multiple of real time. That is 20,000,000 CPU-minutes per day, or 20,000,000 divided by 1,440, about 13,900 cores running continuously. Eliminates transcoding inside the upload API process, and eliminates transcoding on demand at playback time.
4. Output bytes: a ladder of 0.3, 0.7, 1.2, 2.5 and 4.5 Mbps sums to 9.2 Mbps, which is 9.2 times 60 divided by 8, or 69 MB per source-minute. Times 5,000,000 gives 345 TB per day, about 126 PB per year. Eliminates a single hot storage tier and eliminates generating every rung for every video regardless of demand.
5. Delivery: 100,000,000 viewers times 40 minutes is 4,000,000,000 watch-minutes per day. At 2.5 Mbps that is 18.75 MB per watch-minute, so 75 PB per day, which is 868 GB per second, about 6.9 Tbps averaged. Eliminates serving playback from origin under any plausible origin footprint.
6. Origin egress after caching: at a 95 percent edge hit ratio origin carries 5 percent of 6.9 Tbps, about 345 Gbps. At 90 percent it is 690 Gbps. Eliminates treating hit ratio as an implementation detail: five points of hit ratio doubles the origin tier.
7. Read to write ratio: 4,000,000,000 watch-minutes against 5,000,000 source-minutes is 800 to 1. Eliminates any design that makes the read path more complex to shave latency off the write path.

V0: the smallest design that meets the requirements (video)

+----------+     +---------------+
| Uploader |---->| Upload API    |----+
+----------+     +---------------+    |
                                      |
+----------+     +---------------+    |
| Viewer   |---->| Play API      |----+
+----------+     +---------------+    |
                                      |
       +------------------------------+------------------+
       | Metadata database: videos, job state, manifests |
       +------------------------------+------------------+
                                      |
       +------------------------------+------------------+
       | Object store: source master and ladder segments |
       +------------------------------+------------------+
                                      |
       +------------------------------+------------------+
       | Transcoder worker pool, polls job state in the  |
       | metadata database                               |
       +-------------------------------------------------+

Caption: V0 for a video platform. The upload API writes the master to the object store and a pending row to the metadata database. A worker pool polls that table, reads the master, writes ladder segments back to the same bucket, and marks the row published. The play API returns a manifest whose segment URLs point straight at the object store. Five boxes, no queue, no content delivery network.

This satisfies every functional requirement. It is correct up to two computable limits.

• Transcode limit. One 32-core worker gives 32 times 1,440, or 46,080 CPU-minutes per day. Divided by the assumed 4 CPU-minutes per source-minute, that is 11,520 source-minutes, or about 1,150 uploads per day at 10 minutes each.
• Delivery limit. ASSUME the object store front end gives you 10 Gbps, which is 1.25 GB per second. One concurrent viewer at 2.5 Mbps consumes 0.3125 MB per second. That is about 4,000 concurrent viewers.

Below roughly 1,000 uploads a day and 4,000 concurrent viewers, adding a queue, a job orchestrator or a cache is work you cannot justify with a measurement.

Breaking points, and what each version adds (video)

Break one, at about 1,150 uploads per day. The polling worker pool cannot drain the backlog, so job age grows without bound and p95 publish latency crosses 30 minutes. You observe it as oldest pending job age, not as queue depth: depth is ambiguous when workers are scaling, age is not.

V1: split the pipeline from the API and chunk the work.

+----------+    +------------+    +-------------------+
| Uploader |--->| Upload API |--->| Object store      |
+----------+    +-----+------+    | master            |
                      |           +-------------------+
                      |
                +-----+------+    +-------------------+
                | Job queue, |--->| Segmenter: cut    |
                | durable    |    | master into parts |
                +------------+    +---------+---------+
                                            |
                +---------------------------+---------+
                | Encoder workers, one task per part  |
                | per rung, autoscaled on job age     |
                +---------------------------+---------+
                                            |
                +---------------------------+---------+
                | Stitcher and packager: manifests,   |
                | then mark published                 |
                +-------------------------------------+

Caption: V1 transcode pipeline. The upload API only stores the master and enqueues one job. A segmenter cuts the master at keyframe boundaries, encoder workers process parts in parallel across every rung, and a stitcher assembles segments plus manifests before publishing.

Chunking is what buys the latency target: a 10 minute video split into 30 second parts is 20 parallel tasks per rung, so wall-clock publish time falls to roughly the time for one part plus fixed overhead, instead of the time for the whole file.

Break two, at about 4,000 concurrent viewers. Object store egress saturates. You observe it as segment time-to-first-byte climbing and the player's rebuffer ratio rising; playback start p95 crosses 2 seconds well before the store returns errors.

V2: put a cache hierarchy in front, and make the objects cacheable.

+--------+   +----------+   +--------------+   +-----------+
| Player |-->| Edge     |-->| Origin       |-->| Object    |
|        |   | cache    |   | shield cache |   | store     |
+--------+   +----------+   +--------------+   +-----------+
     |
     +-----> +--------------------------------------------+
             | Play API: manifest, entitlement, URL signer |
             +--------------------------------------------+

Caption: V2 delivery path. Players fetch immutable segments through an edge cache backed by a shared origin shield, so a cold segment is fetched from the object store once for the whole edge fleet rather than once per edge. Manifests come from the play API on a short time to live; segments never change and carry a long one.

Break three, at catalogue scale. From the sizing, full ladders for everything cost 126 PB per year, and most of it is never requested. You observe it as bytes stored per byte delivered, sliced by video age. When the tail rung of a two-year-old video has zero requests in 90 days, you are paying rent on nothing.

V3: generate cold rungs on demand and tier the storage. Keep the top two rungs and the lowest rung eagerly. Package rare rungs just in time on the first request, accepting a slower first play for a video nobody is watching. Move masters to archival storage after 30 days.

Boxes not to add without a measurement (video)

Deep dive one: chunk boundaries and the encoding ladder

This dive earns its place because chunked encoding is where a video pipeline is uniquely hard: the parallelism is only correct if every worker agrees on where the cuts are.

Why cuts must be keyframe aligned. A decoder can only start at a keyframe. If part boundaries land at different frames on different rungs, a player that switches rungs mid-stream has no common resume point and either stalls or shows a glitch.

The rule. Force a fixed group-of-pictures length, ASSUME 2 seconds, on every rung, and cut parts at multiples of it. Two seconds is a reasonable default because it bounds switch latency to one segment while keeping keyframe overhead small; measure the bitrate cost on your own content before committing.

The stitching trap. Rate control decisions do not carry across an independently encoded part, so a two-pass encoder that targets an average bitrate per part will produce a stream whose bitrate wobbles at part boundaries. State the mitigation: pass a fixed quantizer target rather than a bitrate target per part, then check the assembled stream's peak bitrate against the rung's ceiling.

The ladder itself. A fixed ladder wastes bits on simple content and starves complex content. Netflix published in December 2015 that it replaced a fixed ladder with a per-title ladder derived from running many encodes and choosing points on the resulting quality curve (Per-Title Encode Optimization, Netflix Technology Blog, December 2015). Naming that trade in the interview, and then saying you would ship a fixed ladder first because per-title costs several extra encodes per video, is a stronger answer than proposing per-title immediately.

Deep dive two: cache hit ratio as the economic centre of the design

The second dive is deliberately not another pipeline topic. Delivery is where the money is at 800 to 1, and hit ratio is the one variable that moves it.

What determines hit ratio:

The arithmetic to say out loud. From the sizing, 6.9 Tbps of delivery at 95 percent hit is 345 Gbps of origin fill. Adding a shield in front of, ASSUME, 30 edge regions turns 30 independent cold fetches for a newly viral segment into one, which is the difference between a manageable spike and an origin brownout.

Failure modes and the operating contract (video)

Service level indicators to publish: publish latency p50 and p95, playback start time p95, rebuffer ratio, segment error rate, edge hit ratio, oldest pending job age.

Re-check against the requirements.

• Publish p95 under 30 minutes is met by chunked encode plus autoscaling on job age; the failure mode that threatens it is the poison-pill retry storm, so the attempt cap is a requirement, not a nicety.
• Playback start p95 under 2 seconds is met by the edge cache plus a short first segment; it is threatened by manifest time to live choices, so manifest latency gets its own indicator.
• Zero acknowledged uploads lost is met by acknowledging only after the master is durably written, which is why the upload API writes to the object store before it writes the metadata row.

Where this answer is a simplification (video)

• Hardware encoding changes the compute arithmetic entirely. Google published a paper on deploying custom video transcoding hardware at warehouse scale (Warehouse-scale video acceleration: co-design and deployment in the wild, ASPLOS, April 2021). If your interviewer works on video, expect the CPU-minutes assumption to be challenged.
• Shot-aware encoding splits on scene changes rather than fixed intervals, which interacts badly with the fixed-interval chunking above. Netflix published its approach in March 2018 (Optimized shot-based encodes, Netflix Technology Blog).
• The manifest formats are standards with real constraints. HTTP Live Streaming is specified in RFC 8216 (IETF, August 2017); the common segment container that lets one set of segments serve both major streaming formats is the Common Media Application Format, ISO/IEC 23000-19.
• Delivery at this scale is often not a rented content delivery network. Netflix documents its own appliance programme publicly at Open Connect. Saying "at this scale you eventually build the cache" is fine; saying every large platform does is not.

Level delta: the same prompt at three levels (video)

The mid answer is not wrong; it is undifferentiated. The senior delta is derivation and one genuine depth area. The staff delta is a stated objective function, a deletion, and an interface between teams.