Product Architecture Interview

Prerequisites

Seniority is a weak readiness signal for this round in particular, because plenty of staff engineers have never owned an interface they could not change on Monday. Prerequisite mastery is the signal that works. Answer every Quick check in under a minute. Two or more misses means start with the linked material rather than with Module 1.

Time budget

Learning objectives

Each objective is tested by exactly one numbered item in the mastery check, and the numbering matches: objective 1 is tested by mastery item 1.

1. Derive a resource model from a stated domain in under ten minutes, listing entities, aggregate boundaries and at least three invariants, and name the single invariant that decides where the transaction boundary sits.
2. Choose resource, procedure or event exposure for a stated operation, citing the client count and the coupling cost that decides it, and name one condition under which exposing the same domain two ways is correct rather than lazy.
3. Design an error model for a stated interface: a code taxonomy, a retryability signal, and partial-failure semantics for a batch call, such that a client can branch on every class without parsing a human message.
4. Specify idempotent creation for a stated write: the key, its lifetime, the request fingerprint and the transaction boundary, and name the exact retry sequence that still duplicates if the boundary is drawn one line too late.
5. Compare offset, cursor and keyset pagination on a collection receiving concurrent inserts and deletes, naming the anomaly each produces (duplicate, skip, unstable order) and the depth in pages at which offset stops being defensible.
6. Plan a breaking change for clients you cannot force to upgrade, giving the compatibility classification, the signalling mechanism, the migration order, and the measurement that ends the deprecation window rather than a date chosen in advance.
7. Rewrite a mid-level contract answer into a staff-level one by adding a deprecation path, a quota and metering consequence, a webhook delivery guarantee and a blast-radius statement, annotating what each addition changed about the decision.

Warm-up retrieval

Three to five minutes. Answer out loud or in writing before opening anything else. Retrieval beats rereading, and a visible answer converts one into the other.

Q1. The hub's delivery clock lists six situations where following the clock is wrong. One of them names this round type explicitly. State it, and state the one-line move it tells you to make instead.

Q2. From the trade-off atlas on the hub, several pages back: state the rule for choosing between a representational style, a remote procedure style and events, and name the quantity that decides it.

Q3. The hub works a payments idempotency prompt as its example of skipping estimation. It concludes that exactly one number still deserves to appear. Name that number and say which option it kills.

Module map

+---------------------------------------------------------+
| MODEL                                       M1 to M4    |
| round type and scope, the contract clock, entities,     |
| aggregates and invariants, resource or procedure or     |
| event as three ways to expose one domain                |
+----------------------------+----------------------------+
                             |
                             v
+---------------------------------------------------------+
| CONTRACT                                    M5 to M8    |
| style selection, schema as the design artefact,         |
| error taxonomy and partial failure, pagination on a     |
| collection that changes while you read it               |
+----------------------------+----------------------------+
                             |
              +--------------+--------------+
              v                             v
+-----------------------------+ +-------------------------+
| CORRECTNESS      M9 to M11  | | POLICY      M12 to M14  |
| idempotency keys, entity    | | grants and scopes,      |
| tags and lost updates,      | | quotas and limits,      |
| versioning and deprecation  | | long-running operations |
+--------------+--------------+ +------------+------------+
               |                             |
               +--------------+--------------+
                              v
+---------------------------------------------------------+
| DELIVERY                                   M15 to M18   |
| webhooks and event delivery, batch partial success,     |
| client and network constraints, AI tool contracts       |
+----------------------------+----------------------------+
                             |
                             v
+---------------------------------------------------------+
| OPERATE AND JUDGE                          M19 to M21   |
| interface as product, per-operation indicators and      |
| resilience budgets, level calibration                   |
+---------------------------------------------------------+

Caption: six clusters and their reading order. Model feeds
Contract, because endpoints are derived from a domain rather
than guessed. Contract feeds two parallel clusters:
Correctness, which is what a retrying client does to you, and
Policy, which is what you charge and permit. Both feed
Delivery, the outbound and bulk surfaces. Everything feeds
Operate and Judge, where mid and staff answers separate.
Arrows point from prerequisite to dependent.

Low ends total 480 minutes, high ends total 600. That is the 8 to 10 reading hours in the time budget above.

The delivery clock for this round

Most candidates who fail a contract round did not miss a component. They drew a service diagram, listed six endpoint paths, and never said what a client receives when a call times out, arrives twice, or is written against last year's schema. This clock is a budget rather than a script, and it is shaped differently from a generalist one: modelling and semantics take the minutes that a distributed systems round spends on capacity and topology.

One structural difference worth naming immediately: there is no estimation phase. The hub's clock allows two to three minutes for a number that eliminates an option, and in this round that number usually does not exist, for the reason worked through in the honest skip. Offer it, do not perform it.

The 45 minute round

The phases sum to 45 by construction: 2 plus 6 plus 8 plus 8 plus 8 plus 8 plus 5.

The 60 minute round

These sum to 60: 2 plus 7 plus 10 plus 9 plus 10 plus 8 plus 10 plus 4. The extra 15 minutes buys a real evolution block and a second dive. It does not buy more endpoints.

The first two minutes

Say four sentences, in this order. Rehearse them once so they are automatic and your attention is free for listening.

1. "Let me restate it as a domain: the nouns are X and Y, the operations people care about are A and B, and the consumers are Z." (If you cannot fill the consumer slot, that is your first clarifying question.)
2. "I am going to spend about eight minutes on the model and the invariants before I write a single path, because the endpoints fall out of that." (You just told them your clock, and you justified it.)
3. "I will treat the error and retry semantics as part of the design, not as a detail at the end." (Cheapest senior signal available in this round, and almost nobody says it.)
4. "Stop me if you want depth somewhere specific rather than coverage." (Interviewers take this invitation more often than candidates expect.)

Declaring what you are skipping

Skipping silently reads as not knowing. Skipping out loud reads as prioritising. The pattern: name it, say why it cannot change a decision in the next twenty minutes, say in one sentence what you would do if it were in scope.

• "I am not designing the storage engine behind this. Every option we have discussed holds the same rows, so the choice cannot change the contract, which is what the round is about."
• "I am parking the graph style. It is a real option, but with two internal consumers and one partner it buys flexibility nobody asked for and costs us a query-cost problem. Flag me if you want it and I will come back with the comparison."
• "I am leaving the sandbox and key-issuing flow out. It matters commercially and it is where developer experience is won, but it does not change the resource model or the error semantics we are working on."

Checking in without sounding unsure

A weak check-in asks for approval. A strong one offers a fork. Two per 45 minute round is about right, one after the model and one before the deep dive.

How the split shifts by level

The shape: mid level spends its minutes proving a usable interface could exist, senior spends them proving the semantics were chosen rather than defaulted, staff spends them proving the interface survives five years of clients, integrations and pricing changes it does not control.

Module 1. Scope and honesty: what this round actually asks for

Time: 15 to 20 minutes reading, 10 to 15 minutes practice.

1a. Predict first

Three prompts, all taken from the same day of the same loop. Read them before anything else on this page.

Three rounds hide behind one course title

The phrase "product architecture interview" is used by different companies for at least three different things. Preparing for the wrong one is the most expensive mistake available here, because it costs weeks and you will not discover it until minute five.

How to tell inside 90 seconds. Listen for what the prompt says already exists.

• A consumer exists and is external ("a partner", "a mobile app we do not control", "a third party"): interface design.
• A user count exists and no consumer is named: distributed system design.
• A complaint or a metric exists: product sense.
• Nothing exists and the interviewer says "start wherever you want": ask. One sentence buys you forty minutes.

The title history, once, and then we move on

This course sits under a title that several publishers use, and the equivalent paid course was previously published under an API Design Interview title, wording that still survives in places on its own page. That history is why two syllabi that look unrelated by name cover the same ground. The hub documents the publisher-by-publisher picture with dated primary links in the naming section.

The practical consequence for you is not commercial, it is about preparation:

• If a recruiter says "product architecture", assume interface and domain design until told otherwise, because that is what the syllabus under that title has historically contained.
• If a recruiter says "API design", it is the same round. Do not prepare twice.
• If the interviewer opens with a user count and no named consumer, you are in a distributed systems round wearing the wrong name. Say so and steer, using the script in module 2.

What a contract round is actually scored on

Six things, in rough order of how often they decide the rating.

The rows are not equally weighted by accident. Domain model and error semantics carry the most, because they are the two a candidate cannot fake by recall.

The routing diagram

              first line of the prompt
                        |
        +---------------+---------------+
        |               |               |
   a consumer      a user count     a complaint
   is named        is given         or a metric
        |               |               |
        v               v               v
+--------------+ +-------------+ +--------------+
| interface    | | distributed | | product      |
| and domain   | | system      | | sense        |
| design       | | design      | | round        |
+--------------+ +-------------+ +--------------+
        |
   public or partner consumer
        |
        v
+--------------------------+
| evolution and versioning |
| become half the round    |
+--------------------------+

Routing the first line of a round to one of three rubrics, and
the extra weight a public consumer adds to a contract round

1d. Worked example

The recruiter email says: "Onsite for Senior Software Engineer, Platform. Four rounds: Coding (45), Product Architecture (60), Distributed Systems (60), Behavioural (45). The Product Architecture interviewer maintains our public developer platform."

Block 1. PURPOSE: convert the round list into deliverables, because I rehearse deliverables and not topics.

Three technical rounds, three different artefacts: working code, a domain model plus a contract, a box diagram plus numbers. I write those three phrases at the top of my plan. Anything I study has to serve one of them, and material that serves none gets cut.

The strongest signal in the whole email is "maintains our public developer platform". A public platform means external callers I cannot upgrade, which promotes versioning, deprecation and error contracts from nice to have into likely probes.

The error most readers make here is treating the two 60 minute rounds as one topic and preparing shared material for both. They share almost nothing. One is about aggregates and contracts, the other is about partitions and failure.

Block 2. PURPOSE: decide what I will deliberately not prepare, because preparation is subtraction.

Given a public platform, I drop three things from the contract round entirely: consensus algorithms, cache topology, and anything about cluster sizing. Those belong to the other round and I have already booked time for them there.

That frees my contract round time for five things: aggregate design, error taxonomy, idempotency, pagination stability, and a deprecation path for a client I cannot force to upgrade.

The error most readers make here is additive preparation. They keep the general system design syllabus and add contract material on top, then arrive having half learned both.

Block 3. PURPOSE: buy information cheaply, because two sentences of recruiter reply beat two hours of guessing.

I send one message: "For the Product Architecture round, is the prompt usually a greenfield interface, or an existing public interface being changed? And should I assume external third party callers or internal service callers?"

Both answers move real hours. Greenfield plus external means I rehearse domain modelling and versioning. Brownfield plus external means I rehearse the breaking change migration path first, because that will be the whole round.

Block 4. PURPOSE: rehearse the opening, because it is the only fully predictable forty seconds of the round.

I write and say out loud: "Before I draw any operations I want to spend about eight minutes on the domain: what the entities are, which of them can change independently, and what must never be true. Then I will pick an interface style and sketch six operations, then spend real time on errors and retries. If you would rather I go deep on one area instead, tell me now."

The error most readers make here is improvising the opening and then listing endpoints in minute two, which locks in a resource model nobody has justified.

Result. The reply was "existing public interface, external callers, and she usually asks how you would remove something". That moved the last two hours of preparation from resource modelling drills to deprecation mechanics, which is exactly where the round went.

1e. Fade the scaffold

Faded example 1. The email says: "Onsite, Backend Engineer, Payments. Rounds: Coding (45), Product Architecture (60), Behavioural (45). The Product Architecture interviewer works on the merchant integrations team." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: deliverables are code and a contract. Only one design round, so it carries all the design weight.
• Block 2: I drop general resource modelling breadth and concentrate on money semantics: idempotency, retries, reconciliation, webhooks.
• Block 3: I ask whether the prompt is the merchant facing interface or the internal ledger interface.

Faded example 2. The email says: "Virtual onsite, Software Engineer, Developer Platform. Rounds: Product Architecture (60), Distributed Systems (60), Coding (45). The company published a v2 of its public API eighteen months ago and v1 is still live." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: three deliverables. The v1 and v2 detail in the email is not decoration, it is the interviewer's daily problem.
• Block 2: I drop greenfield only material. I keep the breaking change taxonomy, deprecation signalling, and client migration curves.

1f. Check yourself

Q1. An interviewer opens with: "Our partners integrate with our booking API. Design it." You have 60 minutes. Which of the six scored signals in this module should get the largest single block of your time, and why?

Q2. Ten minutes into a round titled "product architecture", the interviewer asks how you would shard the storage for 40 terabytes of images. What just happened, and what do you do?

1g. When not to use this

Round triage is a preparation tool, and it is over applied far more often than under applied.

Module 2. The delivery clock for a contract design round

Time: 15 to 20 minutes reading, 10 to 15 minutes practice.

2a. Predict first

Two candidates get the same 60 minute prompt: "Design the interface a partner uses to manage bookings." Here is how each spent the working minutes.

The budget

The hub owns the shared clock and the working minute deductions (the clock). This module only states how a contract round differs, which it does in two ways: there is no capacity estimation phase, and there is a phase the other tracks do not have at all, which is evolution.

Working minutes are 40 in a 45 minute slot and 52 in a 60 minute slot, using the hub's deduction for introductions and closing questions.

The operation sketch number is derived, not chosen. Writing one operation with its path, its request fields, its response fields and its failure cases takes roughly two minutes at whiteboard speed. Six operations therefore cost about twelve to thirteen minutes, which is what the table allocates. If you find yourself needing eleven operations, you have not found the aggregates yet.

+-------+----------+-----+-----------+--------+------+-----+--+
| Scope | Domain   |Style| Operations| Errors | Dive | Evo |Wr|
| 5     | 10       | 3   | 13        | 7      | 8    | 4   |2 |
+-------+----------+-----+-----------+--------+------+-----+--+
0       5          15    18          31       38     46   50 52

Working minutes in a 60 minute contract round. The two blocks
either side of the operation sketch are worth more than the
sketch itself

The first two minutes of a contract round

Four sentences, in this order, before you draw anything.

1. Name the consumer: "I am assuming an external partner we cannot force to upgrade. Correct me if it is an internal caller."
2. Name the boundary: "I will design the contract and the domain behind it, not the storage layout, unless you want that."
3. Name the order: "Entities and invariants first, then operations, then errors and retries, then how it changes over time."
4. Offer the trade: "If you would rather I go deep on one of those, say so now and I will drop the others."

Sentence one is the highest value sentence in the round. Internal caller and external caller answers diverge within ten minutes on versioning, error verbosity and authorisation, and guessing wrong wastes half the hour.

How the split moves by level

Same 52 working minutes, reallocated. Read the columns, not the cells.

A mid level answer is judged on whether a coherent, complete contract appears. A senior answer is judged on the error and retry semantics and on whether the model survives a hostile follow up. A staff answer spends its budget at the two ends: negotiating what the interface should even own, and stating how it will be evolved and operated.

When to break the clock

Three situations where following the budget is the wrong move.

2d. Worked example

The prompt: "Design the interface a hotel partner uses to manage room availability and bookings." 60 minutes.

Block 1. PURPOSE: spend the first five minutes buying constraints, because every later phase is priced by the answers.

I ask four questions and nothing else: is the partner external and unupgradable, how many partners are there, does availability change from our side as well as theirs, and is a booking cancellable after confirmation.

The answers come back: external, roughly 800 partners, availability changes from both sides, and cancellation is allowed within policy windows. I write those four facts on the board where the interviewer can see them.

The error most readers make here is asking questions whose answers do not change the design, for example the expected request volume. In a contract round throughput rarely forks the interface. Bidirectional mutation always does.

Block 2. PURPOSE: build the domain before the operations, because the operation list is a consequence and not a choice.

I name three aggregates: Property, RatePlan, Booking. I state one invariant per aggregate out loud, for example "a Booking never overlaps another Booking on the same room for the same night".

Then I say the sentence that earns the round: "Because availability changes from both sides, the invariant cannot live only in the partner's system, so the operation that consumes inventory has to be ours and has to be conditional."

The error most readers make here is jumping from nouns to URLs. "Property, rate plan, booking" becomes three collection paths and the invariant is never mentioned, so the conditional write requirement never surfaces.

Block 3. PURPOSE: sketch only the operations the invariants require, and stop.

Five operations, about two minutes each: list properties, replace a rate plan for a date range, read availability, create a booking conditionally, cancel a booking. I write request and response fields for the conditional create only, and say I will expand others on request.

I explicitly do not add search, reporting or a webhook here. I name them as deliberately out of scope in one sentence, so it is clear they were considered rather than forgotten.

Block 4. PURPOSE: spend the errors block on the two failure cases the interviewer will probe.

The two are: the partner retries a create after a timeout, and the partner writes availability that conflicts with a booking we already accepted. I give the first an idempotency key with a 24 hour lifetime, and the second a conditional write with a version token plus a specific conflict error.

The error most readers make here is describing errors as a status code list rather than as answers to "what does the caller do next".

Result. With five minutes left I stated what I skipped (search, reporting, webhooks), named one thing I would measure first (conflict errors per partner per day, because a partner with a rising conflict rate is about to file a support ticket), and answered two follow ups. The interviewer spent the whole deep dive on conditional writes, which is where block 2 had pointed.

2e. Fade the scaffold

Faded example 1. Prompt: "Design the interface an e-commerce seller uses to manage a product catalogue." 45 minutes, so 40 working minutes. Blocks 1 to 3 are given. Produce block 4.

• Block 1: four questions bought: external sellers, roughly 50,000 of them, images uploaded separately, and catalogue changes are reviewed before going live.
• Block 2: aggregates are Product, Variant, Listing. Invariant: a Listing is never live while its Product is in review.
• Block 3: five operations sketched, with full field detail only on the bulk upsert.

Faded example 2. Prompt: "Design the interface an internal fraud service exposes to our checkout service." 60 minutes. Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: internal caller, one consumer today and two planned, synchronous, and checkout has a 300 millisecond budget for the whole call.
• Block 2: one aggregate, Assessment. Invariant: an Assessment is immutable once returned, so a later re-score is a new Assessment.

2f. Check yourself

Q1. You are 30 minutes into a 60 minute contract round and you have five operations sketched but have not said the word "retry". What do you cut to make room, and what do you say while cutting?

Q2. The interviewer says nothing for the first fifteen minutes, then asks only about authorisation for the remaining forty. Which phases of the budget do you sacrifice, and which do you protect?

2g. When not to use this

Module 3. Domain modelling first: entities, aggregates and invariants

Time: 30 to 35 minutes reading, 25 to 35 minutes practice.

3a. Predict first

A candidate proposes this data model for a seat booking system, then proposes one write operation per table.

The four words, defined before use

Bounded context arrives later in this module, because it only makes sense once aggregates exist.

The rule that decides aggregate boundaries

One rule, stated as a question: what must be true the moment the write returns, and what may be repaired later?

Two consequences follow, and both are load bearing:

• Aggregates reference each other by identity only, never by embedded object. If Booking holds a whole Property object, you have merged two transaction boundaries by accident.
• One aggregate instance per transaction. If an operation must change two aggregates atomically, either the boundary is wrong or the operation needs a saga with compensation.

Why the boundary is a throughput decision, not a taste decision

Aggregate size sets your write concurrency, and the arithmetic is short enough to do at a whiteboard.

Inputs, all stated here: a 40,000 seat venue, an on sale burst of 2,000 hold requests per second, and an assumed 5 milliseconds of transaction time per hold. That assumption is reasonable because a hold is one indexed lookup plus one insert on a warm primary, and 5 milliseconds is a conservative round number for that pair.

The event as aggregate choice is 10 times short of the 2,000 per second requirement, so it is eliminated by arithmetic rather than by opinion. The section choice clears it with a factor of 20 in hand.

The per seat choice looks best on this table and is usually wrong, because "pick any two adjacent seats" then spans two aggregates and needs a saga to stay atomic. The section boundary is the smallest one that still contains the adjacency invariant.

+---------------------------------------+
| Aggregate Section 12                  |
|                                       |
|  +----------+  +----------+           |
|  | Seat A12 |  | Seat A13 |  200 seats|
|  | held     |  | free     |           |
|  +----------+  +----------+           |
|                                       |
|  invariant  one holder per seat       |
|  invariant  adjacent pairs stay whole |
+---------------------------------------+
        |  by id only
        v
+-----------------+     +------------------+
| Aggregate Order |     | Aggregate Payment|
| seat ids only   | --> | order id only    |
+-----------------+     +------------------+

Aggregate boundaries for seat booking. Lines crossing a boundary
carry identifiers only, so each box is one transaction

Bounded contexts: the same word meaning two things

A bounded context is a boundary inside which one word has exactly one meaning. It is the answer to the observation that "Order" in fulfilment and "Order" in billing are not the same object and should not share a model.

The interview relevant consequence: one contract should serve one context. When a single operation returns fields from two contexts, you get an object that no team fully owns and that cannot change without cross team agreement. If you catch yourself adding a billing_status field to a fulfilment resource, you have crossed a context boundary and should say so out loud.

Deriving the interface from the model

Once aggregates and invariants exist, most of the operation list is mechanical.

That last row is where the resource style starts to strain, which is exactly what module 4 is about.

3d. Worked example

Prompt: "Model the domain behind an interface that lets a partner sell tickets to our events."

Block 1. PURPOSE: separate entities from value objects, because the split decides what gets an identifier.

I list candidate nouns from the prompt and from two clarifying answers: Event, Seat, Section, Hold, Order, Money, DateRange, Attendee.

Money and DateRange are value objects. Two amounts of 4500 cents in the same currency are interchangeable, so they need no identity. Everything else has a lifecycle I need to reference later, so it gets an identifier.

The error most readers make here is giving Money an id because it lives in its own table. Storage layout is not the domain. If the only reason a thing has an identifier is a foreign key, it is a value object wearing a row.

Block 2. PURPOSE: state invariants before boundaries, because the invariants are the input and the boundaries are the output.

I write four invariants:

1. A seat has at most one active hold or sale at any instant.
2. A hold expires at a stated time and frees the seat.
3. An order's total equals the sum of its seat prices plus fees.
4. An event's sold count never exceeds its capacity.

Invariant 1 must hold at the instant of the write, because two people cannot receive the same seat. Invariant 4 may lag by seconds, because it is a derived counter and a brief undercount harms nobody.

The error most readers make here is treating all four as equally urgent, which forces everything into one aggregate and destroys write throughput.

Block 3. PURPOSE: choose boundaries using the arithmetic, not the aesthetics.

Invariant 1 forces seat level atomicity. Adjacency requirements force a boundary larger than one seat. Using the numbers from earlier in this module (2,000 holds per second at peak, 5 milliseconds per transaction, 200 serialised writes per instance), a section of 200 seats gives 200 instances and 40,000 writes per second of headroom, which is twenty times the requirement.

Invariant 4 becomes a projection updated from a SeatSold event, not a field inside the Section aggregate. It is allowed to be a few seconds stale, and I say that out loud with the number attached.

Block 4. PURPOSE: turn the model into an operation list, so the interface is derived rather than invented.

• Section aggregate: hold seats and release hold as named operations, because both enforce invariant 1 and both are transitions rather than field edits.
• Order aggregate: create, read, cancel.
• Event: read only through this contract, because partners do not create our events.
• Sold count: a read field on the event, documented as eventually consistent with a stated bound of a few seconds.

The error most readers make here is exposing a patch operation on a seat with a writable status field. It lets a caller write "sold" with no order behind it, which makes invariant 1 unenforceable at the contract level even if the database still holds.

Result. Four invariants produced three aggregates, six operations and one deliberately stale projection. Nothing in the operation list was invented from the prompt's nouns, and every field a caller can write traces to an invariant that permits it.

3e. Fade the scaffold

Faded example 1. Prompt: "Model the domain behind a food delivery interface used by restaurants." Blocks 1 to 3 are given. Produce block 4, the operation list.

• Block 1: entities are Restaurant, MenuItem, Order, OrderLine, Courier. Value objects are Money, Address, TimeWindow.
• Block 2: invariants are that an Order never contains an unavailable MenuItem at acceptance time, that an accepted Order has exactly one Courier assignment at a time, and that a Restaurant's daily order count is reportable within a minute.
• Block 3: aggregates are Restaurant with its menu, and Order with its lines and its current assignment. The daily count is a projection.

Faded example 2. Prompt: "Model the domain behind a payroll interface a partner uses to run payroll for its own customers." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: entities are Employer, Employee, PayRun, Payslip, Payment. Value objects are Money, TaxCode, PayPeriod.
• Block 2: invariants are that a PayRun is immutable once submitted, that the sum of Payslip nets equals the PayRun total, and that an Employee has exactly one active TaxCode per PayPeriod.

3f. Check yourself

Q1. A candidate says "I will make the whole event one aggregate so that seat allocation is simple." Give the one sentence arithmetic rebuttal, then the design change.

Q2. You are told "an order's total must always equal the sum of its lines, and an order's lines reference products whose prices change daily". Where does the invariant live, and what does that force into the contract?

3g. When not to use this

Aggregate modelling has a real cost: it forces eventual consistency between boundaries, and every boundary crossing becomes an event, a compensation and an observable intermediate state.

Module 4. Resource, procedure or event: three ways to expose one domain

Time: 25 to 30 minutes reading, 20 to 25 minutes practice.

4a. Predict first

The same domain change, "a customer cancels an order that has not shipped", exposed three ways.

The three styles, defined

None of these is a protocol choice. All three can be carried over the same transport, and module 5 handles transport separately. This module is only about which unit you hand the caller.

The selection rule, in one question per row

Ask these in order and stop at the first row that answers yes.

The interview relevant sentence: a resource model is the default because it is cheap to read, cache and evolve, and every named procedure you add is a small admission that the resource model did not fit. Adding three or four such admissions is normal and healthy. Adding twenty means the resources were chosen from nouns rather than from aggregates.

Chattiness has a number attached

Inputs stated here: a mobile order detail screen needs four things (the order, the customer, the line items with product names, the shipment status), and the assumed round trip time to a regional edge is 120 milliseconds. That figure is an assumption, and it is reasonable as a round number for a mobile client on a typical cellular connection to a nearby edge.

The 480 millisecond figure is what kills naive resource models on mobile, and it comes entirely from dependency, not from the number of calls. If the client can issue all four at once, resource style costs one round trip and the argument disappears. The design question is therefore not "how many resources" but "how many of them can the client address without first reading another".

Events change the integration count, not the latency

Inputs: 3 systems produce order changes, 6 systems need to react (billing, warehouse, email, analytics, fraud, support).

At 6 consumers the broker halves the integration count. At 2 consumers it is 3 x 2 = 6 versus 3 + 2 = 5, which is not worth a broker, an ordering story and a dead letter queue. The crossover in this arithmetic sits at roughly 3 consumers per producer, which is the threshold worth saying out loud rather than asserting that events are cleaner.

+----------+   +----------+   +----------+
| Producer |   | Producer |   | Producer |
+----------+   +----------+   +----------+
      |              |              |
      +-------+------+------+-------+
              |             |
              v             v
        +-----------------------+
        | Event log             |
        | ordered per order id  |
        +-----------------------+
           |      |       |
           v      v       v
     +--------+ +-----+ +---------+
     | Billing| |Email| |Analytics|
     +--------+ +-----+ +---------+

Three producers and three consumers through a log. Direct calls
would need 9 links here and 18 at six consumers

The hybrid that is usually correct

Most real contracts are a resource spine with a small number of named transitions and an event stream beside them.

• Resources for the things the caller reads, lists, caches and holds.
• Procedures for transitions that carry preconditions, arguments or side effects the caller must be told about.
• Events for everything the caller does not need to wait for and other systems must observe.

Saying this out loud is worth more than picking one style purely, because a candidate who insists on pure resource modelling has to smuggle transitions in as status writes, and one who insists on pure procedures loses caching and discoverability.

4d. Worked example

Prompt: "Design how a partner publishes a job posting on our board. Postings are reviewed before they go live, and three internal systems care when a posting goes live."

Block 1. PURPOSE: separate the things from the transitions, because that split decides the style per operation.

The thing is a JobPosting. The transitions are submit for review, approve, reject, publish, expire. Only the first is caller triggered from outside.

I say: "Reads and drafts are resources. Everything with a precondition is a named operation. Going live is an event, because three systems care and none of them can block the partner."

The error most readers make here is exposing a state field on the posting and letting the partner write it. That hands the state machine to the caller and makes every illegal transition indistinguishable from a typo.

Block 2. PURPOSE: shape the resource part so the common reads are one round trip.

The partner's dashboard shows postings with their review state and applicant counts. Rather than a posting read plus a review read plus a counts read, I return review state as an embedded object on the posting and applicant counts as a separate collection, because counts change constantly and would ruin caching on the posting.

Using the round trip arithmetic above, embedding the review state removes one dependent call, which at an assumed 120 milliseconds per round trip is 120 milliseconds off the dashboard's critical path.

The error most readers make here is embedding everything, including the volatile counts, which makes the posting representation uncacheable and forces a fresh read on every poll.

Block 3. PURPOSE: give the transition a name, arguments and a refusal reason.

submitForReview takes a posting id and an optional note. It refuses with a specific code when the posting is missing required fields, and with a different code when it is already under review. Two codes, not one, because the caller's next action differs: fix and resubmit, versus wait.

The error most readers make here is a single generic validation error, which forces the partner's engineer to parse a human message to decide whether to retry.

Block 4. PURPOSE: publish the event once and let consumers pull what they need.

PostingPublished carries the posting id, the employer id, the publish timestamp and a version. It does not carry the posting body. Consumers read the body if they need it, which keeps the event small and stops the event schema from becoming a second copy of the resource schema that must be evolved in step.

With 3 consumers and 1 producer, direct calls would be 3 links versus 4 with a broker, so the broker is not justified by integration count alone here. It is justified by the requirement that a slow consumer must not block publishing, which is a different argument, and I state which argument I am using.

Result. One resource, two named transitions exposed externally, one event. Every style choice traced to a property of the operation rather than to a preference, and the one place where the usual arithmetic did not justify the broker was called out rather than glossed.

4e. Fade the scaffold

Faded example 1. Prompt: "Design how a warehouse system tells our order service that a shipment left the building. Two other systems (customer email and analytics) also care." Blocks 1 to 3 are given. Produce block 4.

• Block 1: the thing is a Shipment. The transition is dispatch. It carries a carrier, a tracking number and a timestamp.
• Block 2: Shipment is readable by order id, and the tracking number is embedded because it never changes after dispatch.
• Block 3: markDispatched is a named operation with two refusal codes, already dispatched and shipment cancelled.

Faded example 2. Prompt: "Design how a customer changes the delivery address on an order that may already be in a picking queue." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the thing is an Order with a shipping Address value object. The change is not a simple field write, because the picking stage may have already printed a label.
• Block 2: the Address is embedded on the Order, and the Order carries a fulfilment stage field that is read only to the customer.

4f. Check yourself

Q1. A candidate proposes 22 named procedures and no resources for a document management interface. Give the strongest single objection, and the specific thing you would change first.

Q2. Your event stream carries OrderUpdated with the full order body. A consumer complains that it cannot tell what changed. What is the design error, and what are the two candidate fixes?

4g. When not to use this

Each style carries its own over-engineering threshold.

The measurement that decides between chattiness fixes is specifically dependent round trips per screen. Count them in a trace. If the count is 1 or 2, composing endpoints buys nothing and costs you a cache.

Module 5. Style selection: representational, graph and binary

Time: 25 to 30 minutes reading, 20 to 25 minutes practice.

5a. Predict first

The same list screen, fifty rows, requested three ways. Assume each full entity serialises to 4 kilobytes of JSON, and the screen actually displays four fields per row that total about 400 bytes.

The comparison, on the axes that actually decide it

The bandwidth number, worked

Continuing the inputs above (200 KB versus 20 KB versus 7 KB) and adding one more: an assumed 1.5 megabits per second of effective mobile downlink, which is a deliberately pessimistic round number for a cellular connection under load.

The gap between A and B is about one second on a screen a user is watching, which is the argument that wins style debates. The gap between B and C is 70 milliseconds, which is below the threshold most users notice, so choosing binary purely for bytes on a user facing list is not justified by this arithmetic. Choose it for streaming or for internal service call volume instead.

The caching number, worked

Inputs: a list endpoint receiving 1,000 requests per second, of which an assumed 70 percent are repeats of an identical query within any 60 second window. That assumption is reasonable for a popular list that most users see unpersonalised.

• Representational, cacheable at an edge with a 60 second freshness window: the edge serves 700 requests per second and the origin sees 300.
• Graph or binary, not cacheable by an intermediary: the origin sees all 1,000.

That is a factor of 3.3 in origin load from the style choice alone, before any code is written. It is also the honest counterweight to the bandwidth table, and a strong answer states both numbers rather than one.

+---------+  identical GETs  +-------+  misses  +--------+
| Clients | ---------------> | Edge  | -------> | Origin |
| 1000 s  |                  | cache |  300 s   |        |
+---------+                  +-------+          +--------+
                                 |
                             700 s served
                             from cache

+---------+  opaque query bodies        +--------+
| Clients | --------------------------> | Origin |
| 1000 s  |                             | 1000 s |
+---------+                             +--------+

Same traffic under a cacheable read style and an opaque query
style. The second diagram has no cache box because there is
nothing an intermediary can key on

Each style's characteristic production failure

This is the part that separates a comparison table from an opinion worth hiring.

The graph failure deserves the extra sentence: because the caller composes the query, a single client change can multiply datastore load without any server deployment, and the first sign is usually a shard at 100 percent with no correlated release.

5d. Worked example

Prompt: "We have a mobile app, a partner API used by 400 integrators, and eleven internal services. Pick the interface styles."

Block 1. PURPOSE: refuse the premise that this is one decision, because three audiences have three cost functions.

I say out loud that there are three surfaces and I will decide each separately, then check they can share a domain layer. Deciding one style for all three is the most common wrong answer here, and it is wrong in a specific way: whichever style you pick, one of the three audiences pays.

The error most readers make here is choosing a favourite style and then justifying it three times.

Block 2. PURPOSE: price the partner surface first, because it is the one I cannot change later.

400 integrators means every breaking change is 400 conversations. The dominant costs are tooling, documentation and stability, not bytes. Representational style wins on those axes, and the bandwidth penalty from the earlier table is irrelevant when the caller is a server on a fast link.

I add sparse field selection to the list reads, which recovers most of the over-fetch cost without giving the caller unbounded query power.

Block 3. PURPOSE: price the mobile surface on latency and battery, not on elegance.

The mobile app has four dependent reads on its main screen, which the earlier arithmetic prices at 4 x 120 = 480 milliseconds. The two candidate fixes are a graph style, or two deliberate composite reads in the representational style.

I choose the composite reads, and I say why: with one first party client, the flexibility of a graph style buys little, while the unbounded query cost buys a real operational risk. If the app had many screens changing weekly and a separate team shipping them, I would flip that decision, and I say that too.

The error most readers make here is adopting a graph style because the over-fetch table looks decisive, without pricing the operational control it now requires: depth limits, cost budgets and resolver batching are not optional extras.

Block 4. PURPOSE: price the internal surface on call volume and schema discipline.

Eleven services with high call volume, all deployed by us, no browser in the path. Binary schema based calls win: smaller payloads, generated stubs, and native streaming for the two paths that need it.

The cost I accept is schema distribution, so I name the control immediately: a registry, compatibility checks in continuous integration, and a rule that retired field tags are reserved and never reused.

Result. Three surfaces, three styles, one domain layer behind them. The answer that scores badly here is not a wrong style, it is a single style asserted for all three without pricing what each audience loses.

5e. Fade the scaffold

Faded example 1. Prompt: "A dashboard product where every customer configures their own widgets, and each widget needs a different slice of the same twelve entities." Blocks 1 to 3 are given. Produce block 4, the recommendation and its controls.

• Block 1: one audience, first party web client, no partner surface.
• Block 2: widget shapes change per customer, so the server cannot enumerate the needed field sets ahead of time.
• Block 3: bandwidth analysis shows a full-entity approach sends about 8 times the displayed bytes on a typical dashboard.

Faded example 2. Prompt: "An internal service that returns a 2 kilobyte risk score object, called 40,000 times per second by four services we own." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: internal only, no browser, we deploy all callers.
• Block 2: the response shape is fixed and small, and the call is on a latency critical path.

5f. Check yourself

Q1. Your partner API is representational. An integrator complains that building their dashboard takes nine calls. Give two fixes and state which you would ship first and why.

Q2. A team proposes moving a public partner interface from representational to binary to save bandwidth. What is the strongest counter argument, and what measurement would settle it?

5g. When not to use this

The general failure mode is adopting a style for its best case and inheriting its characteristic failure without the matching control. If you cannot name the control in the same sentence as the choice, you are not ready to make the choice.

Module 6. Contract first: the schema as the design artefact

Time: 20 to 25 minutes reading, 15 to 20 minutes practice.

6a. Predict first

Here is a proposed change to a shipped response schema, written as a diff.

  Booking
    id                string
-   guest_name        string
+   guest             object
+     first_name      string
+     last_name       string
    nights            integer
-   status            string
+   status            enum  CONFIRMED HELD CANCELLED PENDING_REVIEW
+   total_amount      Money  required

The two rules that generate every compatibility answer

• Readers must ignore fields they do not recognise. A client that fails on an unknown field converts every additive server change into an outage.
• Writers must never be required to send something new. A server that adds a required field breaks every caller that has not shipped yet, which is all of them.

Together these give the asymmetry that most compatibility tables merely list: additions are safe going out and dangerous coming in. Say it once at the whiteboard and you can derive the rest live.

The change taxonomy, derived from those two rules

That last row is the change no automated checker catches. Repurposing status from a shipment state to a payment state passes every schema linter and every generated stub, and breaks every consumer at runtime. The control is a review rule, not a tool.

Why the schema is the artefact and not the documentation

Three consequences follow from writing the schema before the implementation, and each is worth stating in the round.

The direction matters and candidates rarely name it:

• A public response surface needs backward compatibility: new servers must keep old readers working.
• A request surface with many old writers needs forward compatibility: new servers must keep accepting old requests.
• An event stream usually needs both, because consumers and producers upgrade independently and a replayed old event must still parse.

The number that makes this concrete

Inputs: 5 million mobile installs, and an assumed client upgrade half life of 90 days, meaning half the remaining installs on any version upgrade every 90 days. That assumption is reasonable for an app users open weekly with automatic updates enabled, and it should be labelled as an assumption when you use it.

A breaking response change shipped six months after the client release strands 1.25 million installs. That number, not a principle, is what justifies the cost of a schema registry and a compatibility gate. It is also the number that decides sunset dates in module 11.

+------------------+     +------------------+
| Interface        | --> | Generated server |
| definition file  |     | stubs and types  |
+------------------+     +------------------+
        |    |
        |    +--------> +------------------+
        |               | Generated client |
        |               | libraries        |
        |               +------------------+
        v
+------------------+     +------------------+
| Lint rules       |     | Compatibility    |
| naming shape     |     | check against    |
| pagination       |     | last released    |
+------------------+     +------------------+
        |                        |
        +-----------+------------+
                    v
             +-------------+
             | Build fails |
             | on breach   |
             +-------------+

The definition is the input to generation and to two gates, so a
breaking change stops in the pipeline rather than at a partner

6d. Worked example

Prompt: "Review this proposed change to our shipped booking schema and decide what ships." The diff is the one at the top of this module.

Block 1. PURPOSE: classify each change by direction, because a single verdict for the whole diff is always wrong.

I take the four changes and place each in the taxonomy: the guest split breaks both directions, the enum addition breaks readers, the enum narrowing breaks writers, the required field breaks writers.

I say the count out loud: four changes, three distinct breakage directions, so this cannot ship as one release regardless of how it is versioned.

The error most readers make here is verdicting the diff as "breaking" and stopping. The useful output is a per change plan, because two of these four can ship today.

Block 2. PURPOSE: convert each breaking change into a non breaking sequence, because the sequence is the actual deliverable.

• Guest split: add guest alongside guest_name, populate both, mark guest_name deprecated in the definition, remove it only after the sunset date.
• Required total_amount: add it as optional, populate it on every response immediately, and require it on requests only for clients that opt into a newer contract.
• Enum narrowing: keep accepting the old free string on writes, reject unknown values only after measurement shows nobody sends them.

The error most readers make here is treating "add both fields" as a hack. It is the standard expand and contract sequence, and naming it as deliberate rather than apologetic is part of the answer.

Block 3. PURPOSE: handle the enum addition, which has no non breaking sequence, by fixing the contract rather than the change.

PENDING_REVIEW cannot be added safely to clients that switch exhaustively. There is no expand and contract path, because the value itself is the change.

So the fix is a documented rule going forward: clients must treat unknown enum values as a stated fallback, and the definition ships a permanent UNKNOWN member so that generated code has somewhere to land. For clients already in the wild, I gate the new value behind a request flag until the install base has turned over, using the half life table to pick the date.

Block 4. PURPOSE: leave behind a rule so the next diff does not need this meeting.

Three lint rules go into the pipeline: no field removals without a deprecation marker older than the sunset window, no new required request fields, and no enum additions on response types that lack a documented fallback.

Result. Two changes ship immediately, one ships as an expand and contract sequence with a dated removal, and one is gated behind an install base threshold with a number attached. The output of a good schema review is a schedule, not a verdict.

6e. Fade the scaffold

Faded example 1. A proposed diff renames created to created_at across nine response types for consistency, and the lint rule that flagged it is the naming rule the team just adopted. Blocks 1 to 3 are given. Produce block 4.

• Block 1: a rename is a remove plus an add, so it breaks every reader of all nine types.
• Block 2: the expand and contract sequence is to emit both fields, deprecate the old one, and remove it after the sunset window.
• Block 3: with an assumed 90 day upgrade half life and a stated policy of stranding fewer than 5 percent of installs, the removal cannot happen for about 390 days, since 0.5 raised to the power 390 over 90 is roughly 0.05.

Faded example 2. An event schema needs a new field that consumers must use to route messages correctly. Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: events need both directions of compatibility, because producers and consumers upgrade independently and old events are replayed from the log.
• Block 2: adding an optional field is safe in both directions, but "consumers must use it" is a semantic requirement that no schema rule can enforce.

6f. Check yourself

Q1. A colleague argues that a schema registry with build time compatibility checks is bureaucracy for a five person team. Give the measurement that decides it.

Q2. Which single schema change is invisible to every automated compatibility checker, and what is the control?

6g. When not to use this

The honest threshold: contract first is a tax paid to buy independent deployability. If nothing deploys independently, you are paying the tax and receiving nothing.

Module 7. Error model design: the part candidates visibly fail

Time: 25 to 30 minutes reading, 20 to 30 minutes practice.

7a. Predict first

The same failure, "the payment method was declined by the issuing bank", returned three ways.

The three questions every error must answer

Two supporting fields carry their weight in every real interface: a request identifier the caller can quote in a support ticket, and a documentation link keyed by the code.

The taxonomy, on axes rather than as a list

Most error catalogues are a flat list of codes. Build it on three axes instead, because the axes are what a caller branches on.

The row candidates most often get wrong is 409 versus 422. A conflict means the request is valid but the current state forbids it, so re-reading and retrying can work. A validation failure means the request itself is wrong, so retrying it unchanged never works. Collapsing both into 400 destroys that distinction and is the single most common error model defect.

Stable codes and unstable messages

That last row deserves a sentence. Returning 200 with {"success": false} breaks every intermediary that reasons about status: retries, circuit breakers, monitoring, load balancer health logic. It is a self inflicted wound that shows up as invisible failures on dashboards that look green.

Retry amplification, with the arithmetic

Inputs, stated here: a service receiving 10,000 requests per second, clients configured to retry up to 3 times, and no retry budget.

The important row is the last one. The service that just failed now receives sixteen times its normal traffic, which is why a partially degraded dependency so often becomes a total outage. Nothing about this is exotic: it is 4 raised to the number of retrying layers.

Two controls follow directly, and both belong in the contract rather than only in the client library:

• A retryability flag, so callers never retry a permanent failure. That removes the 4x on the entire class of caller errors.
• A retry budget, for example retries capped at 10 percent of the successful request rate, which bounds amplification at 1.1x regardless of failure rate.

Jitter matters too, but it changes the shape of the load rather than the total, so state it as a separate control rather than as the fix.

+---------+   1 try  +----------+   1 try   +----------+
| Client  | -------> | Gateway  | --------> | Service  |
+---------+          +----------+           +----------+

fully down with 3 retries at each layer

+---------+ 4 tries  +----------+ 16 tries  +----------+
| Client  | -------> | Gateway  | --------> | Service  |
+---------+          +----------+           +----------+
                                            | 160k per s
                                            +----------+

Retry amplification multiplies per layer. Two retrying layers
turn 10k per second into 160k against a service already failing

Partial failure in batch operations

A batch operation has three possible contracts, and picking one silently is how interfaces get support tickets.

For the independent case, three details decide whether integrators can actually use it:

• The overall transport status is not 200 with a success body. Use a status that says "this was processed, look inside", and never a status that says "everything worked".
• Every item result carries the client's own key for that item, not just an index, because the caller may have reordered or retried a subset.
• Item errors use the same code vocabulary as single item operations, so the integrator writes one error handler and not two.

7d. Worked example

Prompt: "Design the error model for an authorisation call on a payments interface."

Block 1. PURPOSE: enumerate the failure sources before the codes, because the sources decide the classes.

Four sources: the caller's request is wrong, the caller's credentials or permissions are wrong, the card or its issuer refuses, and our own path fails including the downstream gateway.

I map them to classes immediately: caller fixable and permanent, caller fixable and permanent, business outcome, and ours plus transient. The third one is the interesting one, and I say why out loud: a decline is not an error in our system, it is a successful call with a negative business result.

The error most readers make here is putting declines in the same bucket as validation failures, which makes an integrator treat "insufficient funds" as a bug in their code.

Block 2. PURPOSE: decide the decline representation, since it is the highest volume non success outcome.

I choose a distinct status for declines rather than folding them into a generic client error, and I carry two fields: a stable top level code, and a narrower decline_reason from a documented set.

The reason for two fields rather than one is evolution. Issuers invent new decline reasons regularly, and new reasons must be addable without breaking a client that branches on the top level code. So the top level set is small and frozen, the narrow set is documented as extensible with an explicit fallback value.

The error most readers make here is exposing the raw issuer response code, which welds a third party's vocabulary into your contract forever.

Block 3. PURPOSE: state retryability per class rather than per code, so the caller's logic stays small.

• Caller errors: retryable: false, no retry-after.
• Declines: retryable: false at the interface level, even where a human might retry later with another card. The distinction is that our contract cannot promise a different outcome.
• Rate limits: retryable: true with an explicit delay.
• Our faults and gateway timeouts: retryable: true, and only safe because every authorisation carries an idempotency key, which module 9 covers.

I say the dependency explicitly: retryable on a money moving operation is a lie unless idempotency is designed, and the two features have to ship together.

Block 4. PURPOSE: bound the blast radius, because a payments interface degrading is the scenario that matters.

With clients retrying three times and no budget, the arithmetic above gives 4x amplification per layer against a gateway that is already failing. So the contract states a retry budget, returns retry-after on 429 and 503, and documents jittered backoff as required client behaviour rather than a suggestion.

Result. Four classes, roughly nine top level codes, one extensible narrow set, one retryability rule per class, and a stated dependency on idempotency. The thing that earns a senior rating is not the code list, it is naming which codes a client may retry and proving that retrying them is safe.

7e. Fade the scaffold

Faded example 1. Prompt: "Design the error model for a bulk contact import that accepts up to 10,000 contacts per request." Blocks 1 to 3 are given. Produce block 4.

• Block 1: failure sources are malformed request envelope, per contact validation failures, duplicate detection against existing records, and our own faults.
• Block 2: the envelope failures are all-or-nothing, and the per contact failures are independent.
• Block 3: the response is a per item array keyed by a client supplied client_ref on each contact.

Faded example 2. Prompt: "Our public interface returns 400 for every client side problem and 500 for everything else. Two integrators have filed tickets about retry behaviour." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the two ticket patterns are integrators retrying permanent failures forever, and integrators not retrying transient ones at all.
• Block 2: the root cause is that a single status carries three different meanings, so no branch can be written.

7f. Check yourself

Q1. An interviewer asks why you would ever return 409 rather than 400. Answer in two sentences, then name what the caller does differently.

Q2. Your service returns retryable: true on a 503. A client retries three times with no jitter, and so does the gateway in front of you. Quantify the load on a fully failed service and name the two controls.

7g. When not to use this

The threshold worth remembering: an error model earns its complexity from the number of independent teams writing handlers against it. One team, five codes. Four hundred integrators, a taxonomy with a documented fallback and a stability policy.

Module 8. Pagination: stability, cost and the failure nobody tests

Time: 20 to 25 minutes reading, 15 to 20 minutes practice.

8a. Predict first

A feed sorted newest first, paginated with offset and limit. Page size is 20. A reader takes 3 seconds to read a page before requesting the next. New items arrive at 10 per second.

The three schemes, on the axes that decide it

An opaque cursor is keyset pagination with the state encoded and signed, so the caller cannot construct one by hand. That distinction matters more than it sounds, because a cursor the caller can author is a cursor you can never change.

The deep pagination number

Inputs: a table with 10 million rows matching the filter, an index that covers the sort, and an assumed 100 nanoseconds to walk one index entry that is already in memory. That assumption is reasonable as a round number for a cached B-tree traversal step, and it is the one number to label if you use it at a whiteboard.

The shape is the point: offset cost grows linearly with how deep you are, and keyset cost is flat. A crawler walking the whole collection with offsets does total work proportional to the square of the collection size, which is why the endpoint that was fine for a year falls over the week a partner writes an export script.

The cursor's contents, and why each part is there

The tiebreaker row is the one candidates miss, and it is worth deriving out loud. With a sort on created_at alone and 500 rows sharing one timestamp from a bulk import, "give me rows where created_at is less than X" either returns the same block forever or jumps past all 500, depending on the comparison. The fix is comparing the pair, sort key first and identifier second.

+-------------------------------------------+
| Page request                              |
|  cursor  sort key 2026-08-01T10 id 8842   |
|  limit   20                               |
+-------------------------------------------+
                  |
                  v
+-------------------------------------------+
| Seek where sort key id pair is less than  |
| the cursor pair  then take 21 rows        |
+-------------------------------------------+
                  |
                  v
+-------------------------------------------+
| Return 20 rows plus next cursor built     |
| from row 20  the 21st row only answers    |
| has more                                  |
+-------------------------------------------+

Keyset paging. Fetching limit plus one row is how has-more is
answered without a count query

Total counts, priced honestly

Using the earlier inputs, an exact count over 10 million matching index entries at 100 nanoseconds each costs about 1 second of work per request, which is why exact totals quietly become the most expensive part of a list endpoint. Offering a capped count instead is a one line contract decision that removes that cost entirely.

8d. Worked example

Prompt: "Design pagination for comment threads. Comments nest up to three levels, threads on popular posts reach 50,000 comments, and people comment while others are reading."

Block 1. PURPOSE: pick the sort before the mechanism, because the sort decides what a cursor can even contain.

Two sorts are wanted: newest first, and a ranked "top" order. Newest first is a stable monotonic key plus an identifier tiebreaker, so keyset works directly.

Ranked order is not stable at all, because a score changes as votes arrive. I name that immediately rather than discovering it in block 3.

The error most readers make here is designing one pagination mechanism and assuming both sorts can use it.

Block 2. PURPOSE: make the unstable sort stable by freezing something, because you cannot page a moving order.

For ranked order I snapshot the ranking at first page request and put the snapshot identifier in the cursor. Later pages read against that snapshot, so a comment cannot move from page 4 to page 2 and vanish.

The snapshot expires, and I state the number: an assumed 10 minute lifetime, which is reasonable because it comfortably exceeds a normal reading session while keeping storage bounded. An expired cursor returns a specific error telling the client to restart from page one.

The error most readers make here is silently restarting the scan when the snapshot expires, which shows the reader page one again with no explanation.

Block 3. PURPOSE: handle nesting without turning one screen into hundreds of calls.

Top level comments paginate with keyset. Each returns its first 3 replies inline plus a reply count and a reply cursor. Deeper replies paginate on demand.

The arithmetic that justifies inlining 3: a page of 20 top level comments where each needs a separate reply call is 21 requests, and at the assumed 120 milliseconds per round trip from module 4 that is 2.5 seconds if serialised. Inlining the first three replies covers the large majority of threads in one request, and I would measure the reply count distribution to confirm the 3 rather than assert it.

Block 4. PURPOSE: state the failure modes in the contract, so integrators are not surprised.

Three statements go in the documentation: cursors expire after 10 minutes and return a named error, new comments arriving during a newest-first scan appear only on a fresh first page, and total thread counts are capped rather than exact above 1,000.

Result. Two sorts, two mechanisms, one snapshot with a stated lifetime, and three documented failure behaviours. The senior signal here is not choosing cursors, it is noticing that the ranked sort cannot be paginated by the same means and saying so before being asked.

8e. Fade the scaffold

Faded example 1. Prompt: "Paginate an audit log that is append only, never deleted, and queried by administrators filtering on actor and date range." Blocks 1 to 3 are given. Produce block 4.

• Block 1: the natural sort is the append sequence, which is monotonic and unique, so it is its own tiebreaker.
• Block 2: keyset on the sequence number, with the filter hash in the cursor.
• Block 3: no snapshot needed, because appends only ever arrive at one end and deletions never happen.

Faded example 2. Prompt: "A partner exports our entire 40 million row catalogue nightly through the public list endpoint." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: today the endpoint uses offset and limit with a page size of 100.
• Block 2: the partner's export therefore issues 400,000 requests, the last of which uses an offset near 40 million.

8f. Check yourself

Q1. Give the two failure modes of offset pagination on a mutating collection, and say which one is worse and why.

Q2. A cursor is built from created_at only. A bulk import writes 500 rows with an identical timestamp. What happens, and what is the one line fix?

8g. When not to use this

The strongest form of this answer names when pagination is the wrong tool entirely: a caller who wants the whole collection every night needs an export or a change feed, and paginating them is optimising the wrong operation.

Module 9. Idempotency: making retries safe on operations that move money

Time: 30 to 35 minutes reading, 25 to 35 minutes practice.

9a. Predict first

A client charges a card. Here is the log from both sides.

The definitions, kept separate

The last row is worth saying out loud in a round. There is no exactly-once delivery over an unreliable network, because the acknowledgement can be the message that is lost. What you build is at-least-once delivery plus a deduplication record, and calling that "exactly once processing" is fine as long as you can say where the deduplication state lives.

Method level idempotency, and what it does not cover

The last two rows are the ones that catch people. balance = 100 is idempotent, balance += 5 is not, and the two look nearly identical in a request body. If your update format allows relative operations, every one of them needs a key.

The key's lifecycle

An idempotency key is not a flag, it is a small state machine with a record behind it.

             +-----------------+
 first call  | NEW             |
 ----------> | record created  |
             | lock held       |
             +-----------------+
                     |
        +------------+------------+
        |                         |
        v                         v
+----------------+       +------------------+
| COMPLETED      |       | FAILED           |
| response saved |       | terminal error   |
+----------------+       | saved            |
        |                +------------------+
        | replay                  |
        v                         v
+-------------------------------------------+
| Later identical call returns saved         |
| response  concurrent call gets 409 retry   |
+-------------------------------------------+

Key states. The saved response is what makes a replay identical
rather than merely harmless

Four rules make it correct, and each maps to a failure it prevents.

That last rule is the one that separates a correct implementation from a plausible one. If the key is written to a separate store after the charge succeeds, there is a window in which a crash leaves the charge without its key.

Sizing the key store

Inputs, stated here: 500 state changing writes per second sustained, a 24 hour key lifetime, and an assumed 300 bytes per record. The 300 bytes is a rough sum of a 36 byte key, a 32 byte fingerprint, roughly 200 bytes of saved response and some metadata, and it is reasonable for a small creation response.

13 gigabytes is small, which is the honest conclusion: idempotency keys are cheap at this scale and the reason people avoid them is complexity, not storage. The number that actually needs care is the lifetime.

Choosing the lifetime. It must exceed the caller's maximum retry window. If clients retry with exponential backoff up to 6 hours, a 24 hour lifetime gives a factor of 4 in hand. Setting it to 1 hour would silently reintroduce double charges for any client whose queue backs up, and that failure appears only during an incident, which is exactly when you cannot afford it.

9d. Worked example

Prompt: "Design idempotency for a card authorisation operation on a public payments interface."

Block 1. PURPOSE: decide who generates the key, because getting this wrong makes the whole mechanism useless.

The caller generates it, once per logical purchase, and reuses it on every retry of that purchase. I state the two failure shapes explicitly: a caller that generates a new key per HTTP attempt gets duplicate charges, and a caller that reuses one key across different purchases gets a stale response for the second purchase.

Because both failures are the caller's to make, the documentation must state the rule in one sentence and the interface must detect the second case.

The error most readers make here is letting the server generate the key. A server generated key cannot survive the response being lost, which is the only case that matters.

Block 2. PURPOSE: pin the conflict semantics, because they are where implementations quietly differ.

Three cases, three answers:

• Same key, same fingerprint, previous call finished: return the saved response, with a header saying it was a replay.
• Same key, different fingerprint: return a specific error. Never the old response, and never a fresh charge.
• Same key, previous call still in flight: return a conflict with a short retry-after, because a partial result cannot be returned safely.

The error most readers make here is returning the saved response for a mismatched fingerprint, which turns a caller bug into a silent wrong answer that reconciliation finds days later.

Block 3. PURPOSE: make the record and the money movement atomic, because a gap between them recreates the original bug.

The key record and the authorisation record commit in one transaction in the same store. If the gateway call itself sits outside that transaction, the sequence is: write the key row in state IN_PROGRESS, call the gateway, then commit the outcome onto the same row.

A crash between the gateway call and the commit leaves a row in IN_PROGRESS, so the recovery path must query the gateway by our own reference to learn what happened rather than assuming failure.

The error most readers make here is assuming a crashed request did nothing. In payments, the default assumption must be that it may have done everything.

Block 4. PURPOSE: size and bound the mechanism, so it is a design rather than an aspiration.

Using the module's inputs, 500 writes per second with a 24 hour lifetime and 300 byte records gives about 13 gigabytes, which is a single modest store. I set the lifetime at 24 hours against a documented maximum client retry window of 6 hours, and I state that the key is scoped per account so two customers cannot collide.

Result. Caller generated key, three conflict cases, one transaction, a stated lifetime with a stated margin, and a recovery path that assumes the work may have happened. The mechanism is roughly fifty lines of server code, and every one of those five decisions is where a candidate can be probed.

9e. Fade the scaffold

Faded example 1. Prompt: "Add idempotency to an operation that creates a document and also sends an email." Blocks 1 to 3 are given. Produce block 4.

• Block 1: the caller supplies the key, scoped per workspace.
• Block 2: the three conflict cases are handled as above.
• Block 3: the key record and the document write commit together.

Faded example 2. Prompt: "A partner reports occasional duplicate refunds. Their client retries on any 5xx, and our refund operation takes an idempotency key." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the key mechanism exists and appears correct in code review.
• Block 2: duplicates cluster during periods when our database is slow.

9f. Check yourself

Q1. A candidate says "we do not need idempotency keys because our create endpoint uses PUT with a client chosen id, which is idempotent." Is that right? State the condition under which it is.

Q2. Size the key store for 2,000 state changing writes per second with a 48 hour lifetime, and state the one number you would check before accepting your own answer.

9g. When not to use this

The cheapest correct answer is worth naming in a round: for many creation operations, a unique constraint on a caller supplied business identifier gives you idempotency with one line of schema and no new store. Reach for the full mechanism when you must return the original response, not merely prevent the duplicate.

Module 10. Concurrency and caching inside the contract

Time: 20 to 25 minutes reading, 15 to 20 minutes practice.

10a. Predict first

Two support agents edit the same customer record. The interface offers a read and a full replacement write, with no version checking.

The vocabulary, defined before use

The one mechanism that fixes both problems

A single token, returned on read and echoed on write, solves caching and lost updates at once.

Three token sources, and they are not equivalent:

Timestamps are the trap worth naming in a round. At an assumed one second resolution, any entity edited more than once per second has a validator that cannot distinguish versions, so the conditional write silently permits the lost update it exists to prevent.

+--------+  read              +--------+
| Client | <----------------- | Server |
|        |  body plus token 7 |        |
+--------+                    +--------+
     |
     |  write with must match token 7
     v
+--------+                    +--------+
| Server | ----------------->            token is now 8
| checks | current token is 8 | reject |
| token  |                    | 412    |
+--------+                    +--------+

Optimistic concurrency. The rejection is the feature, because the
client learns it was editing a version that no longer exists

How often will the rejection fire?

This decides whether "reject the loser" is a usable design or a support nightmare, and it is derivable.

Inputs: an edit session takes an assumed 30 seconds between the read and the write, which is reasonable for a human filling a form, and other edits to the same entity arrive independently at a rate of λ per second.

Probability that at least one other edit lands in the window is 1 minus e to the power of minus λ times 30, which for small values is close to λ times 30.

The design consequence is sharp. At 3.3 percent, rejecting the loser and asking the human to reapply is fine, and at 1,000 edits per day it produces about 33 retries. At 63 percent, rejection is unusable, and the interface needs either field level merge or a genuinely collaborative model with a different contract shape. Choosing between them without this number is guessing.

The caching number

Inputs: a 200 kilobyte representation, a not-modified response of an assumed 300 bytes of headers, a client polling every 30 seconds, and an assumed 95 percent of polls finding no change. The 95 percent is reasonable for a resource that changes a few times a day and is polled 2,880 times.

That is a factor of 20 for one header. What it does not buy is a saved round trip: the client still pays the network time on every poll, so if latency rather than bandwidth is the problem, conditional requests are the wrong tool and a change feed or a push channel is the right one.

Choosing a conflict policy

Field level merge deserves a warning: it is only safe when no invariant spans two fields. If a discount percentage and a total must agree, merging them independently produces a record that no single author ever intended, and the invariant check has to run after the merge rather than on either input.

10d. Worked example

Prompt: "Design the concurrency story for an interface where support agents and customers both edit an account record."

Block 1. PURPOSE: measure before choosing, because the policy follows from the collision rate.

I ask for two numbers: edits per account per hour, and how long an edit session lasts. The answers are roughly 4 per hour on active accounts and about 30 seconds of form filling.

Using the arithmetic above, 30 x 4 divided by 3,600 gives about 3.3 percent. That is low enough that rejecting the loser is acceptable, and I say the number out loud rather than asserting that optimistic concurrency is fine.

The error most readers make here is choosing optimistic or pessimistic concurrency from taste. The rate decides it, and the rate is one question away.

Block 2. PURPOSE: pick a validator that cannot silently fail.

I use a version integer on the account row, incremented in the same transaction as every write. I reject last modified timestamps explicitly, because at an assumed one second resolution two writes inside the same second share a validator, and that is precisely the case optimistic concurrency exists to catch.

I return the version on every read and require it on every write. A write without it is rejected rather than allowed, since an optional precondition is one that clients omit under deadline pressure.

The error most readers make here is making the precondition optional to be friendly to integrators. That converts a designed safety property into an opt-in, and the callers who skip it are the ones causing the incidents.

Block 3. PURPOSE: make the rejection actionable, because a bare conflict error moves the problem to the caller.

The 412 response carries the current version, the fields that changed since the caller's version, and who changed them. With that, a client can show "the tier was changed to premium by another agent 40 seconds ago" instead of "please try again".

The error most readers make here is returning the conflict with no current state, which forces a re-read and leaves the user with no idea what happened.

Block 4. PURPOSE: connect concurrency to caching, since one token serves both.

The same version token is used as the read validator, so agent dashboards polling every 30 seconds get not-modified responses. Using the module's numbers, that takes one polling client from 576 megabytes a day to about 29.

I state the limit honestly: this saves bytes, not round trips, so if the dashboard needs sub-second freshness the answer is a push channel and not a shorter poll interval.

Result. One version integer, required on writes, echoed on reads, a conflict response carrying the diff, and a measured collision rate that justified the policy. Every choice traced to a number the interviewer watched me derive.

10e. Fade the scaffold

Faded example 1. Prompt: "Add concurrency control to a collaborative pricing sheet where a dozen analysts edit different cells of the same document all afternoon." Blocks 1 to 3 are given. Produce block 4.

• Block 1: measured edits are about 200 per hour on a busy sheet, with sessions of a few seconds.
• Block 2: at a 5 second window and 200 edits per hour, λW is 5 x 200 / 3,600 = 0.28, so collision probability is 1 minus e^-0.28, about 24 percent.
• Block 3: 24 percent rejection would be intolerable for a document analysts edit continuously.

Faded example 2. Prompt: "Our mobile clients poll a settings resource every 10 seconds and our egress bill is dominated by it." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the representation is 40 kilobytes and settings change roughly twice a day.
• Block 2: there are 200,000 active clients at any moment.

10f. Check yourself

Q1. Explain in two sentences why last modified timestamps are unsafe as write preconditions, and give the situation where they are still fine for reads.

Q2. An entity has a 40 percent collision rate under optimistic concurrency. Name two structurally different fixes and the cost of each.

10g. When not to use this

The threshold that matters is the collision rate, and it is cheap to obtain: count writes per entity per hour in your existing logs and multiply by the edit window. Designing a merge strategy before taking that measurement is the classic over-engineering path here.

Module 11. Versioning, evolution and the clients you cannot upgrade

Time: 25 to 30 minutes reading, 20 to 30 minutes practice.

11a. Predict first

Four changes are proposed for a public interface currently at major version 2, with about 1,200 active integrations.

What this module adds to module 6

Module 6 gave the change taxonomy and the compatibility rules. This module answers a different question: the change is breaking, it must ship anyway, and some of your callers will never upgrade.

Four things have to be decided, and candidates typically decide only the first.

Versioning schemes, priced honestly

The recommendation worth defending: version the surface coarsely, evolve additively inside it, and treat a major version bump as a rare and expensive event. A team that bumps a major version to rename a field has chosen the most expensive mechanism for the cheapest problem.

The sunset date is derived, not chosen

Reusing the migration model from module 6: 5 million installs and an assumed 90 day upgrade half life, plus a stated policy of stranding no more than 5 percent of callers.

Solve for the time at which one half raised to the power of t divided by 90 equals 0.05. That is t equal to 90 times the base-2 logarithm of 20, which is 90 x 4.32, or about 389 days.

Two consequences to state out loud. First, a 12 month deprecation window is not generosity, it is arithmetic. Second, the tail never reaches zero, so at some point you switch from waiting to contacting the remaining callers individually, which is only possible if you can identify them per caller. That reporting requirement is a design decision made now, not later.

Keeping old callers working

The rule that keeps this from sprawling: transformation logic lives in exactly one layer, and the core implementation never contains a version check. Once if version == 2 appears inside the domain code, every future change costs more, and the code becomes unremovable because nobody can prove which branch is dead.

+----------+   +----------+   +----------+
| Caller   |   | Caller   |   | Caller   |
| pinned   |   | pinned   |   | current  |
| 2024-06  |   | 2025-03  |   |          |
+----------+   +----------+   +----------+
      |              |              |
      v              v              |
+-------------+ +-------------+     |
| Adapter to  | | Adapter to  |     |
| current     | | current     |     |
+-------------+ +-------------+     |
      |              |              |
      +------+-------+--------------+
             v
      +---------------+
      | Core domain   |
      | no version    |
      | branches      |
      +---------------+

Adapters absorb every past contract so the core has one shape.
Version checks inside the core are what make removal impossible

Consumer driven contract testing, and what it catches

A schema compatibility check knows the shape of your contract. It does not know which parts anyone uses. Consumer driven contract testing closes that gap: each consumer publishes the requests it makes and the response fields it depends on, and the provider's build verifies every published expectation before release.

That last row is the underrated one. The main reason interfaces accumulate dead fields is that nobody can prove a field is unused. Consumer contracts turn that from a guess into a query.

11d. Worked example

Prompt: "We must rename zip to postal_code across our public interface. There are 1,200 active integrations and we cannot force any of them to change."

Block 1. PURPOSE: reject the framing before designing, because the first question is whether this ships at all.

I state the cost in the same breath as the benefit. The benefit is naming consistency for international addresses. The cost, from the arithmetic above, is roughly 13 months of dual fields plus a removal campaign against a long tail.

I then propose the smaller change: apply the new name to new operations and new types only, and grandfather the existing ones with a documented exception. If the interviewer wants the full rename anyway, I continue, having shown that the trade was priced rather than assumed.

The error most readers make here is starting with the migration plan. The most senior move available is asking whether a cosmetic change justifies a year of dual maintenance.

Block 2. PURPOSE: choose the mechanism that does not create a second surface.

I do not bump the major version. A rename is a field level change, and a major bump forces 1,200 integrators to revalidate every operation to receive one renamed field.

Instead: emit both fields on responses, accept either on requests with a documented precedence rule when both are present and differ, and mark zip deprecated in the interface definition with a machine readable sunset date.

The error most readers make here is not defining what happens when a caller sends both fields with different values. It happens more often than people expect, usually from a client library merging defaults, and an undefined rule becomes an inconsistent one.

Block 3. PURPOSE: signal the deprecation in ways a machine can act on, not only a human.

Three signals: a deprecation marker plus a sunset date in the definition file, response headers carrying the same information on requests that touched the deprecated field, and per integrator usage reporting so we know exactly who is still sending it.

The third is the one that converts a broadcast email into a targeted conversation, and it only exists if requests are attributable per caller.

Block 4. PURPOSE: derive the removal date and the exit criteria, so the deprecation ends.

The date comes from the curve: with an assumed 90 day half life, reaching 5 percent takes about 389 days, so the published sunset is 13 months out and stated on the day the deprecation ships.

Exit criteria are two, not one: the date has passed, and per caller usage shows fewer than 5 percent of integrations still sending or reading zip. If the second is not met, the response is not an extension by default but a decision to contact the remaining callers, since an indefinite extension is how a deprecation becomes permanent.

Result. No major version, dual fields for 13 months, a precedence rule for the ambiguous case, three deprecation signals, a derived date and a named exit condition. The credit is in the date arithmetic and the exit criteria, because those are what make the deprecation finishable.

11e. Fade the scaffold

Faded example 1. Prompt: "Our default list operation returns cancelled bookings. We want it to exclude them by default." Blocks 1 to 3 are given. Produce block 4.

• Block 1: this is a semantic change with no schema diff, so no automated check catches it.
• Block 2: the affected callers cannot be identified from the schema, only from behaviour.
• Block 3: an explicit status filter is added first, so callers can express either intent before the default changes.

Faded example 2. Prompt: "A field we want to delete is used by an unknown number of the 1,200 integrations." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the schema check cannot tell you who reads a response field, because reading leaves no trace in the request.
• Block 2: consumer driven contracts exist for the 40 internal consumers but not for external integrators.

11f. Check yourself

Q1. Derive the sunset date for a client population with an assumed 60 day upgrade half life and a policy of stranding no more than 2 percent, then say what you do about the remaining 2 percent.

Q2. Give one change that a schema compatibility checker passes and a consumer driven contract test catches, and explain the mechanism.

11g. When not to use this

The honest summary: every mechanism in this module is a tax paid for the right to change something without coordinating with people you do not control. If you can coordinate, do not pay it. If you cannot, the sunset arithmetic tells you exactly how expensive the change is before you commit to it.

Module 12. Authentication, authorisation and multi-tenancy

Time: 26 to 32 minutes reading.

12a. Predict first

A token and a route, taken together. Commit to an answer before continuing.

+--------------------------------------------------------------+
| ACCESS TOKEN PAYLOAD          |  ROUTE                       |
|  sub:      user_9931          |   GET /tenants/{tenant_id}/  |
|  tenant:   acme               |       documents/{doc_id}     |
|  scope:    admin              |                              |
|  exp:      issued + 90 days   |  handler:                    |
|  no jti, no revocation list   |    load doc by doc_id        |
|                               |    if token.scope == admin:  |
|                               |       return doc             |
+--------------------------------------------------------------+
Caption: a long-lived bearer token and the handler that trusts
the path parameter rather than the token's tenant claim.

Question. Name the two independent ways a customer of tenant acme reads a document belonging to another tenant, and state the worst-case time between "we revoked this token" and "it stops working".

12b. Grants, token lifetimes and the two authorisation questions

Pick a grant for the caller in front of you.

Terms first. Authentication establishes who is calling. Authorisation decides what that caller may do. A principal is the entity a decision is made about: a user, a service, or a user acting through an application. A tenant is the isolation boundary that owns data.

The one sentence that separates a senior answer. Say who the principal is before naming a grant. If the answer is "a service acting on its own behalf", client credentials follows automatically. If the answer is "a service acting on behalf of a user", you need the user's identity to travel with the call, and you must decide whether it travels as a token exchange or as an assertion your service is trusted to make.

Token lifetime is an arithmetic problem, not a preference.

Short lifetimes buy revocation speed. They cost refresh traffic. Both sides are computable from two inputs, so this is never a matter of taste.

Stated inputs for the derivation, both labelled: assume 200,000 concurrently active sessions, which is the number an interviewer would give or you would assume aloud, and assume every session refreshes exactly once per access token lifetime.

Now the decision falls out. If a refresh is a database write plus a rotation record, 667 writes per second is a real system with its own availability risk sitting in front of every login. 56 per second is not. So a 60 minute access token is the default, and anything shorter needs a reason.

What to do about the resulting 60 minute lag, since "we revoked it and it works for an hour" is unacceptable for some operations but fine for most:

The last row is the opinionated one. Uniform token lifetimes are a false economy: the operations where a 60 minute lag is intolerable are usually under 2 percent of calls, so paying introspection cost on 100 percent of traffic to protect 2 percent is a straight overspend.

Refresh token rotation, and the part everyone omits. Rotation means each refresh returns a new refresh token and invalidates the old one. The value is not the rotation, it is the reuse detection: if an already-used refresh token appears, either the client raced itself or the token was stolen, and you cannot tell which. The correct response is to invalidate the whole token family and force re-authentication. Rotation without that response is bookkeeping.

Scopes, roles and attributes answer different questions. Three models, and the failure mode of each.

Scope explosion, computed. With R resource types and A actions per resource, a fully enumerated scope set has R times A members. At 20 resource types and 4 actions that is 80 scopes. A consent screen listing 80 items is a consent screen nobody reads, so the practical ceiling is around 10 to 15 groups: bundle by resource family and by read versus write, and reserve fine-grained control for the roles layer inside the tenant.

The layering that actually ships. Scopes bound what the application may attempt. Roles and attributes decide what this principal may do inside the tenant. Both must pass. Keeping them separate is what lets you answer the two different questions a support ticket asks: "why can this app do that" and "why can this person do that".

12c. Multi-tenancy: an isolation ladder with prices

Every rung isolates more and costs more. Pick the lowest rung that satisfies a stated requirement, and state the requirement.

The number that moves you up a rung. Two conditions, either sufficient. First, a contractual or regulatory requirement for physical separation or independent restore. Second, a single tenant whose share of a shared resource is large enough that its worst day is everyone's worst day.

A workable line for the second condition: if one tenant exceeds roughly 20 to 25 percent of a shared pool's capacity, it is no longer a tenant, it is a co-tenant of your infrastructure, and it should be moved to its own cell before its growth makes the move an emergency.

Make the predicate impossible to forget. Code review does not scale as a control. Three mechanisms, in increasing strength:

1. A repository layer where every query builder requires a tenant context object to construct, so an unscoped query does not compile.
2. Database row-level security with the tenant set from a connection-level variable, so a forgotten predicate returns zero rows instead of everyone's rows.
3. A test that runs every query in the suite twice, once as tenant A and once as tenant B, and fails if any query returns identical row identifiers.

The six cross-tenant leak classes, because a strong answer names them rather than saying "we check permissions":

The last one is why opaque, non-sequential identifiers on a public surface are a design choice and not a fashion.

+-----------+   +--------------+   +---------------------+
| CALLER    |-->| AUTHENTICATE |-->| RESOLVE TENANT      |
| token or  |   | verify sig,  |   | from token claim,   |
| api key   |   | check deny   |   | never from the path |
+-----------+   | list         |   +----------+----------+
                +--------------+              |
                                              v
              +---------------------+  +------------------+
              | POLICY DECISION     |->| DATA ACCESS      |
              | scope allows the    |  | tenant predicate |
              | action, role or     |  | injected by the  |
              | attribute allows    |  | repository layer |
              | this object         |  | not by the query |
              +---------------------+  +------------------+

Caption: one choke point per request. Tenant comes from the
token, the policy decision needs both the scope and the object,
and the data layer adds the tenant predicate itself so that a
handler cannot omit it. The path parameter is validated against
the resolved tenant and is never the source of it.

12d. Worked example: permissions for a business document surface

The prompt: 3,000 business tenants share a document service. Tenants want per-folder sharing, some want single sign-on, three large customers are asking for "our data in our own database", and a partner integration needs to read documents on behalf of users.

Stated inputs: 3,000 tenants, largest tenant holds 38 percent of all rows, 4 tenants are in regulated industries with restore-in-isolation clauses, 1 partner integration exists today with 3 more in the pipeline.

Block 1. Purpose: separate the two authorisation questions before touching a rung of the ladder.

I split the surface into "which application may act" and "which person may act on which object". The partner integration is the first question: it needs delegated access, so it gets scopes and an authorization code grant with the user present at consent time. Per-folder sharing is the second question: it is object-level and data-dependent, so it needs attributes, not scopes.

The error most readers make here is to answer the whole prompt with scopes, because the partner requirement arrived first and scopes are the thing you name when you hear "third party". A scope named documents:read cannot express "this folder, shared with this user, until Friday". Two mechanisms, layered, both required to pass.

Block 2. Purpose: pick the lowest rung that satisfies a stated requirement.

The 4 regulated tenants have a restore-in-isolation clause, which shared tables cannot satisfy at any amount of care: you cannot restore one tenant's rows to a point in time from a shared backup without a bespoke extraction. That is a requirement, not a preference, so those 4 get their own databases.

The tenant holding 38 percent of rows is over my 20 to 25 percent line, so it moves to its own cell. Not for security, for blast radius: its index maintenance, its worst query and its migration duration are already everyone else's problem.

The remaining 2,995 stay on shared tables with row-level security. So the answer is a ladder with three rungs occupied at once, and the sentence that makes it credible is that each exception is attached to a named requirement rather than to a size instinct.

Block 3. Purpose: make the predicate structurally unforgettable, then prove it.

Tenant comes from the token claim. The path segment is compared to it and a mismatch is a 404, not a 403, because a 403 confirms the resource exists in another tenant and that is an information leak in itself.

Row-level security is enabled with the tenant set from a connection variable at request entry. The repository layer will not construct a query without a tenant context. Then the proof: the differential test from earlier, running every suite query as two tenants and failing on any shared row identifier. Plus one production canary, a synthetic tenant with a single document that no code path should ever return to anyone else, alerted on.

Block 4. Purpose: give the sharing model a shape a support engineer can explain.

Per-folder sharing is attribute-based, which risks the "nobody can answer who can see this" failure. I buy that back with two things: an effective-permissions endpoint that answers "who can read this document and via which grant", and a rule that every grant is materialised as a row with a source (owner, folder share, role, admin override) so the answer is a join rather than an engine replay.

The error most readers make at this step: designing the evaluation and skipping the explanation. In a business product, "why can this person see this file" is a support ticket that arrives weekly, and a permission model with no readable explanation surface converts each one into an engineering escalation.

12e. Fade the scaffold

Faded example 1. Last block blanked. A machine-to-machine analytics surface: 40 partner services call a metrics endpoint, no humans involved, each partner may read only its own accounts, and partners rotate infrastructure often.

• Block 1: the principal is a service, so client credentials, and no user identity should appear anywhere.
• Block 2: no isolation ladder decision, because partners read aggregates derived from a shared store and none has a restore clause.
• Block 3: the account filter comes from the credential's mapped partner id, and a partner id in the query string is validated against it, never trusted.
• Block 4, credential lifecycle: you write this.

Faded example 2. Last two blocks blanked. An internal platform serving 60 engineering teams, one shared datastore, and a new requirement that a compliance team must read every team's data while no team may read another's.

• Block 1: the principal is a human in a team, plus a small set of humans in a cross-cutting role.
• Blocks 2, 3 and 4: you write these.

12f. Check yourself

Q1. Why should a request for a resource belonging to another tenant return 404 rather than 403, and when is that rule wrong?

Q2. You have 200,000 active sessions and are asked to cut revocation lag from 60 minutes to under 30 seconds. Give two designs and the cost of each, using the arithmetic from this module.

12g. When not to use this

Two components in this module are routinely over-applied. Each gets its own answer.

Attribute-based policy engines.

Database or schema per tenant.

The honest sentence to say in an interview: "I would stay on shared tables until a specific tenant or a specific clause forces me off, and I would rather spend that budget on making the tenant predicate structurally impossible to omit, because that is where the real incidents come from."

Module 13. Rate limiting and quotas as a product surface

Time: 22 to 28 minutes reading.

A rate limit is not a defensive measure that happens to be visible. It is a published promise about how much of your system a client may have, and it is the mechanism through which a free tier, a paid tier and an abuse policy are all expressed. Candidates who treat it as an infrastructure detail lose the whole product half of the answer.

13a. Predict first

A limit specification and a client, side by side.

+--------------------------------------------------------------+
| POLICY:  1,000 requests per minute per API key               |
| ALGORITHM: fixed window, aligned to the wall clock minute    |
| ON REJECT: 429 with  Retry-After: 60                         |
|                                                              |
| CLIENT:  a nightly sync that wakes at :59.5 and fires its    |
|          whole backlog as fast as the connection allows      |
+--------------------------------------------------------------+
Caption: a fixed-window policy, its rejection response, and a
client whose traffic is timed close to the window boundary.

Question. What is the largest number of requests this policy can admit in any two-second interval, and what does the Retry-After: 60 do to a thousand clients that are all rejected at the same moment?

13b. Algorithms, priced on burst, memory and coordination

Six mechanisms. The two questions that separate them: what burst do they admit, and what does one node need to know about the others?

Memory, derived. Assume 1 million active keys and a policy of 1,000 requests per minute, both stated by the interviewer or assumed aloud.

• Sliding window log: up to 1,000 timestamps per key. At 16 bytes each that is 16 KB per key, so 16 GB for a million keys. That number alone eliminates the log at this scale.
• Sliding window counter: 2 counters at 8 bytes, so 16 bytes per key, 16 MB for a million keys. Fits in one node's memory with room for the key strings.

The elimination is the point. The log is not "expensive", it is 1,000 times the memory, and stating the ratio is what makes the choice a derivation rather than an opinion.

Coordination, derived. Suppose 30 service nodes and a published global limit of 1,000 per minute per key.

Take the third row seriously, because it is what most large systems actually do. With a 200 ms sync interval, 30 nodes, and a per-node admission rate of 33 per minute during the drift window, the worst-case overshoot is 30 times 33 times 0.2 seconds divided by 60 seconds, which is about 3 requests. Overshooting a 1,000 per minute limit by 3 is free. That single calculation is how you justify approximate distributed limiting without hand waving.

13c. Dimensions, units and what the client is told

Limit on the right unit. Rate limiting per request is the default and it is usually wrong on a surface where requests differ in cost by orders of magnitude. If a client may fetch 1 item or 100 items per call, a per-call limit means your cost per unit of admitted work varies by 100x.

The repair is cost units. Publish a weight per operation, derived from measured work rather than invented:

Then the published limit is "6,000 units per minute", and a client can spend it on 6,000 cheap reads or 120 searches. Two properties follow: your capacity planning is in one currency, and a client that gets more efficient is rewarded rather than punished for batching.

Place limits at more than one level. A single limit per key is the beginner shape. Real surfaces stack them, and each layer answers a different threat.

+---------------------------------------------------------+
| EDGE          per source address, per unauthenticated   |
|               route. Stops floods before they cost you  |
+----------------------------+----------------------------+
                             v
+---------------------------------------------------------+
| GATEWAY       per API key, in cost units. This is the   |
|               limit you publish and bill against        |
+----------------------------+----------------------------+
                             v
+---------------------------------------------------------+
| TENANT        per tenant across all its keys, so one    |
|               tenant cannot buy the system with 40 keys |
+----------------------------+----------------------------+
                             v
+---------------------------------------------------------+
| RESOURCE      concurrency cap per expensive backend,    |
|               shared by everyone. The last line         |
+---------------------------------------------------------+

Caption: four stacked limits, outermost first. Each layer
protects something the layer above cannot see. Only the gateway
layer is a product promise; the others are protection, and only
the gateway layer belongs in your published documentation.

What the response must carry. A 429 with no information is a bug. The client needs three facts to behave: what the limit was, how much is left, and when it refills. The IETF httpapi working group has been standardising RateLimit and RateLimit-Policy fields for exactly this; as of the date I checked, the document is still an Internet-Draft, not an RFC (checked 2026-08-01). Use the shape, do not claim the standard.

Three status codes, three different meanings. Confusing them is one of the fastest ways to give a client the wrong instruction.

• 429: you exceeded a rate or quota that applies to you. Retry after the stated instant. The system is healthy.
• 503: the system is shedding load. Retry with backoff and jitter, and expect the wait to be unrelated to your own usage.
• 402: you are out of purchased quota. Retrying will never work. A human must act.

13d. Worked example: quotas for a search surface with a real bill

The prompt: a search endpoint is being opened to external developers. Design the limits and the quota model.

Stated inputs, all of which drive a number later:

Block 1. Purpose: choose the unit before choosing the algorithm.

I refuse to publish a limit in requests, because a search costs 50 times a read and both live on this surface. I publish cost units, with search weighted 50 and a plain read weighted 1. Now one number governs both.

The error most readers make here is to design the algorithm first. Algorithm choice is nearly free to change later; the published unit is a contract that is expensive to change, because it appears in every customer's plan and every invoice. Decide the thing that is hard to reverse first.

Block 2. Purpose: derive the tier sizes from capacity, not from a pricing meeting.

400 searches per second at weight 50 is 20,000 units per second of fleet capacity, which is 1.2 million units per minute. I hold back 30 percent for headroom, spikes and internal traffic, leaving roughly 840,000 units per minute to sell.

If the free tier is 600 units per minute (12 searches a minute, enough to build and demo something), then 2,000 free developers would commit 1.2 million units per minute, which exceeds the entire sellable budget. That kills a naive free tier immediately.

Two repairs, and I would say both aloud. Either the free tier is smaller, say 100 units per minute, so 2,000 developers commit 200,000 units per minute against 840,000 available, or the free tier is oversubscribed deliberately, on the observation that committed limits are not consumed limits. I would take oversubscription with an explicit ratio and a measured backstop, because the alternative makes the free tier useless for evaluation, and an unusable free tier costs conversions.

The measured backstop is the fourth layer in the stack above: a concurrency cap on the search fleet that sheds when actual load approaches capacity, independent of anybody's quota. Quotas are a promise about typical behaviour, and the concurrency cap is what makes the promise safe to oversubscribe.

Block 3. Purpose: choose the algorithm now that the numbers constrain it.

Sliding window counter at the gateway, per key, in units. The log is out on memory grounds from the earlier derivation.

Token bucket is tempting, and I reject a large bucket: a bucket of 10 minutes of savings lets a quiet key spend 10 minutes of fleet capacity in one burst, and with a 30 percent headroom margin that is not survivable if a handful of keys do it at once. I allow a small bucket, at most 2x the per-second rate, so an interactive client's natural burstiness is not punished.

Coordination is approximate with a 200 ms sync, justified by the overshoot arithmetic from the previous section: bounded overshoot of a few units against a budget of hundreds of thousands is not worth a central counter on the hot path.

Block 4. Purpose: connect the quota to the bill and to fairness.

Cost per unit falls out of the given numbers: 12,000 per month for the fleet, and 840,000 sellable units per minute is about 36 billion units per month if fully consumed. At 10 percent average utilisation, which I assume because sellable capacity is a peak figure and average traffic runs far below peak, the realistic consumption is 3.6 billion units, so the infrastructure floor is about 0.33 per million units.

Whatever the price is, it must clear that floor plus support and margin, and the fact that a candidate can produce a floor at all is the signal.

Fairness gets its own mechanism rather than being an emergent property. Per-tenant concurrency caps, set so no single tenant can hold more than a stated share of in-flight search slots, are the cheapest fairness device available: they cost one counter and they bound the damage a single well-behaved-but-large customer does to everyone else's latency.

The error most readers make at this step: reporting quota consumption only when it is exhausted. A developer needs the running total in every response, because the support ticket you are trying to prevent is "my job died at 3am and I do not know why".

13e. Fade the scaffold

Faded example 1. Last block blanked. A write-heavy webhook ingestion surface accepts events from 5,000 partner systems. Inputs: ingestion capacity 8,000 events per second, each event costs the same, partners are machines and retry automatically on any failure.

• Block 1, unit: events, because all events cost the same, so weights would be ceremony.
• Block 2, tier sizes: 8,000 per second with 30 percent headroom leaves 5,600 sellable, so a flat 1 event per second per partner commits 5,000 against 5,600 and is defensible without oversubscription.
• Block 3, algorithm: token bucket with a 60 event bucket, because partner systems batch naturally and a smoothing queue would add latency to an ingestion path that has none to spare.
• Block 4, what the rejection tells an automatic retrier: you write this.

Faded example 2. Last two blocks blanked. An internal platform endpoint used by 60 teams, one shared database, no billing relationship at all.

• Block 1, unit: database time in milliseconds, measured per call, because the teams' calls vary by 200x and there is no product tier to simplify it.
• Blocks 2, 3 and 4: you write these.

13f. Check yourself

Q1. Your published limit is 1,000 requests per minute per key, enforced by a fixed window. Capacity planning is done against 1,000. State the bug and the two ways to fix it.

Q2. A client asks why it received 429 when its dashboard shows it used only 40 percent of its plan. Give two explanations that are both your fault, not theirs.

13g. When not to use this

Distributed exact rate limiting.

Rate limiting at all.

The line worth saying in an interview: a rate limit protects you from clients, and a concurrency cap protects you from yourself. If you can only have one on an internal surface, take the cap.

Module 14. Long-running operations

Time: 20 to 25 minutes reading.

14a. Predict first

+--------------------------------------------------------------+
| POST /exports                                                |
|   builds a CSV of a tenant's last 24 months of invoices      |
|   returns 200 with the file body                             |
|                                                              |
| MEASURED:  p50  8 s     p95  110 s     p99  240 s            |
| PATH:      client -> CDN -> load balancer -> service         |
| CLIENT:    an SDK with a 60 s default timeout and 3 retries  |
+--------------------------------------------------------------+
Caption: a synchronous export endpoint with its measured latency
distribution, the hops in front of it and the client's timeout
and retry settings.

Question. Name three independent components that will terminate this request before it finishes, and say what the retry configuration does to the database at p99.

14b. Give the work an identity

The move is to stop returning the result and start returning a resource that represents the work. Everything else follows from that one decision.

The state machine, and the rule that makes it usable.

   +---------+      +---------+      +-----------+
   | PENDING | ---> | RUNNING | ---> | SUCCEEDED |
   +----+----+      +----+----+      +-----------+
        |                |
        |                +---------> +-----------+
        |                |           |  FAILED   |
        v                v           +-----------+
   +-----------+    +-----------+
   | CANCELLED |    | CANCELLED |    +-----------+
   +-----------+    +-----------+    |  EXPIRED  |
                                     +-----------+

Caption: five states. PENDING and RUNNING are the only states a
client should poll from. SUCCEEDED, FAILED and CANCELLED are
terminal and never change. EXPIRED is reached from SUCCEEDED
when the result is deleted after its retention window, and it is
a separate state so a client can tell "gone" from "never was".

The rule: terminal states are immutable. A client that has read SUCCEEDED must never see that operation become FAILED. Without that rule, clients cannot cache the outcome and must poll forever, and reconciliation logic on the client becomes impossible to write correctly.

Progress that does not lie. Two options, and one of them is nearly always wrong.

• A percentage. It must be monotonic. A percentage that goes backwards, or that sits at 99 for four minutes, teaches clients to ignore progress entirely.
• Completed and total counts. Better, because the client can compute its own estimate and can display something honest when total is unknown (report completed only, and say total is null rather than guessing).

14c. Polling, cancellation and retention, each derived from a number

Polling cost, derived. Assume 10,000 export jobs per hour and a mean duration of 5 minutes, both stated by the interviewer or assumed aloud.

833 requests per second of pure polling for 10,000 useful jobs is a 300:1 waste ratio, and it is entirely under your control through one response field. Return a suggested next-poll delay on every operation read and have the SDK honour it: start at 1 second, double to a 30 second cap, and reset on any state change. That single field cuts the load by more than an order of magnitude, and it is the cheapest thing on this page.

Cancellation has a guarantee, and you must state which one.

Saying "cancel returns 202 accepted, and the operation moves to CANCELLED only when a worker acknowledges it" is a small sentence that reads as someone who has operated one of these.

Result retention, derived. Assume 10,000 exports per day at an average 4 MB, and assume 95 percent of results are downloaded within one hour, which is reasonable because an export is usually triggered by a person who is waiting for it.

• 24 hour retention: 10,000 times 4 MB is 40 GB of live results at any time. Trivial.
• 30 day retention: 1.2 TB, plus the obligation to keep them consistent with deletions and access control changes over that period.

The 24 hour window covers the 95 percent case with a large margin and turns "where is my export from last month" into a one-line answer: re-run it. State the window in the response as an expiry instant, so the client does not have to remember your policy.

14d. Worked example: an export operation for a reporting surface

The prompt: reports currently time out. Make them work.

Block 1. Purpose: establish that the work needs an identity before designing anything else.

I check one number first: p99 duration against the smallest timeout in the path. Here p99 is 240 seconds and the smallest timeout is 60. The gap is not closable by optimisation of the kind you can promise in an interview, so the operation becomes asynchronous. If p99 had been 4 seconds against a 60 second timeout, I would say out loud that this whole module is unnecessary and keep it synchronous.

The error most readers make: reaching for a job queue because the word "export" appeared. The trigger is the ratio of p99 to the smallest infrastructure timeout, and saying that ratio out loud is what distinguishes a decision from a reflex.

Block 2. Purpose: make submission safe to retry, which is the whole reason clients duplicated work before.

POST /exports carries an idempotency key. Same key inside the retention window returns the same operation resource, with the same identifier, rather than starting a second export. Now the client's automatic retry is free: it converges on one operation instead of forking four.

I also add request fingerprinting, because the key alone is not enough: if the same key arrives with different parameters, that is a client bug and it must be rejected loudly rather than silently served the first result. The full treatment of keys, fingerprints and the transaction boundary is Module 9's job; here I am only inheriting it.

Block 3. Purpose: bound polling cost and make progress honest.

Every read of the operation returns state, completed and total row counts, and a suggested delay. The delay starts at 1 second, doubles to a 30 second cap, and resets to 1 second whenever the state changes, so a client sees a fast transition into RUNNING and then relaxes.

I put a hard cap on operation lifetime too: any operation still RUNNING after 30 minutes moves to FAILED with a specific code, because an operation with no upper bound is a resource leak that nobody notices until the worker pool is full of them.

Block 4. Purpose: decide what happens after success, including the two failure paths people forget.

On success the operation carries a result URL valid for the retention window, plus the expiry instant. Downloads are served from object storage, so a 4 MB file never crosses the application tier.

Two forgotten paths, and naming them is where this answer stops sounding like a tutorial:

1. Authorisation changes between submission and download. The result was computed with the submitter's permissions; if those permissions are revoked an hour later, the pre-signed URL still works. So the download goes through an authorisation check at fetch time, or the URL is bound to a short life measured in minutes rather than hours, and the trade-off is stated.
2. The tenant deletes data that the export contains. Now a file in storage outlives a deletion request, which is a compliance problem rather than an engineering one. The retention window has to be short enough that the deletion pipeline can treat exports as a known location, and exports must be enumerable per tenant so they can be swept.

14e. Fade the scaffold

Faded example 1. Last block blanked. A video transcoding surface: submissions take 2 to 40 minutes, clients are server-side integrations, and results are large.

• Block 1: p99 far exceeds any timeout, so an operation resource is not optional.
• Block 2: submission carries an idempotency key derived from the source asset and the profile, so a retried submission joins the existing job.
• Block 3: progress reported as completed and total segments, with a suggested poll delay, and a hard 90 minute lifetime cap.
• Block 4, how clients learn about completion without polling at all: you write this.

Faded example 2. Last two blocks blanked. A bulk permission change across a large tenant's document tree: 1 to 20 minutes, must be cancellable, partial application is visible to users while it runs.

• Block 1: asynchronous, and unusually the intermediate state is user visible, so progress is a product feature rather than a diagnostic.
• Blocks 2, 3 and 4: you write these.

14f. Check yourself

Q1. A client polls your operation resource every second for a 10 minute job and complains about being rate limited. Whose bug is it, and what is the one-field fix?

Q2. Your operation resource sometimes moves from SUCCEEDED back to FAILED when a downstream check runs late. Name two client behaviours this breaks.

14g. When not to use this

Two boundary cases worth naming, because they are where the simple rule stops applying:

• Duration is fine but the result is huge. That is a streaming problem, not an asynchronous one. A streamed response keeps the request short-lived at the byte level and avoids the entire job machinery.
• Duration is fine at p99 but the work is expensive and non-idempotent. Then you may want an operation resource purely for its identity, so that retries converge, even though the latency never required it. Say that reason explicitly, because it is a different justification and interviewers notice when the justification matches the design.

Module 15. Webhooks, event delivery and consistency across a service boundary

Time: 28 to 35 minutes reading.

An outbound event is the only part of your interface where you are the client and someone else's availability is your problem. It is also where the two hardest words in this course meet: a delivery guarantee is a distributed systems claim, and you are making it to strangers in writing.

15a. Predict first

+--------------------------------------------------------------+
| def capture_payment(req):                                    |
|     with db.transaction():                                   |
|         p = payments.insert(state='captured', ...)           |
|                                                              |
|     for attempt in range(3):                                 |
|         r = http.post(merchant_url, body=event(p))           |
|         if r.ok: break                                       |
|         sleep(2 ** attempt)                                  |
|     return 200                                               |
+--------------------------------------------------------------+
Caption: a handler that commits a payment and then delivers the
event inline, retrying three times inside the request.

Question. Name the two distinct consistency failures this code can produce, and say which of them is fixed by increasing the retry count from 3 to 30.

15b. The dual write, the outbox, and the guarantee you publish

The dual write problem, stated exactly. You have two systems to update: your database and someone else's endpoint. There is no transaction across both. Four interleavings exist and two of them are wrong.

The outbox, in three sentences. In the same transaction that changes state, insert a row into an outbox table describing the event. A separate relay process reads unsent outbox rows, delivers them and marks them sent. Because the state change and the obligation commit atomically, the missing-event and phantom-event rows above become impossible.

What you buy and what you pay:

Relay lag, derived. Assume the relay polls every 200 ms and delivers in batches of 100 per destination. At 10,000 events per hour, that is 2.8 events per second, so a poll finds around 0.6 events and lag is dominated by the poll interval: typically under half a second.

Now assume a burst of 50,000 events in a minute, which is 833 per second: the relay must sustain 833 deliveries per second or the backlog grows, and the lag becomes backlog divided by drain rate. Publish the steady-state number, alert on the derived one.

+--------------------+        +---------------------+
| WRITE PATH         |        | RELAY               |
| one transaction:   |  ==>   | reads unsent rows   |
| state change  +    |        | in key order, marks |
| outbox row         |        | sent after ack      |
+--------------------+        +----------+----------+
                                         |
                     +-------------------+------------------+
                     v                                      v
        +------------------------+            +------------------------+
        | PER DESTINATION QUEUE  |            | ATTEMPT LOG            |
        | own retry schedule and |            | every try: time, code, |
        | own concurrency cap    |            | body excerpt, next try |
        +-----------+------------+            +------------------------+
                    |
        +-----------+-----------+
        v                       v
+----------------+   +------------------------+
| CONSUMER       |   | DEAD LETTER            |
| verifies sig,  |   | after the ladder ends, |
| dedupes by id  |   | replayable by the user |
+----------------+   +------------------------+

Caption: the write path commits state and obligation together;
the relay fans obligations into a queue per destination so one
slow consumer cannot starve the others; every attempt is logged
where the consumer can read it; exhausted deliveries land in a
dead letter store that supports replay.

One queue per destination is the structural decision. With a single shared queue, one merchant that answers in 30 seconds occupies workers that every other merchant needs, and your p99 delivery latency becomes the worst consumer's latency. A queue and a concurrency cap per destination bounds the blast radius of a slow consumer to that consumer.

The guarantee you publish, in the words a client can act on. Write these four lines into your documentation, because each one tells a consumer what code to write.

• You will receive every event at least once. Deduplicate on the event identifier, which is stable across retries.
• You may receive an event more than once, including much later. Make your handler idempotent.
• Events for the same object arrive in order under normal operation, and may arrive out of order after a retry. Every event carries a version, so discard any event whose version is older than what you have stored.
• Return 2xx within 5 seconds. Anything else, including a timeout, is a failure and will be retried on the published schedule.

That last line is the one candidates omit, and it is the one that decides consumer architecture: told to answer in 5 seconds, a competent consumer enqueues and returns, rather than processing inline.

15c. Retries, signing, and the surface a consumer debugs with

Deriving a retry ladder rather than picking one. Consumer failures come in two clusters, and the ladder has to cover both.

Cover both with exponential growth. To reach 24 hours from a 15 second first retry with a growth factor of 3, the number of steps is the logarithm base 3 of (86,400 divided by 15), which is about 7.9, so 8 to 10 attempts. A concrete ladder that satisfies both clusters:

Ten attempts, a horizon of just over 46 hours, which covers an outage that starts on a Friday evening and is fixed on a Monday morning only if someone is working Saturday. State that limitation out loud rather than pretending the ladder covers everything, and pair it with dead lettering plus replay so a Monday fix is still recoverable.

Add full jitter to every delay. Without it, a consumer that fails at 09:00 gets all of its pending events re-presented at exactly the same offsets, which reproduces the load spike that may have caused the failure.

Draining a backlog without re-breaking the consumer. Assume an endpoint receives 10,000 events per hour in steady state, which is 2.8 per second, and it is down for 6 hours. The backlog is 60,000 events.

• Release at your own maximum, say 500 per second: the consumer receives 180 times its normal load and falls over again, converting a 6 hour outage into a repeating one.
• Release at 3 times steady state, 8.3 per second: the backlog drains in 60,000 divided by 8.3, which is about 2 hours, while the consumer handles a load it can plausibly survive.

So the per-destination queue needs a drain rate cap expressed as a multiple of that destination's observed steady rate, not as a global constant. This is the single most valuable operational detail in the module, and almost nobody says it.

Signing, and the three details that matter.

Send the signature and the timestamp as separate fields, and document the exact string that was signed, byte for byte. A consumer that cannot reproduce your canonical string cannot verify anything, and this is the most common integration support ticket on any webhook product.

Ordering, honestly. Global ordering across destinations is not achievable at reasonable cost and is not worth promising. Per-object ordering under normal operation is achievable and is what consumers actually need. Ship both halves: order events per object key in the relay, and put a monotonically increasing version on every event so a consumer that receives them out of order can drop the stale one. The version is what makes the weak guarantee usable.

The debugging surface, which is half the product. A webhook system without one generates support tickets forever.

The disable rule must be published, because a consumer that comes back after a two-day outage and finds its endpoint switched off with no notice will treat that as your bug, and they will be half right.

15d. Worked example: payment events to 800 merchant endpoints

The prompt: our payments product needs to tell merchants when a payment is captured, refunded or disputed. Design it.

Stated inputs:

Block 1. Purpose: write the guarantee before writing any mechanism.

I start with the sentence I will publish, because it constrains every later choice: at least once, per-payment ordering under normal operation, versioned so stale events can be discarded, 5 second response deadline, and a published retry ladder with dead lettering.

I choose at least once rather than at most once because a missed capture event means a merchant does not ship an order, which is a customer-visible failure, whereas a duplicate is something the merchant can deduplicate with an identifier I supply. That trade is the whole reason the guarantee is what it is, and stating it as a trade rather than as a default is the difference between a senior and a mid answer.

The error most readers make here: promising exactly once. There is no exactly once across a network boundary you do not control. What exists is at least once plus consumer-side deduplication, and calling that "effectively once" is fine as long as you say where the deduplication happens.

Block 2. Purpose: make the event owed at the moment the state changes.

The payment state change and an outbox row commit in one transaction. The relay is separate, reads unsent rows in payment-identifier order, and marks sent only after a 2xx.

Two consequences I say out loud. First, duplicates are now inherent, because the relay can deliver and die before marking sent, which is exactly why the published guarantee is at least once. Second, relay lag is now a service level indicator with an alert on it, because a stalled relay is silent: payments keep working, merchants keep receiving nothing, and the first symptom is a support ticket hours later.

Block 3. Purpose: size the delivery fleet and stop one slow merchant from taxing everyone.

90,000 events per hour is 25 per second, and the 4x sale-day peak is 100 per second. One queue and a concurrency cap per merchant, so the 6 percent of merchants who answer slowly consume only their own capacity.

Sizing with the slow ones as the constraint: if a slow merchant takes 5 seconds and I allow 4 concurrent deliveries to it, that merchant absorbs 4 workers for its whole backlog. 48 slow merchants (6 percent of 800) times 4 is 192 workers dedicated to slow endpoints. That number is why per-destination caps exist: without them, those 192 worker-slots would be taken from the shared pool at the worst moment, on a sale day, when the fast 94 percent need them most.

Drain caps are set per merchant at 3 times that merchant's observed steady rate, from the backlog arithmetic earlier.

Block 4. Purpose: give the consumer everything it needs to be correct without asking me.

The payload carries: event identifier (stable across retries, and the deduplication key), event type, a monotonically increasing version per payment, the payment identifier, the timestamp, and a minimal body. Not the full payment object.

That last choice deserves its reason. A minimal payload plus a fetch is more robust than a fat payload, because the fetch returns current state rather than state as of an event that may be 20 hours old, and because it keeps the signed body small. The cost is one extra call per event on the merchant's side, and for merchants who cannot afford that I offer the fat payload as an option with a documented staleness caveat.

The 35 percent of merchants on hosted platforms cannot write custom verification code, so the practical mitigation is an official plugin or a documented no-code path, and, more importantly, that population is the reason the signature scheme must be simple enough to describe in one paragraph.

The error most readers make at this step: designing the producer perfectly and leaving the consumer to figure out idempotency. Publish the consumer's algorithm explicitly: check the signature, check the timestamp is recent, look up the event identifier and return 200 immediately if seen, otherwise enqueue and return 200, then process asynchronously and discard any event whose version is not newer than stored state. Four lines, and they prevent most of your integration tickets.

15e. Fade the scaffold

Faded example 1. Last block blanked. A calendar product must notify integrators when events change. Inputs: 300 integrators, 2 million calendar changes per day, and a single busy calendar can generate hundreds of changes per minute during a bulk import.

• Block 1, guarantee: at least once, ordered per calendar, versioned, 5 second deadline.
• Block 2, write path: outbox in the same transaction as the calendar mutation.
• Block 3, fleet: 2 million per day is about 23 per second average, and the per-calendar burst is the real problem rather than the average.
• Block 4, how to stop a bulk import from flooding integrators: you write this.

Faded example 2. Last two blocks blanked. An internal service must tell three other internal services when an account is closed. All four services are in one deployment, one region, and there is already a message bus.

• Block 1, guarantee: at least once with per-account ordering, and consumers deduplicate.
• Blocks 2, 3 and 4: you write these.

15f. Check yourself

Q1. Your webhook system delivers at least once and consumers are told to deduplicate by event identifier. A merchant reports being told about a refund it had already processed, three hours late, after which its own state went backwards. Which of your guarantees was insufficient, and what field fixes it?

Q2. Someone proposes removing the outbox because "the message bus has a 99.99 percent publish success rate, so we lose one event in 10,000 and can reconcile those". Give the argument against, and the one condition under which they are right.

15g. When not to use this

A webhook delivery system.

The pull alternative, priced. A consumer polling an events feed every 10 seconds gets at most 10 seconds of staleness and costs 8,640 requests per day. For 50 consumers that is 432,000 requests per day, or 5 per second, which is nothing. In exchange you delete the entire delivery system: no signing, no retries, no dead letters, no disabled endpoints, no replay tooling. The consumer keeps its own cursor, which means missed events are impossible by construction rather than by your diligence.

Choose push when consumers need sub-second freshness, or when there are thousands of them and the polling load becomes real, or when consumers are unsophisticated and a cursor is too much to ask. Choose pull when consumers are few, when they sit behind a firewall, or when they are browsers, which cannot receive a webhook at all. Saying both options out loud, with the poll cost computed, is a stronger answer than designing the push system flawlessly.

Module 16. Batch, bulk and streaming interfaces

Time: 15 to 19 minutes reading.

16a. Predict first

+--------------------------------------------------------------+
| POST /contacts/bulk                                          |
|   body:  { "contacts": [ ...100 objects... ] }               |
|                                                              |
|   200 OK                                                     |
|   { "created": 97 }                                          |
+--------------------------------------------------------------+
Caption: a bulk create endpoint, its request and its entire
response for a request of one hundred items.

Question. A client sends this and gets that response. List everything it now cannot do.

16b. Partial success is a design decision, not an accident

Two honest semantics exist, and you must choose one and say which.

For per-item semantics, the response is an array positionally aligned with the request, and each entry carries the outcome for that item. Return an overall 200 even when some items failed, and say so in the documentation: the call succeeded, some items did not.

Overloading the transport status to mean "some of your items were invalid" leaves clients unable to distinguish that case from a genuine request failure. The WebDAV specification's 207 status exists for this shape, and borrowing it into a JSON product interface tends to confuse more clients than it helps, so state your choice explicitly either way.

Each item entry needs four things:

• The client's own reference for the item, echoed back, so alignment does not depend on array position surviving every proxy and serialiser.
• The created or affected identifier on success.
• A machine-readable error code plus the offending field on failure, following the error taxonomy from Module 7.
• A retryability signal per item, because in one response some failures are permanent (invalid email) and some are transient (a lock timeout).

Item-level idempotency. The whole call gets an idempotency key, and each item carries a client reference that is unique within the tenant. On a retry of the same key, return the stored result. On a partial retry of only the failed items, the client references let you detect duplicates for anything that actually succeeded. Without the per-item reference, a partial retry is unsafe and the client's only correct behaviour is to give up.

16c. Sizing, chattiness and flow control, each with a number

Sizing a batch, derived. Assume per-item work of 20 ms measured, and an endpoint latency budget of 5 seconds.

• Maximum items that fit the budget: 5,000 ms divided by 20 ms, which is 250.
• Publish 100, not 250. The margin absorbs slower items, a cold cache and a busy database, and it leaves room for the budget to be met at p99 rather than at p50.
• State the limit in the documentation and enforce it with a specific error code that names the maximum, because a client that discovers your limit through a 500 will pick a smaller number by trial and error and never revisit it.

Chattiness, priced. Assume a client on a link with 80 ms round trip time importing 100 contacts.

Note what that table says: parallelism recovers most of the benefit without any new endpoint. The bulk endpoint wins decisively on server-side overhead and on giving the client a single retry unit, not on network time alone. Use that reasoning rather than "batching is faster".

Streaming and backpressure, briefly. When the result set is large and the client can process incrementally, streaming beats both pagination and bulk. The design rule that matters: a stream must have flow control, or a fast producer will overwhelm a slow consumer's buffers.

Either the transport provides it, or you build acknowledgement windows into the protocol, for example the client acknowledges every N records and the server stops at N unacknowledged. State which of the two you are relying on, because "we stream it" with no flow control is how a consumer runs out of memory.

16d. Worked example: a contact import endpoint

The prompt: customers import contact lists from spreadsheets, typically 500 to 50,000 rows.

Block 1. Purpose: decide whether this is a bulk call at all.

50,000 rows at 20 ms is 1,000 seconds of work, which is not a request, it is a job. So the top of the range is Module 14's territory: upload the file, receive an operation resource, poll for progress, download a per-row result report.

The 500-row case is 10 seconds of work, still over any comfortable synchronous budget, so the honest answer is that the whole feature is asynchronous and the bulk endpoint exists only for programmatic clients sending small batches.

The error most readers make here: designing one endpoint for the full range. Two different shapes serve two different clients, and pretending otherwise produces a synchronous endpoint that times out for half the users.

Block 2. Purpose: give the small-batch endpoint honest semantics.

Per item, capped at 100, with a client reference required on every item. Response is an array of results with identifier or error code plus retryable flag, and an overall 200 whenever the request itself was well formed.

I require the client reference rather than making it optional, because optional identity produces exactly the systems that cannot retry safely, and it costs the client nothing at import time.

Block 3. Purpose: connect the large path to the operation resource without duplicating semantics.

The file upload returns an operation. Progress is rows completed against rows total. The terminal result is not a summary, it is a downloadable report with one line per input row carrying the same four fields the small-batch response uses. Same semantics, different envelope, which means one documentation page and one mental model for the client.

Retention on that report follows Module 14's rule, and here it should be longer than the export case: the report is how a customer fixes 300 bad rows, and they will do that over days, not hours.

16e. Fade the scaffold

Faded example 1. Last block blanked. An analytics ingestion endpoint accepts up to 1,000 events per call from mobile clients on poor networks.

• Block 1: bulk is right, because the client is buffering anyway and network cost dominates.
• Block 2: per-item semantics, each event carrying a client-generated identifier for deduplication.
• Block 3, what the response should contain given a mobile client on a poor network: you write this.

Faded example 2. Last two blocks blanked. A payments product is asked for a bulk refund endpoint accepting up to 500 refunds in one call.

• Block 1: refunds are independent, so per-item semantics, but money changes the calculus.
• Blocks 2 and 3: you write these.

16f. Check yourself

Q1. A client complains your bulk endpoint is unreliable: about 1 percent of its calls "half work". Nothing in your logs shows errors. What is the most likely design defect, and what do you change?

Q2. Your per-item work is 40 ms and your endpoint budget is 2 seconds. A client asks for a batch limit of 500. What do you tell them?

16g. When not to use this

The corollary for streaming: do not stream when the result fits comfortably in one response and the client cannot start work until it has all of it anyway. Streaming buys incremental processing, and if the consumer is going to collect the whole set into an array before doing anything, you have added flow control and partial-failure semantics in exchange for nothing.

Module 17. Client constraints as design input

Time: 12 to 16 minutes reading.

17a. Predict first

+--------------------------------------------------------------+
| GET /feed  returns 50 items, 42 fields each                  |
| Measured payload: 180 KB uncompressed, 46 KB gzipped         |
|                                                              |
| CLIENT NETWORK (assumed, stated aloud):                      |
|   throughput 400 kbps = 50 KB/s                              |
|   round trip time 300 ms                                     |
|   the screen renders 6 items and 9 of the 42 fields          |
+--------------------------------------------------------------+
Caption: one response, its measured size compressed and not, and
the network and screen constraints of the client reading it.

Question. Compute time to first render for this response, then say which single change to the contract reduces it most, and by how much.

17b. Three levers, in the order they pay

Delta sync, derived and bounded. Assume a client holds 200 entities at about 0.9 KB each, so a full sync is 180 KB, and assume 3 entities changed since the last sync.

• Full sync: 180 KB.
• Delta: 3 times 0.9 KB, so 2.7 KB, plus the cursor overhead. Roughly 66 times less.

Now find where it stops paying. If 30 percent of the 200 entities change between syncs, the delta is 60 items, which is 54 KB against 180 KB: still a 3x saving, but you are now maintaining a change log, tombstones, a cursor per client and a resync path to buy 3x. Around and above that churn level, a conditional full fetch with an entity tag is simpler and nearly as good, because an unchanged collection costs one small response and a changed one costs the full body either way.

The three things a delta protocol must specify, all of which get skipped in weak answers:

1. Deletes. A delta that only carries changed rows can never tell a client that something disappeared. You need tombstones with their own retention window, and the window must exceed the longest plausible client absence or you need a forced resync.
2. The resync trigger. When a client's cursor is older than the tombstone window, or the schema changed, the server must be able to say "discard everything and start over" in a single response. Clients that cannot be told to resync accumulate permanent corruption.
3. Ordering and atomicity. If a client applies half a delta and dies, its cursor must not advance. Advance the cursor only after the whole batch is durably applied locally.

17c. Chattiness is a property of the contract, not of the client

Why a chatty interface costs more than its bytes. This is an engineering point about radios, not a claim about any particular device. A cellular radio does not return to a low-power state the instant a transfer ends; it stays in a higher-power state for a tail period so that a following transfer does not pay the setup cost again.

The consequence for interface design is direct: twenty small requests spread over a minute keep the radio active far longer than one request that transfers the same total bytes, so the energy cost is driven by the number and spacing of requests, not by the byte count alone.

Two design rules follow, and both belong in an interface, not just in a client:

• Give the client one call that returns what a screen needs, rather than making it assemble a screen from five. Screen-shaped responses are a legitimate contract decision, not a leaky abstraction, when your client count is small and known.
• Give the client a way to batch its writes and to defer non-urgent ones, so background telemetry and read receipts can ride along with the next real request instead of waking the radio on their own.

The device-side half of this argument, including background execution limits and battery accounting, belongs to Mobile System Design. Here the point is narrower: chattiness is a property of the contract you published, so it is your design problem.

17d. Worked example: shaping a feed contract for a slow network

The prompt: our feed is unusable in markets with slow connections. Fix the contract.

Block 1. Purpose: get a measurement before proposing anything.

I ask for two numbers, and I refuse to design without them: the measured payload size at p50 and p95, and the client's round trip time distribution in the affected markets. Without those, every proposal is a guess and I cannot say which one wins.

Using the numbers from the pre-question, transfer dominates at 920 ms against a 300 ms round trip, so the priority is bytes rather than call count. If round trip had been 800 ms and payload 5 KB, the priority would invert entirely and I would be merging calls instead of shrinking them.

The error most readers make here: proposing the fix they know best rather than the fix the ratio indicates. State the ratio out loud, then choose.

Block 2. Purpose: change the cheapest thing that moves the number.

Page size becomes a required parameter with a documented maximum and a default that matches a phone screen rather than a desktop. This is a one-line change on the server and an 88 percent cut for the affected clients.

Then sparse field selection, in the restrained form: a small number of named field sets (compact, standard, full) rather than arbitrary field lists. Named sets keep the response cacheable, because there are three cache entries per resource instead of one per combination of 42 fields, and they keep the documentation finite.

Block 3. Purpose: decide about delta sync using the churn number, not enthusiasm.

I ask what fraction of a client's held items change between typical sessions. If the feed is chronological and mostly append-only, the answer is small and a cursor-based incremental fetch is already what pagination gives me, so I need nothing new. If items are heavily edited or reordered by ranking, churn is high and delta sync would deliver most of the collection anyway while adding tombstones and a resync path.

For a ranked feed, the honest conclusion is usually that delta sync is the wrong tool: the ranking changes for reasons unrelated to the items, so "what changed" is nearly everything. Saying that, and choosing conditional requests plus small pages instead, is a stronger answer than building the more sophisticated thing.

17e. Fade the scaffold

Faded example 1. Last block blanked. A field service application syncs a technician's job list to a device that spends hours offline in buildings with no coverage.

• Block 1: measure. Jobs per technician per day, about 40, at 2 KB each; sessions are long and offline gaps are hours.
• Block 2: page size is irrelevant here because 40 items is small; the binding constraint is the offline gap, not the byte count.
• Block 3, the sync design and its resync rule: you write this.

Faded example 2. Last two blocks blanked. A public read-only content interface serves 40 million requests a day to browsers worldwide, with a content delivery network in front.

• Block 1: measure. Payloads are 20 KB, round trip is low for most users because of edge caching, and the cache hit rate is 88 percent.
• Blocks 2 and 3: you write these.

17f. Check yourself

Q1. A mobile team asks for a single endpoint that returns everything their home screen needs, arguing it saves five round trips. Give the case for and the case against, and the condition that decides it.

Q2. Sparse field selection is proposed for a public interface sitting behind a cache. State the cost in one sentence and the restrained version of the feature.

17g. When not to use this

Delta sync.

Sparse field selection.

Module 18. AI-era interfaces: tool schemas as contracts

Time: 24 to 30 minutes reading.

Everything in this course so far assumed a caller that reads your documentation, compiles against your schema and behaves deterministically. A model-driven caller does none of those things. It reads a schema at runtime, chooses arguments probabilistically, and will happily call your refund endpoint because a user's message contained the word "refund". The contract has to absorb that.

18a. Predict first

+--------------------------------------------------------------+
| {                                                            |
|   "name": "refund_payment",                                  |
|   "description": "Refund a payment",                         |
|   "parameters": {                                            |
|     "payment_id": { "type": "string" },                      |
|     "amount":     { "type": "number" },                      |
|     "reason":     { "type": "string" }                       |
|   }                                                          |
| }                                                            |
+--------------------------------------------------------------+
Caption: a tool schema exposed to a model that handles customer
support conversations, exactly as written, with no other fields.

Question. List five things missing from this schema that would matter to a deterministic client and matter far more to a model-driven one. Then name the one addition that most reduces the blast radius of a wrong call.

18b. Design rules for a schema a probabilistic caller reads

A tool schema is a contract with three unusual properties: the caller reads the description as instructions, the caller cannot be recompiled, and the caller can be steered by the data it processes. Design accordingly.

The side-effect classification, which is the load-bearing idea.

Classify once, in the schema, and the harness enforces the rule uniformly. The alternative, deciding case by case in prompt text, fails the first time someone adds a tool without reading the convention.

Errors written for a caller that will act on them. Compare two responses to the same condition:

• Weak: {"error": "Invalid request"}. The caller has no next action, so it will retry the identical call, then try a different tool, then apologise to the user.
• Strong: {"code": "amount_exceeds_payment", "retryable": false, "detail": "amount_minor_units must be <= 4200", "next": "Ask the user whether to refund the full 4200 or a smaller amount."}.

The next field looks unusual to anyone from a traditional interface background, and it is the highest-value field in the response. You are writing to a caller that composes its next action from your text, so tell it what the action is. Keep the human message separate from the machine code, exactly as Module 7 requires, because here the "human" reading the message is another program.

Prompt injection changes who your caller is. If the model processes untrusted content, for example a customer's email, that content can attempt to steer the tool calls. Two consequences for interface design, both of which belong in your answer:

• Authorisation is never delegated to the caller's judgement. The tool call executes with a token scoped to the session's actual principal, and the scope is enforced server side, so a successful injection can only do what that user could already do.
• Irreversible tools sit behind gates that the model cannot open. If the model can decide that approval is unnecessary, there is no gate.

18c. Cost, streaming and version pinning

Metering, derived. Assume input tokens cost 3 per million and output tokens 15 per million. Those two numbers are an illustrative assumption used only to make the arithmetic concrete, chosen because they sit in a plausible range for a capable model; substitute your own and the method is unchanged.

One agent turn with 12,000 input tokens (system prompt, tool schemas, conversation, retrieved context) and 800 output tokens:

• Input: 12,000 divided by 1,000,000, times 3, which is 0.036.
• Output: 800 divided by 1,000,000, times 15, which is 0.012.
• Total: 0.048 per turn.

A support session of 50 turns is 2.40. Now the product consequences fall straight out:

Retry cost, derived. If you enforce structured output by validating and retrying, and the failure rate is 8 percent, the expected cost multiplier is about 1 plus 0.08, which is 1.08 for a single retry, plus a long tail if you allow several. Two design responses: prefer schema-constrained generation where the platform offers it, so the failure rate approaches zero without retries, and cap retries at two with an explicit failure to the caller, because an uncapped validate-and-retry loop is an unbounded bill with no error ever surfacing.

Streaming and cancellation, with the semantics stated. Streaming a response is easy. The contract question is what a partial stream means.

The trap in this area: allowing a tool call to execute during a stream, then cancelling, then leaving the client unable to find out what happened. The repair is that every tool execution produces an event in the stream and a durable record keyed by the call identifier, so a cancelled or dropped stream can be reconciled afterwards.

Version pinning, because a model is an undeclared dependency. Your schema does not change when the underlying model changes, but the distribution of outputs does. A client compiled against your schema can break on a Tuesday because the model got better at something.

So treat the model version as part of your contract: pin it per credential, expose the version in every response, and give clients an explicit opt-in to move. Module 11's compatibility rules then apply to a dimension they were never written for, and saying that out loud is a strong senior signal.

18d. Worked example: a refund tool for a support agent

The prompt: our support agent should be able to refund payments on behalf of a customer, from a chat conversation.

Stated inputs: refunds average 42 (currency units), 3 percent of conversations reach a refund, the agent handles 20,000 conversations a day, and current human-agent refund policy allows full refunds under 100 without a supervisor.

Block 1. Purpose: classify the operation before designing it.

Refunds move money to a third party and cannot be recalled, so this is the irreversible class. That classification, not the prompt engineering, decides everything downstream: idempotency key mandatory, approval gate required unless a policy allowance covers the case, and every call audited with the conversation identifier.

I mirror the existing human policy rather than inventing one: under 100 proceeds, at or above 100 requires a supervisor. That is not laziness, it is the right instinct in an interview. The organisation has already priced this risk for humans, and matching it makes the automated path auditable against a policy that already exists.

The error most readers make here: designing an approval gate for everything, which sounds safe and destroys the feature. If a human agent can refund 42 without asking anyone, requiring approval for a 42 refund means the automated path is slower than the manual one and nobody uses it.

Block 2. Purpose: write the schema so wrong calls are mostly impossible.

• payment_id: string, pattern constrained to your identifier format, required.
• amount_minor_units: integer, minimum 1, maximum supplied dynamically as the payment's remaining refundable amount, required.
• currency: not a parameter at all. It is derived server side from the payment. Removing a parameter is the strongest possible constraint, and any argument the server can derive should not be an argument.
• reason: closed enumeration of about six values that match the existing support taxonomy, required.
• idempotency_key: not a model parameter either. The harness generates it deterministically from the conversation identifier plus the payment identifier plus the amount, so a retried tool call converges and a genuinely second refund attempt does not.

Two of the five original fields have left the schema entirely. That is the pattern: every field you can derive or generate is a field the caller cannot get wrong.

Block 3. Purpose: put the gate where the policy says, and make the response usable by the caller.

Under 100: execute, return the refund identifier, the new refundable balance and the usage figures.

At or above 100: return a pending approval object with an identifier and an explicit next telling the caller to inform the customer that a supervisor is reviewing it, and not to retry. The approval is a resource in its own right, with its own state machine, which is Module 14's operation resource reused rather than a new concept.

Failure responses carry code, retryable and next. For example, payment_already_refunded is not retryable and its next instructs the caller to tell the customer the refund is already processed and to give the original refund identifier.

Block 4. Purpose: meter and bound the cost, and make the whole thing auditable.

Per-tenant metering in cost units, from the arithmetic above: at 20,000 conversations a day and, say, 12 turns each at 0.048, the daily model cost is around 11,500 currency units. That figure, derived in the room, is what turns "should we cache the system prompt" from an engineering preference into a funded decision.

Audit: every tool call records the conversation identifier, the principal whose token was used, the arguments, the approval decision if any, and the resulting refund identifier. The single most useful field is the principal, because the answer to "who refunded this" must never be "the assistant".

The error most readers make at this final step: treating the model's own transcript as the audit record. A transcript is a narrative and it can be influenced by the content the model processed. The audit record is the server-side log of what the contract actually executed, and only that log has standing.

18e. Fade the scaffold

Faded example 1. Last block blanked. A tool that searches an internal knowledge base and returns passages to the model.

• Block 1, classification: read only, so no gate, free retries, and caching is allowed.
• Block 2, schema: a query string, a result count with a maximum of 10, and an optional collection filter from a closed enumeration.
• Block 3, response shape: passages with source identifiers and a relevance score, plus usage.
• Block 4, the cost and safety considerations specific to a read-only tool: you write this.

Faded example 2. Last two blocks blanked. A tool that sends an email to a customer on the user's behalf.

• Block 1, classification: irreversible and external, because an email cannot be recalled and it reaches a third party.
• Blocks 2, 3 and 4: you write these.

18f. Check yourself

Q1. Your tool schema has a confirm: boolean parameter, and the harness executes the write when the model sets it to true. Explain why this is not an approval gate.

Q2. A client complains that identical requests to your model-backed endpoint returned different shapes last Thursday, and your schema has not changed in months. Diagnose it and say what you owe them.

18g. When not to use this

A tool-calling surface at all.

Streaming, in this context specifically.

Module 19. Interface as product: developer experience

Time: 15 to 18 minutes reading.

19a. Predict first

+--------------------------------------------------------------+
| ONE MONTH OF THE DEVELOPER FUNNEL                            |
|                                                              |
| visited the documentation      1,000                         |
| created an account               420                         |
| obtained an API key              180                         |
| made a first successful call      96                         |
| made a call on a second day       41                         |
+--------------------------------------------------------------+
Caption: five funnel stages with their monthly counts, from the
documentation page through to a second day of usage.

Question. You have one engineering week. Which step do you attack, and what is the arithmetic that justifies it rather than attacking the biggest absolute drop?

19b. Metrics that make developer experience an engineering problem

Vague goals produce vague work. Five measurements turn it into a backlog.

The last metric is strategically the most important and is almost never mentioned in interviews. If 80 percent of your calls come from a current library, a large class of breaking changes becomes a library release plus a deprecation window. If 20 percent do, every change is a broadcast negotiation with thousands of strangers.

What a client library should own, and what it must not.

The rule underneath: a library removes work, never information. Any behaviour a developer cannot observe in a log is a behaviour that will cost you a support ticket at 3am.

Generated or handwritten.

The third row is the honest recommendation for a surface with real external usage, and naming why (the generated part is where drift is dangerous, the handwritten part is where drift is invisible) is what makes it an answer rather than a preference.

19c. Sandboxes, keys and telling people a surface is going away

The sandbox, and the two features that make it worth building. A sandbox with realistic data is nice. A sandbox is only valuable if it can produce failures on demand. Two features:

1. Deterministic triggers: documented inputs that force specific outcomes, for example a particular test value that always produces a declined result or a rate limit response. Without triggers, nobody tests their error paths, and their first experience of your error model is in production at their busiest hour.
2. Event triggers: a control to fire any outbound event on demand. Otherwise integrators cannot develop against your webhooks at all without waiting for real activity.

Keys, and the four properties that matter.

• A visible, distinctive prefix, so leaked keys can be recognised by automated secret scanners and by humans reading a screenshot.
• Stored hashed, shown once. If you can display an existing key, so can anyone who reaches your database.
• Two active keys per account, always, so rotation is a rollout rather than an outage.
• Scoped and separately revocable, so a key for a reporting script is not also a key that can move money.

Deprecation communication, ranked by what it actually reaches. Module 11 built the mechanism; this is the human half. Ordered by observed reach, not by effort:

The sentence to say: every channel above brownouts is optional for the recipient, so the plan has to assume a residue that only a brownout reaches, and the residue's size is a measurement you already have from your traffic split.

19d. Worked example: cutting time to first call from 47 minutes to under 10

The prompt: our integrators say getting started is painful. Fix it.

Stated inputs, measured: median time to first successful call 47 minutes, first-call success rate 61 percent, and the top error code in week one is a missing required header on 38 percent of failing first calls.

Block 1. Purpose: split the 47 minutes into parts before optimising any of them.

I instrument the funnel by timestamp: landing to account, account to key, key to first call attempt, first attempt to first success. Suppose it splits as 6, 22, 9 and 10 minutes. Now the target is obvious: 22 minutes between having an account and having a key is a process problem, not a documentation problem.

The error most readers make here: rewriting the getting-started guide, because that is the visible artefact. The guide covers the 9 minute segment. Even perfect writing there cannot reach the 22.

Block 2. Purpose: remove the step rather than speed it up.

Sandbox keys issued instantly, no account, scoped to test data, rate limited hard, expiring in 14 days. The account requirement moves to the point where the developer asks for production access, which is a moment they are motivated to complete.

Any approval step that remains, for example for production keys on a regulated product, gets a stated turnaround and a status the developer can see. An invisible queue is what turns a 4 hour approval into a lost integrator.

Block 3. Purpose: make the first call succeed by construction.

Three changes, in order of value per hour of work:

1. A copy-paste snippet with the developer's own sandbox key already substituted, on the page where the key is issued. This removes the header mistake, because they never type the header.
2. An error response for the missing header that names the header, shows the corrected call, and links to the exact documentation anchor. Errors are documentation that arrives at the moment of need.
3. A first-call telemetry alert: if a new key's first ten calls all fail with the same code, that is a signal for a proactive message, and at low volumes a human can send it.

Recheck the metric afterwards rather than declaring victory: median time to first call, first-call success rate, and the error mix in week one. If the header error drops and a different code rises to the top, the work is not finished, it has moved.

19e. Fade the scaffold

Faded example 1. Last block blanked. A partner integration surface where every partner must be contractually approved before receiving production credentials, so instant self-service is impossible.

• Block 1: split the funnel, and accept that the approval segment is fixed by legal process rather than by engineering.
• Block 2: since the step cannot be removed, make it parallel rather than sequential.
• Block 3, what "parallel" means concretely: you write this.

Faded example 2. Last two blocks blanked. An internal platform interface used by 60 teams, where onboarding a new team takes about two days of a platform engineer's time.

• Block 1: split the two days. Suppose it is 3 hours of access provisioning, 4 hours of questions answered in chat, and the rest waiting.
• Blocks 2 and 3: you write these.

19f. Check yourself

Q1. Your library adds automatic retries with backoff. Name the operation class this makes dangerous by default, and the design that makes it safe.

Q2. You are asked to support six languages with hand-written libraries. Give the argument for cutting that to two, in terms an interviewer will accept.

19g. When not to use this

A separate one for client libraries.

Module 20. Operability and resilience budgets for a contract

Time: 24 to 30 minutes reading.

A contract is not only a set of shapes. It is a set of promises about time and failure, and those promises have to be measurable by you and derivable by the client. This module is about the indicators a contract needs, which are not the ones a service dashboard usually carries, and about the budgets that make timeouts, retries and shedding agree with each other instead of fighting.

20a. Predict first

+--------------------------------------------------------------+
| SERVICE DASHBOARD, LAST 30 DAYS                              |
|                                                              |
|   overall availability            99.97 %                    |
|   overall p99 latency               240 ms                   |
|                                                              |
| TRAFFIC MIX                    ERROR RATE BY OPERATION       |
|   GET  /invoices    98 %         GET  /invoices    0.01 %    |
|   POST /payments     2 %         POST /payments    0.80 %    |
+--------------------------------------------------------------+
Caption: an aggregate availability figure alongside the traffic
mix and the per-operation error rates that produce it.

Question. Reconstruct the 99.97 percent from the mix, then say what a merchant taking 500 payments a day actually experiences, and what the aggregate figure is therefore useless for.

20b. The indicators a contract needs

Three terms, defined before use. A service level indicator is a measurement, for example the fraction of payment creations that succeed. A service level objective is a target for that indicator over a window, for example 99.9 percent over 30 days. An error budget is the allowed shortfall, which for 99.9 percent over 30 days is 0.1 percent of 43,200 minutes, so 43.2 minutes.

Per operation, not per service. Every operation on a public surface gets its own row, and the rows differ in what they even measure.

The five indicators that only exist because you shipped a contract. These are what separate an operable interface from an operable service, and none of them appears on a default dashboard.

Correlation, and the support workflow it enables. Assign a request identifier at the edge if the client did not supply one, echo it in every response including every error body, and log it on every hop. Propagate standard trace context so a single call can be followed across services.

The reason to insist on it in an interview is concrete: a support ticket that contains an identifier can be answered in minutes by anyone, and a ticket that says "it failed around 3pm" costs an engineer an hour. Put the identifier in the error body, not only in a header, because the client's log line usually captures the body and drops the headers.

20c. Budgets: deadlines, retries and what you shed first

Deadline propagation, derived. Assume a client budget of 800 ms end to end, 60 ms of network round trip, dependency A at p99 120 ms, dependency B at p99 300 ms, and 100 ms of your own work.

• Your server-side budget: 800 minus 60, which is 740 ms.
• Expected consumption: 120 plus 300 plus 100, which is 520 ms, leaving 220 ms of slack.
• Now the case that matters. If A is slow and takes 500 ms, the remaining budget is 740 minus 500 minus 100, which is 140 ms. B needs 300 ms at p99. Starting B is a decision to blow the budget and to spend B's capacity on a request nobody will wait for.

So the rule: pass a remaining-deadline value to every dependency, and refuse to start a call whose typical duration exceeds what is left. Return a deadline-exceeded error immediately instead. This is worth saying precisely, because "we set timeouts" and "we propagate deadlines" are different designs, and only the second stops work that is already doomed.

Retry amplification, derived. Assume a partial outage in which 30 percent of requests fail, and every client retries three times.

• Load multiplier: 0.7 for the successes, plus 0.3 times 4 attempts, which is 1.2, giving 1.9 times normal load.
• If the outage was caused by overload, that 1.9x guarantees it persists. This is the metastable failure class from the hub's failure library: the system stays down after the original trigger is gone, because the retries are now the load.

The repair is a retry budget rather than a retry count. Cap retries at, say, 10 percent of the successful request count over a rolling window, measured client side by the library and server side as a check. Then the worst-case amplification is 1.1x rather than 4x, and a client that is failing everything stops retrying altogether, which is exactly the desired behaviour.

Three supporting rules worth stating: jitter every backoff, never retry a non-idempotent write without an idempotency key, and honour the server's retryability signal instead of guessing from the status code.

Shedding order is part of the contract. When you must drop work, drop it in a published order, because an unpublished order means every client believes it is in the protected class.

Publishing this converts an incident from a mystery into an expectation. It also forces a useful design conversation, because a surface where everything is class one has no shedding strategy at all.

Alerting on burn rate, derived. Take a 99.9 percent monthly objective on payment creation. The budget is 0.1 percent of 43,200 minutes, which is 43.2 minutes of failure for the month.

Two alerts, both derived from one objective, and neither is a threshold somebody picked. That derivation, done aloud in about forty seconds, is one of the highest-value things you can do in the last ten minutes of a round.

20d. Worked example: making a payments contract operable

The prompt: we have the payments interface from earlier in this course. Make it operable.

Block 1. Purpose: pick the indicators the contract implies, not the ones the dashboard already has.

Payment creation: availability as non-5xx share, latency as p99 to durable commit, and correctness as duplicate creations per million. Payment reads: availability, p99, and staleness against the creating write. Webhook delivery: share delivered within 60 seconds and count in dead letter.

The duplicate-creation indicator is the one that separates this answer. Availability and latency are generic; duplicates per million is a measurement of whether the idempotency design from Module 9 is actually working, and it is the number a payments team would ask about first.

The error most readers make here: naming the golden signals and stopping. Every service has traffic, errors, latency and saturation. A contract additionally has correctness properties that only its semantics define, and those are the interesting rows.

Block 2. Purpose: set objectives that are derivable from what clients actually need.

I derive rather than assert. A merchant taking 500 payments a day at a 99.9 percent objective sees an expected 0.5 failures a day, which is roughly one every two days: tolerable if each failure is safely retryable, and not otherwise. That makes the objective conditional on the semantics, which is the point: 99.9 percent with idempotency keys is a much better product than 99.95 percent without them, because the failures are recoverable by the client rather than by a phone call.

So: 99.9 percent monthly on creation, 99.95 percent on reads (cheaper to hold, and reads carry the interactive traffic), and 99 percent of webhooks delivered within 60 seconds. Then the burn-rate alerts follow from the arithmetic in the previous section, without a second discussion.

Block 3. Purpose: make the budgets agree with each other.

Client budget 800 ms, so the server budget is 740 ms after network. Deadlines are propagated to the ledger write and to the gateway call, and a call is not started if the remaining budget is under its p99.

Client libraries carry a retry budget of 10 percent of successful requests, so amplification is capped at 1.1x, and every retry carries the idempotency key generated by the library. The server publishes retryability per error code so the library is not guessing.

Shedding order is published: enrichment first, exports second, reads third, payment creation last. The publication matters as much as the mechanism, because an integrator planning a sale day needs to know where they sit.

Block 4. Purpose: make one call traceable end to end, because most incidents are one client's problem.

Every response carries a request identifier, echoed in error bodies. The identifier is stored with the payment, with the idempotency key record, and with every webhook attempt, so a support engineer can go from a merchant's log line to the ledger entry and to the delivery history in one query.

Then the indicator that closes the loop: 4xx rate by code by merchant, alerted per merchant rather than in aggregate. One merchant going from 0 to 400 validation failures an hour is invisible in an aggregate that serves millions of calls, and it is almost always either your release or their release. Catching it before they open a ticket is what an operable contract feels like from the outside.

20e. Fade the scaffold

Faded example 1. Last block blanked. A search interface with a 99.5 percent monthly objective and a 400 ms p99 target.

• Block 1, indicators: availability, p99 by page depth, and result-set stability under concurrent writes.
• Block 2, objectives: 99.5 percent monthly gives a budget of 0.5 percent of 43,200 minutes, which is 216 minutes.
• Block 3, budgets: a 400 ms p99 target with a 250 ms index call leaves 150 ms, so no serial second index call is affordable.
• Block 4, the alerting derived from that budget: you write this.

Faded example 2. Last two blocks blanked. An outbound event system with 800 consumers, some of which are frequently unavailable.

• Block 1, indicators: share of events delivered within 60 seconds, relay lag, dead letter count, and per-destination success rate.
• Blocks 2, 3 and 4: you write these.

20f. Check yourself

Q1. Your objective is 99.95 percent monthly. Derive the error budget in minutes and both burn-rate alert thresholds, showing the arithmetic.

Q2. A client asks you to raise your timeout from 2 seconds to 30 because their requests keep failing. Give the answer that is not simply yes or no.

20g. When not to use this

Full distributed tracing and multi-window burn-rate alerting.

Per-operation objectives.

Module 21. Level calibration: mid, senior and staff on one prompt

Time: 18 to 22 minutes reading.

Nobody is told the rubric. The comments that come back are things like "solid but not senior" and "good breadth, thin depth", and neither says what to change. This module makes the difference concrete enough to edit your own answer against.

21a. Predict first

Three candidates answered the same prompt, "design a webhook system so merchants learn when a payment is captured". One sentence each, taken from the middle of the answer.

+--------------------------------------------------------------+
| A. "I will POST the event to the merchant's URL, and retry    |
|     with exponential backoff if it fails."                    |
|                                                               |
| B. "I will commit an outbox row in the same transaction as    |
|     the payment, so a crash cannot lose the event, and a      |
|     relay delivers it at least once with a signed payload."   |
|                                                               |
| C. "Before mechanism: the guarantee is at least once with     |
|     per-payment ordering, and that choice makes merchant      |
|     deduplication mandatory, which changes what we document,  |
|     what we support, and what breaks when we change the       |
|     payload shape in two years."                              |
+--------------------------------------------------------------+
Caption: three single sentences from three answers to the same
webhook design prompt.

Question. Rank these by level, then state the specific thing each one adds over the one below it. Do not say "more detail".

21b. Three axes that actually separate levels

Not "depth". Three specific axes, and you can self-score on each.

The third row is the one candidates never rehearse and interviewers weight heavily. Being contradicted is a designed part of the round, and the response is scored.

The five additions that reliably move an answer up a level. Each is one or two sentences, and each is checkable rather than atmospheric.

Down-levelling tells, which are cheap to remove.

That last row deserves emphasis. Talking about influence and organisations does not read as staff in a design round. Naming a five-year cost that the current design creates does.

21c. The same prompt at three levels, side by side

Prompt: "Design the interface for a feature flag service."

Read the staff column carefully: almost every cell is about a failure that happens after launch, to someone else, and can only be prevented by a decision made now. That is the pattern to reproduce.

21d. Worked example: editing one answer up two levels

Here is a mid-level answer to the payments webhook prompt, then the edits. This is the exercise to run on your own recorded mocks.

Block 1. Purpose: capture the mid answer exactly, so the edits are visible.

"When a payment is captured, we POST a JSON body with the payment details to the merchant's configured URL. If it fails we retry three times with exponential backoff. We use HTTPS so it is secure. If it still fails we log an error."

The error most readers make when self-editing: rewriting from scratch. The point of the exercise is to see which specific additions change the level, and that is only visible if the base survives.

Block 2. Purpose: add what makes it senior, which is correctness under failure.

Three edits, each attached to a failure the mid answer cannot survive:

1. Outbox in the same transaction, because a crash between commit and POST loses the event permanently. This one edit is the difference between a design that works and one that works when nothing goes wrong.
2. A published guarantee, at least once with a stable event identifier, plus per-payment versioning, because retries reorder events by construction.
3. Signature over the timestamp and raw body, with a tolerance window, because "we use HTTPS" authenticates your server to the merchant and does nothing to authenticate the delivery to them.

Also replaced: "retry three times" becomes a derived ladder covering both the restart cluster and the business-day cluster, and "log an error" becomes a dead letter store with replay.

Block 3. Purpose: add what makes it staff, which is consequences over time and money.

Four further edits, matching the five additions from the table above:

1. Evolution: "The payload will need new fields, so it carries a minimal body plus a fetch, and the event type namespace is designed for additive growth. If we ever must remove a field, here is the traffic measurement that would end the old shape."
2. Metering and quotas: "Delivery costs us per attempt, and a merchant that returns 500 for a day costs 10 attempts per event. Per-destination drain caps bound that, and endpoints failing for 24 hours are disabled with published rules."
3. Guarantee as client obligation: "At least once means merchants must deduplicate, so the identifier is stable, the documentation states the algorithm in four lines, and the client library implements it so most merchants never think about it."
4. Blast radius: "If the relay is down, payments still succeed and events are delayed, which is the correct direction. If we inverted it and blocked payments on delivery, one merchant's outage would stop our revenue. I would rather explain a delay than an outage."

Then the closing sentence that is worth its length: "The thing I would measure first is delivery latency at p99 per destination, because a stalled relay is silent and merchants will not tell us for hours."

Block 4. Purpose: name what did not change, which is how you avoid mistaking length for level.

The core design is the same in all three versions: an event goes to a merchant's URL. No box was added for seniority. What changed was the set of futures the answer accounts for, and every addition is checkable by an interviewer. That is the honest definition of the level difference, and it is why padding an answer with components moves it down rather than up.

21e. Fade the scaffold

Faded example 1. Last block blanked. A mid-level answer to "design pagination for a comments feed": "We use ?page=2&size=20 and return the items plus a total count."

• Block 1: base captured.
• Block 2, the senior edits: keyset or cursor pagination because offset skips and duplicates rows under concurrent inserts and deletes, a stable sort key including a tiebreaker, and dropping or bounding the total count because it is a full scan on a large collection.
• Block 3, the staff edits: you write these.

Faded example 2. Last two blocks blanked. A mid answer to "design the error model for a bulk import endpoint": "We return 400 with a message saying what went wrong."

• Block 1: base captured.
• Blocks 2 and 3: you write these.

21f. Check yourself

Q1. An interviewer says "I would not do it that way" about your choice of cursor pagination. Give the mid, senior and staff responses.

Q2. You have four minutes left and your answer so far is competent but reads as mid. Name the two additions with the best return, and say why those two rather than more components.

21g. When not to use this

That last sentence is not a weakness. Interviewers hear invented depth constantly and it is easy to detect, whereas a clean boundary on your own experience plus real depth elsewhere is both rare and credible.

Two more situations where this module's framing should be set aside:

• You are interviewing for the level you already hold and the round is a confirmation rather than a stretch. Optimise for a complete, clean answer with stated trade-offs, not for reaching upward.
• The interviewer has explicitly asked for breadth, for example "walk me through everything you would consider". Depth in one area against that instruction reads as not listening, which is a delivery failure and it is scored as one.

Case study 1: Payments interface

Most treatments of this prompt draw a box labelled "payment service" and move on. The interesting part is that money makes retries dangerous, and retries are the one thing you cannot stop a client doing. This case study spends its depth on the retry contract and on how the merchant finds out what happened.

The prompt (payments)

"Design the public interface a merchant integrates to charge a customer's card, learn when the money actually moved, and issue refunds."

Questions that fork the design (payments)

Ask these because each answer moves a box or changes a resource. Do not recite them.

Requirements (payments)

Functional:

• Create a payment intent for an amount, currency and payment method, carrying a client-supplied retry key.
• Read a payment intent and its ordered history of state transitions.
• Cancel an intent before capture, and capture an authorised intent.
• Create refunds against a captured intent, fully or partially, more than once.
• Deliver state changes to a merchant-supplied endpoint, and let a merchant read missed changes without contacting support.
• List intents and refunds with a cursor that survives concurrent writes.

Non-functional:

Estimation that eliminates (payments)

Every line ends with the option it kills. A number that kills nothing is deleted from the answer.

V0, the simplest thing that works (payments)

Predict first. Which single property of V0 fails first, and at what number?

+-----------+     +-------------------+     +-----------------+
| Merchant  | --> | Payments API      | --> | Postgres        |
| server    | <-- | key check, charge | <-- | intents         |
+-----------+     | one transaction   |     | retry_keys      |
                  +-------------------+     | ledger_entries  |
                          |                 +-----------------+
                          v
                  +-------------------+
                  | Card gateway      |
                  | third party       |
                  +-------------------+

Caption. V0 has four components. The merchant's server calls one payments service, which checks the retry key and writes the intent and its ledger entries in a single database transaction, calling the third-party card gateway inside that transaction. The merchant learns the outcome by polling the intent.

Why this is correct at the stated scale:

• 450 inserts per second on one primary is unremarkable. The unique index on (merchant_id, retry_key) gives duplicate protection with no extra component.
• Putting the retry key and the effect in one transaction is the only version of idempotency that is actually safe, because there is no window where one exists without the other.
• No webhook infrastructure. At 39 million events per day a polling endpoint is more work for the merchant but zero work for you, and the merchant already has to poll for anything they missed.

Breaking point, then V1 (payments)

Breaking point one: connection pool exhaustion, at any sustained rate above roughly 120 requests per second.

What fails: the transaction stays open for the whole gateway call. With a pool of 300 connections and a 2.5 second hold, the service can only complete 300 divided by 2.5, or 120 requests per second. Above that, requests queue for a connection, the merchant's 10 second client timeout fires, and the merchant retries with the same key while the first attempt is still open. The retry blocks on the row lock, consuming a second connection, and the pool collapses.

How you observe it: connection pool wait time at p99, and a counter of concurrent requests arriving on a key that is already locked. The signature is that server-side success rate stays high while merchant-side success rate falls, because the merchant gave up before you answered.

+-----------+     +----------------+     +----------------+
| Merchant  | --> | API tier       | --> | Postgres       |
| server    | <-- | write intent   | <-- | intents        |
+-----------+     | return 202     |     | retry_keys     |
                  +----------------+     | outbox         |
                                         +----------------+
                                                 |
                  +----------------+             v
                  | Charge worker  | <-----------+
                  | calls gateway  |
                  +----------------+
                          |
                          v
+-----------+     +----------------+
| Merchant  | <-- | Event          |
| hook URL  |     | dispatcher     |
+-----------+     +----------------+

Caption. V1 has six components. The API tier writes the intent, the retry key and an outbox row in one short transaction and returns 202 immediately. A separate worker reads the outbox, calls the gateway and records the outcome. An event dispatcher signs and posts state changes to the merchant's endpoint.

V1 changes, each with the observation that justified it:

Breaking point two: one dead merchant endpoint stalls delivery for everyone, at about 7 dead endpoints.

What fails: the dispatcher has 200 workers and a 30 second timeout per delivery attempt. A hanging endpoint occupies a worker for the full 30 seconds. If enough traffic is directed at hanging endpoints, throughput falls to 200 divided by 30, or 6.7 attempts per second, against the 1,353 per second the system produces. Seven merchants each holding 5 percent of volume is enough to consume the pool.

How you observe it: worker occupancy grouped by destination, and delivery lag at p99 per destination plotted against the global figure. When one destination's lag and the global lag move together, you have head-of-line blocking, not a capacity problem, and adding workers buys minutes.

V2 changes:

• Delivery is partitioned by destination endpoint, with a per-destination concurrency cap and a per-destination circuit breaker. One merchant's outage is bounded to that merchant's queue.
• Events that exhaust the 72 hour retry window move to a dead-letter store with a replay operation the merchant can trigger themselves.
• A catch-up read, GET /events?after={cursor}, lets a merchant who missed deliveries reconcile without opening a ticket. This is the single cheapest support-cost reduction in the design.
• A reconciliation sweeper re-queries the gateway for any intent left in processing for more than 5 minutes, because a lost gateway callback otherwise means money moved with no record of it.

Deep dive one: what a retry key actually stores, and for how long

This is the dive the short treatments skip, and it is the most common senior probe on this prompt.

A retry key has three states, not two. Candidates almost always design for two.

The fingerprint is what stops the worst bug. Store a hash of the operation, the key, the authenticated principal and a canonicalised body. If a merchant reuses a key with a different amount, that is a bug in their code, and the only safe response is to refuse it with a distinct error rather than silently replay a charge for the wrong amount.

Scope the key per merchant, never globally. A global key space lets one merchant's key collide with another's, which is both a correctness bug and an information leak. The unique index is on the pair, and the pair is what the fingerprint covers.

Store the key in the same transaction as the effect. If the key lives in a cache and the intent lives in the database, a crash between the two writes produces either a double charge or a charge nobody can find. Both are worse than the latency you saved.

Size and lifetime, derived. From the estimation table, 13 GB of retry records per day. A 24 hour lifetime keeps that at 13 GB live, which one table holds comfortably. Extending to 7 days costs 91 GB and buys protection against a merchant whose weekly reconciliation job replays requests.

What retry keys do not give you. They deduplicate one client's retries of one request. They do nothing about a merchant submitting the same customer order twice with two different keys, which is a different problem and needs a merchant-supplied order reference with its own uniqueness constraint. Saying this unprompted reads as experience.

Retryability differs by operation, and the contract should say so. Reading an intent is safe and repeatable. Creating an intent or a refund changes state and needs a key. Cancelling is naturally repeatable because the second cancel finds the intent already cancelled and returns the same representation.

The IETF HTTP API working group has a draft, The Idempotency-Key HTTP Header Field, whose latest revision is 07 from 15 October 2025 and which is currently expired (accessed August 2026). Citing it is useful precisely because it is a draft: there is no standard here, so your design has to state its own rules.

Deep dive two: event delivery as a promise you can be held to

The dives on this page vary by problem. Here delivery is interesting because the merchant's endpoint is a component in your availability that you cannot deploy, monitor or fix.

Name the guarantee before designing the mechanism. At-least-once, unordered, with enough information in the payload for the consumer to reorder and deduplicate. Anything stronger is a promise you will break during your next incident.

Sign the raw bytes, not the parsed object. Re-serialising JSON changes whitespace and key order, so a signature computed over a re-encoded body fails for reasons nobody can debug. The signature covers a timestamp concatenated with the exact body, keyed per endpoint, and the header carries a scheme version prefix so you can rotate the algorithm later. RFC 9530, Digest Fields (February 2024, accessed August 2026), defines Content-Digest for the integrity half of this.

Replay protection needs both halves. The signed timestamp lets the consumer reject anything older than a few minutes. The event identifier lets them reject anything they already applied. Neither alone is sufficient: the timestamp window is wide enough to replay inside, and identifier deduplication alone lets an attacker replay an old signed event forever.

Thin or fat payloads is a real trade-off, not a style choice.

The design I would defend: fat payloads carrying an explicit object version, plus a documented rule that anything moving money must re-read first. The version is what makes the rule enforceable rather than aspirational.

The retry schedule is arithmetic, not folklore. Exponential backoff from 10 seconds with a cap of 1 hour and full jitter reaches 72 hours in about 75 attempts. Full jitter matters because a merchant endpoint recovering from an outage otherwise receives every backlogged event in one synchronised burst, which knocks it over again.

The debugging surface is the feature. A per-attempt log showing the request you sent, the response you got and the timestamp, visible to the merchant without a support ticket, removes more support load than any other single thing in this design. It also removes the argument about whether you sent the event.

Failure modes and evaluation (payments)

Service level indicators to publish:

• Accept-path success rate, measured as non-5xx responses on intent creation.
• Our own overhead at p99, computed as total span minus the gateway span, so a slow gateway does not hide a slow service.
• Retry-key conflict rate, split into matching fingerprints (healthy client retries) and mismatched fingerprints (a client bug you should tell them about).
• Fraction of events delivered within 60 seconds, reported globally and per destination.
• Count of intents in processing older than 5 minutes.
• Ledger imbalance: the count of transactions whose entries do not sum to zero.

Alert on ledger imbalance immediately at any non-zero value, because it is a correctness signal and not a capacity signal. Alert on stuck intents when the count stays above zero for 15 minutes. Alert on error budget burn rate for the accept path, not raw error counts. Do not page on one destination's delivery lag; that is the merchant's outage, and it belongs on a dashboard.

Re-checking against the non-functional requirements:

Where this answer is a simplification (payments)

• Card data changes your compliance boundary, not just your code. If raw card numbers reach your servers, the audit scope covers every system they touch. The usual answer is a browser-side tokenisation endpoint on your own origin so the merchant never holds the number, which turns a two-party integration into a three-party one.
• Strong customer authentication turns the flow inside out. A step-up challenge means the intent has to express "the buyer must be sent somewhere", which is a whole extra shape in the resource and a redirect the merchant has to implement. Designing it in later is a breaking change.
• Disputes are a second state machine with legal deadlines. Evidence submission, deadlines set by the card scheme, and outcomes that reverse money months later. The refund resource does not model any of it.
• Reconciliation is the real product. Merchants care less about the charge call than about whether your numbers match their bank statement, which means exportable, immutable, dated ledger views with documented rounding.
• Primary sources with dates. RFC 9110, HTTP Semantics (June 2022, accessed August 2026), is the authority on which status codes carry which meaning, including 409 and Retry-After.
• More primary sources. RFC 9457, Problem Details for HTTP APIs (July 2023, accessed August 2026), gives you a standard error body so you are not inventing one. The Idempotency-Key draft (revision 07, 15 October 2025, expired, accessed August 2026) is the closest thing to a shared convention. For the queue and outbox mechanics underneath the dispatcher, see the movement building blocks in Modern System Design.

Level delta (payments)

What each level added, in one line each:

• Mid to senior: every mechanism now has a number behind it, and the failure it prevents is named rather than implied.
• Senior to staff: compatibility becomes a design input rather than a later problem, and at least one alert is about correctness rather than availability.

Case study 2: Comment and threading service

This prompt looks like a storage question and is actually a contract question. The first thing that breaks is not the database, it is the promise your list endpoint made to a client that cannot be upgraded. The depth here goes on the tree representation and on the cursor.

The prompt (comments)

"Design the interface for comments on a post, including replies to replies, for a mobile client that renders a thread and lets people load more."

Questions that fork the design (comments)

Requirements (comments)

Functional:

• Create a comment on a post, optionally as a reply to an existing comment.
• Read a page of top-level comments, each with a bounded preview of its replies.
• Read further replies under one comment, independently paginated.
• Edit a comment, with the client able to detect that it is overwriting someone else's change.
• Soft delete a comment without breaking the shape of the thread.
• Page forwards and backwards through a collection that other people are writing to.

Non-functional:

Estimation that eliminates (comments)

V0, the simplest thing that works (comments)

Predict first. V0 is fast enough and correct on paper. What does a real user notice first?

+---------+     +------------------+     +-----------------+
| Client  | --> | Comments API     | --> | Postgres        |
|         | <-- | in-process page  | <-- | comments table  |
+---------+     | cache, 5 s life  |     | id, post_id,    |
                +------------------+     | parent_id, ts   |
                +------------------+     +-----------------+

Caption. V0 has three components. A client calls one comments service that keeps a small in-process cache of rendered pages for five seconds, and on a miss runs a keyset query against a single Postgres table holding every comment as a row with an optional parent reference.

Why this is correct at the stated scale:

• 694 inserts per second on one primary with two indexes is well inside a single node's range.
• The cache absorbs the first-page traffic. With 10,000 hot posts and a five second lifetime, the origin sees at most 2,000 first-page queries per second no matter how much traffic arrives, because each post can only miss once per lifetime per instance.
• Ordering is chronological, so the cursor is a keyset on the pair (created timestamp, identifier), which is immutable per row. Offset was eliminated above and never appears.
• The tree is an adjacency list: each row stores its parent. At depth three, which covers 95 percent of comments, three queries is acceptable.

Breaking point, then V1 (comments)

Breaking point one: non-monotonic reads, at any deployment with more than one instance.

What fails: a page cached independently on each instance for five seconds. Two consecutive requests from the same client can be served by instances whose entries were populated three seconds apart, so the newer response contains fewer comments than the older one. The author of a new comment sees it missing for up to five seconds. Neither is a capacity problem and neither gets better with more hardware.

How you observe it: an author-visibility indicator, measured as the fraction of comment creations whose next read by the same client contains the new comment, and a synthetic client that asserts the head cursor of a thread never moves backwards. Both fail at any traffic level, which is the tell that this is a contract defect.

The fix is a change to the contract, not a box. Split the collection into a part that never changes and a part that is tiny:

+---------+   +---------------+   +-----------+   +-------------+
| Client  |-->| Comments API  |-->| Shared    |-->| Postgres    |
|         |<--| local cache   |<--| cache     |<--| primary     |
+---------+   | 1 s head life |   | pages by  |   +-------------+
              +---------------+   | cursor    |          |
                                  +-----------+          v
                                                  +-------------+
                                                  | Read        |
                                                  | replicas    |
                                                  +-------------+

Caption. V1 has five components. Pages addressed by a cursor are immutable ranges held in a shared cache, the per-post head pointer is the only hot mutable value and carries a one second lifetime, and page queries go to read replicas while the head pointer is written through on the primary.

V1 changes, each with the observation that justified it:

Breaking point two: the client composes the thread, at 50 top-level comments per screen.

What fails: V1 returns a flat page of comments. To render a screen the client asks for 50 top-level comments, then makes one call per comment for the first few replies. That is 51 requests per screen, 918,000 requests per second at peak, and about 5.1 seconds of wall clock on a 400 ms mobile round trip even with four requests in flight. Separately, a thread nested 200 deep needs 200 sequential queries at the origin, which is 200 ms against a 120 ms budget.

How you observe it: requests per rendered screen, reported per client version, and the ratio of reply-list calls to thread-page calls. When that ratio sits near the page size, the contract is forcing the fan-out. On the origin, watch queries per page render rather than queries per second.

V2 changes:

• Storage moves from an adjacency list to a materialised path, so a whole subtree is one indexed range scan instead of one query per level. The derivation for why the other candidates lose is in deep dive one.
• The thread endpoint returns top-level comments each carrying a bounded reply preview and its own nested cursor, so one call paints the screen and "load more replies" is a second call only when the reader asks.
• Depth is capped at 40 in storage and expressed in the contract as a link that continues the thread from a new root, so an unbounded product decision does not become an unbounded query.
• Reply counts become exact below 1,000 and approximate above it, stated in the response so the client renders "1.2k" honestly rather than guessing.

Deep dive one: three ways to store a tree, and why the client never sees which one you chose

The candidates, compared on what this problem makes expensive.

Why the closure table loses here, with the arithmetic. A closure table stores one row per ancestor-descendant pair, so a comment at depth d contributes d plus 1 rows. With a mean depth of 2.5 across 21.9 billion comments, that is 3.5 times 21.9 billion, or 77 billion rows, against a base table of 21.9 billion. You would be paying 3.5 times the storage and 3.5 times the write amplification to make a read fast that a prefix scan already makes fast.

Why nested sets lose here. Inserting a node renumbers every node to its right. On a 200,000 comment thread that averages 100,000 row updates per insert, at a peak insert rate of 694 per second. Nested sets are a read-optimised structure for a tree that almost never changes, and a comment thread is the opposite of that.

Why the recursive query loses at depth. It is one statement, but the engine still resolves one level at a time, so latency grows with depth rather than with result size. At the measured depth of 200 that is the same 200 ms the estimation section already ruled out.

+---------------------------------------------------------+
| comments table, one row per comment                     |
| path = 000a.001f.0003    depth = 3    post_id = 91       |
| subtree read = WHERE path LIKE '000a.001f.%'             |
| index on (post_id, path) makes it one range scan         |
+---------------------------------------------------------+

Caption. The materialised path stores each comment's ancestors as fixed-width ordered segments in one column, so reading a subtree is a single prefix range scan on an index over the post identifier and the path.

Path width is a design number, not a default. Six characters per level, times a depth cap of 40, is 240 bytes plus separators, which sits below the 400 byte row size assumed above. Allowing depth 200 would cost 1,200 bytes of path on a 400 byte row, so the path would be three quarters of the table. The depth cap is what keeps the representation cheap, and it is a product decision you should surface rather than bury.

The contract lesson, which is the actual point of this dive. None of this appears in the interface. The client receives an identifier, a parent identifier, a depth and a cursor. It never receives the path. The moment a path appears in a response, clients parse it, and your storage representation has become a public contract you can never change. Expose the shape of the tree, not the encoding of it.

Bill Karwin's SQL Antipatterns (Pragmatic Bookshelf, 2010) is the clearest catalogue of these four representations and their costs, and it predates every framework you might otherwise cite for them.

Deep dive two: a cursor that survives inserts, deletes and an edit

The dives on this page vary by problem. Here the cursor is the interesting object because the collection is being written to while a client walks it, which is not true of the file listings in the next case study.

The three failure modes of pagination on a mutating collection. Name them before proposing a mechanism.

Keyset is the mechanism, and the tiebreaker is the part people forget. Order by the pair (created timestamp, identifier) and let the cursor carry both values. Ordering by timestamp alone is ambiguous whenever two comments share a millisecond, and at 694 writes per second that happens constantly, producing duplicates and skips that look random. The final sort key must be unique.

Make the cursor opaque, and make it carry its own validity. A cursor is a base64 string over a small structure:

That structure is roughly 41 bytes before encoding, so cursor size is not a constraint and does not need discussing further. What does need discussing is expiry: state a lifetime, and return a distinct machine-readable code when an expired cursor arrives, using the problem details format from RFC 9457, Problem Details for HTTP APIs (July 2023, accessed August 2026). A client that cannot tell "your cursor expired" from "you are unauthorised" will retry forever.

Soft delete is a pagination decision, not a storage preference. A tombstone keeps the row, so every cursor pointing at or past it stays valid and the tree keeps its shape. Hard deletion is still possible because the cursor stores sort values rather than a row reference, but it creates the skip failure above and it orphans replies. Choose soft delete and say why.

Edits and the lost update problem. A comment page cached under a cursor is only safe if edits invalidate it. Serve an entity tag per comment and per page, require a conditional write on edit, and return 412 when the tag does not match. RFC 9110, HTTP Semantics (June 2022, accessed August 2026), defines the precondition semantics in section 13, and RFC 9111, HTTP Caching (June 2022, accessed August 2026), defines how the revalidation works.

Total counts are the expensive part of the contract. An exact count of a 200,000 comment thread means counting 200,000 rows on every page request. Return a boolean for whether more exist, plus a count that is exact below 1,000 and marked approximate above it. Expressing the approximation in the response is what lets you change how it is computed later.

Bidirectional paging needs two link relations, not two endpoints. Emit next and prev links using the relation registry from RFC 8288, Web Linking (October 2017, accessed August 2026), so a client follows links rather than assembling URLs, and so you can change the URL shape without a breaking change.

Failure modes and evaluation (comments)

Service level indicators to publish:

• First page latency at p50 and p99, reported separately from deeper pages, because they are different queries.
• Requests per rendered screen, per client version. This is the indicator that catches a chatty contract before the origin notices.
• Author visibility: the fraction of creates whose next read by the same client includes the new comment.
• Pagination integrity, measured by a synthetic client that walks a live thread and asserts no duplicate and no missing identifier.
• Cursor rejection rate, split by reason: expired, wrong filter fingerprint, bad signature.
• Shared cache hit ratio for pages and for the head pointer, plotted separately.

Alert on pagination integrity failures at any rate above zero, because they indicate a correctness defect rather than load. Alert on author visibility falling below a stated threshold. Alert on error budget burn for the first page. Do not alert on cache hit ratio; it is a diagnostic, and it moves for reasons that are not incidents.

Re-checking against the non-functional requirements:

Where this answer is a simplification (comments)

• Ranking changes everything in this design. A "top comments" order makes the sort key mutable, which invalidates the keyset cursor's core assumption. The honest options are a pinned ranking snapshot per session or a documented admission that results drift. The equivalent problem for a ranked set is worked in case study 4.
• Per-viewer filtering kills the shared cache. Blocks, mutes and moderation states make the page different for every reader, so the immutable-page trick stops working and you cache fragments instead of pages.
• Moderation is a second write path with different latency requirements. A removal must take effect in seconds, globally, including in caches you have already served, which is a purge problem this design does not address.
• Live delivery interacts badly with cursors. A comment pushed to a client that also holds a cursor can arrive twice or out of order, so the push channel has to carry the same identifiers and the client has to deduplicate.
• Primary sources with dates. RFC 9110 (HTTP Semantics, June 2022) for preconditions and status codes, RFC 9111 (HTTP Caching, June 2022) for revalidation, RFC 8288 (Web Linking, October 2017) for pagination link relations, and RFC 9457 (Problem Details, July 2023) for the error body. All accessed August 2026. For the tree representations, SQL Antipatterns by Bill Karwin (Pragmatic Bookshelf, 2010). For the cache tiering underneath V1, see the storage building blocks in Modern System Design.

Level delta (comments)

What each level added, in one line each:

• Mid to senior: the contract now names its own failure modes, and the storage choice has arithmetic behind it.
• Senior to staff: the interface is designed from a measured client cost, and the parts of the contract that would prevent future change are removed on purpose.

Case study 3: File and blob service

The interesting thing about this prompt is that the bytes and the metadata want completely different designs, and the contract is where you decide how much of the byte path you are willing to own. The depth here goes on resumability and on delegated access.

The prompt (files)

"Design the interface an application uses to upload files of any size, download them, attach metadata, and have them expire on a schedule."

Questions that fork the design (files)

Requirements (files)

Functional:

• Begin an upload and receive a session that can be resumed after a disconnection.
• Send bytes in pieces, in order, and ask the server where to continue from.
• Complete an upload with an integrity check that fails loudly on corruption.
• Read and update file metadata independently of the bytes.
• Download a whole file or a byte range.
• Delete a file, and configure a rule that expires files after a stated age.
• List files in a container with a cursor.

Non-functional:

Estimation that eliminates (files)

V0, the simplest thing that works (files)

Predict first. V0 enforces the 32 MB launch cap. What is the first number that tells you the cap has to go?

+---------+     +------------------+     +----------------+
| Client  | --> | Files API        | --> | Object store   |
|         | <-- | streams body     | <-- | bytes          |
+---------+     | 32 MB limit      |     +----------------+
                +------------------+
                        |
                        v
                +------------------+
                | Postgres         |
                | file metadata    |
                +------------------+

Caption. V0 has four components. A client sends a whole file in one request to the files service, which streams it into an object store and writes a metadata row in Postgres. Downloads are proxied back through the same service.

Why this is correct at the launch scale:

• 69 uploads per second and 278 MB per second of ingest is three instances of work, comfortably inside a small fleet.
• A 32 MB file at 3 Mbps takes 85 seconds, which fits inside the 350 second idle timeout with margin, so a single request genuinely works.
• The contract says the limit out loud and returns 413 with a machine-readable code. A design that restricts itself honestly is better than one that accepts a request it cannot finish.
• Metadata and bytes are already separate resources, which is the one decision that makes every later version an addition rather than a rewrite.

Breaking point, then V1 (files)

Breaking point one: the upload success cliff at 131 MB.

What fails: once the limit is raised for mobile clients, uploads succeed for small files and fail for large ones, with the boundary at the file size whose transfer time equals the shortest idle timeout in the path. At 3 Mbps and 350 seconds that is 131 MB. Above it, the connection is closed mid-body, the client retries from zero, and on a 2 GB file the retry also fails, so the file is simply never uploadable.

How you observe it: upload success rate bucketed by file size, on a log scale. A healthy system is flat. This one is flat to 131 MB and then falls off a shelf. Also watch bytes transferred divided by bytes stored: when clients restart from zero, that ratio climbs above 1 and keeps climbing.

+---------+     +------------------+     +----------------+
| Client  | --> | Files API        | --> | Postgres       |
|         | <-- | session state    | <-- | uploads        |
+---------+     | offset authority |     | files          |
     |          +------------------+     +----------------+
     |                   |
     |                   v
     |          +------------------+
     +--------> | Object store     |
      chunks    | parts and bytes  |
                +------------------+

Caption. V1 has four components. The client creates an upload session, sends bytes in chunks, and asks the service for the current offset after any interruption. The service is the sole authority on how many bytes are durable, and it records session state in Postgres rather than in instance memory.

V1 changes, each with the observation that justified it:

Breaking point two: 22 Gbps of egress through the request tier.

What fails: proxying downloads needs 22 instances at peak whose only job is copying bytes. Worse, those instances also serve metadata calls, so a handful of 2 GB downloads occupy connections for minutes and small requests queue behind them. Metadata p99 starts tracking download volume, which is a dependency nobody designed and nobody expects.

How you observe it: the fraction of instance CPU spent in encryption and copying, and the correlation between metadata request p99 and concurrent large transfers. If those two lines move together, the byte path and the control path are sharing a resource they should not share.

V2 changes:

• Downloads are delegated. The service issues a short-lived, scope-limited token and the bytes come from an edge cache in front of the object store. The mechanics and the cost of that delegation are deep dive two.
• Range requests are supported end to end, so a client resuming a download does not restart it. RFC 9110, HTTP Semantics (June 2022, accessed August 2026), defines the range semantics in section 14.
• Lifecycle rules become a resource on the container rather than a background script, so expiry is inspectable, testable and auditable by the customer.

+---------+     +------------------+     +----------------+
| Client  | --> | Files API        | --> | Postgres       |
|         | <-- | issues download  | <-- | files          |
+---------+     | tokens           |     +----------------+
     |          +------------------+
     |
     |          +------------------+     +----------------+
     +--------> | Edge cache       | --> | Object store   |
      bytes     | validates token  | <-- | bytes          |
                +------------------+     +----------------+

Caption. V2 has five components. The files service issues a short-lived download token and never touches the bytes. The client presents the token to an edge cache, which validates it and reads from the object store on a miss.

Deep dive one: designing a resumable upload that survives a tunnel

Start from the failure, not the protocol. The client is on a train. The socket dies at an unknown byte. The only question the protocol has to answer well is: where do I start again, and who decides?

The server decides, and this is the whole design. The client cannot know how many bytes were durably committed, because the bytes it wrote into a socket may have been buffered, discarded or half written. So the session exposes a read that returns the current offset, and the client always asks before resuming. Any protocol where the client asserts the offset is a protocol that produces silently corrupted files.

Commit the offset after the bytes are durable, never before. If the offset advances first and the process dies, the client resumes past a gap and the file is corrupt in a way no retry will fix and no error will report. This ordering is the single most important line in the design and it costs nothing to say.

Make appends conditional. A chunk carries the offset it believes it is writing at, and the server rejects it with 409 if that is not the current offset. This turns three separate hazards into one loud error: a duplicated chunk from a retry, a reordered chunk from a client with parallel writers, and a second client appending to the same session.

Integrity is checked twice, for different reasons. A per-chunk digest catches a corrupted transfer immediately, so the client re-sends 8 MB instead of 2 GB. A whole-file digest at completion catches an assembly error, which per-chunk digests cannot. RFC 9530, Digest Fields (February 2024, accessed August 2026), defines Content-Digest for the chunk and Repr-Digest for the assembled representation, so you do not need to invent a header.

Completion is naturally idempotent here, and that is why it differs from case study 1. In the payments interface the client had to supply a retry key, because two charge requests are genuinely two intents. Here the session identifier already is the key: completing an existing session twice returns the same file resource, because the session names exactly one outcome. Do not add a second key mechanism when the resource already provides one.

Expiry is derived, not chosen by taste. From the estimation table, 200 GB of orphaned partial bytes per day. A seven day expiry caps the leak at about 1.4 TB, which is under 0.05 percent of annual storage, and seven days covers a client that pauses over a weekend. State the number in the contract so a client can plan around it.

One thing not to expose. Do not put the object store's own part numbers, part sizes or upload identifiers in your contract. The moment a client sends you a vendor part number, changing storage vendor is a breaking API change. Your session identifier is yours; the vendor's is an implementation detail.

The tus protocol, version 1.0.0, released 25 March 2016 (tus.io resumable upload protocol, accessed August 2026), defines Upload-Offset and Upload-Length with almost exactly this shape, and is worth reading as prior art before inventing headers.

Deep dive two: who may read these bytes, and for how long

The dives on this page vary by problem. Delegation is interesting here and nowhere else on this page, because this is the only case study where the payload is large enough that carrying it yourself is the dominant cost.

The delegation ladder, compared on what it costs you.

A signed token freezes the authorisation decision at issue time. That is the sentence to say out loud, because everything else follows from it. You cannot revoke a token that is already out; you can only make it short lived, or rotate the signing key and invalidate every outstanding token at once, which is an outage you inflicted on yourself.

The lifetime is derived from the download, not chosen round. A 2 GB file at 3 Mbps takes 89 minutes. If the token is validated only when the transfer starts, a five minute lifetime works and the transfer continues past expiry. If it is validated on every range request, a five minute lifetime breaks every resumed download. Pick the first, say which you picked, and document that a client receiving 403 mid-download should request a fresh token rather than surface an error.

Scope the token as narrowly as the use allows.

Do not bind the token to a client address. Mobile clients change address on every handoff, and you will have built a system that fails specifically for the users who most need resumability.

Caching and the identity trap. If every user receives a uniquely signed URL for the same popular file, the edge cache sees a distinct key per user and the hit ratio goes to zero. This is cache key fragmentation, and it turns a delegated design back into an origin-bandwidth design without anyone noticing. The fix is to keep the token out of the cache key: put it in a header or a cookie the edge validates but does not key on, and let the object path be the key.

Sharing is a resource, not a URL. When a customer wants to share a file, create an access grant with its own identifier, expiry, optional password and revocation, and mint tokens from it. An unguessable URL cannot be revoked, cannot be audited and cannot be listed. A grant can be all three, and "show me everything I have shared" is a feature customers ask for within a month.

What you give up. Once bytes leave through the edge, your own logs record issuance rather than use, so per-access auditing depends on an export from the edge on a delay. If a compliance requirement says every read must be logged within seconds, that requirement alone rules out delegation, regardless of the bandwidth arithmetic.

Failure modes and evaluation (files)

Service level indicators to publish:

• Upload success rate bucketed by file size, on a log scale. This is the indicator that finds the protocol bugs.
• Bytes transferred divided by bytes stored. Above 1.1 means clients are re-sending, which means resumption is not working.
• Resume rate, the fraction of sessions that used the offset read at least once. A value near zero on mobile means clients are not implementing the protocol.
• Completion digest mismatch rate, which should be indistinguishable from zero.
• Time to first byte on download at p50 and p99, split by cache hit and miss.
• Orphaned session bytes, and the age of the oldest unexpired session.
• Token rejection rate, split by expired, wrong scope and bad signature.

Alert on completion digest mismatches at any sustained non-zero rate, because that is corruption and not load. Alert on any size bucket whose upload success rate drops below the fleet rate by a stated margin, which catches a cliff before customers report it. Alert on orphaned bytes growth rather than on the absolute number. Do not alert on cache hit ratio; review it weekly, since it moves with content mix.

Re-checking against the non-functional requirements:

Where this answer is a simplification (files)

• Scanning and transcoding add a non-ready state to every file. The moment bytes must be processed before they are readable, the file resource needs a state, every client must handle it, and adding it after launch is a breaking change. Design the state in even if the pipeline comes later.
• Deduplication by content hash leaks information. If uploading a file that already exists returns instantly, an attacker learns that someone else holds that exact file. The mitigation is per-tenant deduplication, which forfeits most of the saving.
• Client-side encryption changes who you are. If the customer holds the keys, you can no longer scan, transcode, deduplicate or generate previews, and range requests interact awkwardly with block ciphers. That is a product decision wearing a cryptography costume.
• Data residency constrains the byte path. If bytes may not leave a region, your edge strategy, your replication and your disaster recovery all change, and the contract has to expose which region a container lives in.
• Egress is the bill. For most storage products the dominant cost is bytes leaving, not bytes stored, which is why pricing, quotas and cache hit ratio are design inputs rather than finance concerns.
• Primary sources with dates. RFC 9110 (HTTP Semantics, June 2022) for range requests in section 14 and preconditions in section 13. RFC 9530 (Digest Fields, February 2024) for Content-Digest and Repr-Digest. The tus resumable upload protocol, version 1.0.0, released 25 March 2016. All accessed August 2026. For the object store internals underneath this design, see the storage building blocks in Modern System Design.

Level delta (files)

What each level added, in one line each:

• Mid to senior: the protocol is derived from the slow client rather than from the request rate, and every threshold has arithmetic behind it.
• Senior to staff: the ordering of durable writes becomes explicit, and the costs that delegation moves onto other teams are named rather than assumed away.

Case study 4: Search service

Candidates answer this prompt with an index and a ranking function. Neither is the interview. The interview is which parts of your engine you accidentally promised to keep forever, and what happens to a client paging through a list that is being rewritten underneath it.

The prompt (search)

"Design the interface product teams use to search our catalogue: query, filter, facet, sort and page through results."

Questions that fork the design (search)

Requirements (search)

Functional:

• Search the catalogue with a text query and a set of typed filters.
• Sort by relevance or by a named business ordering.
• Return facet counts for an agreed set of fields.
• Page through results without duplicates or gaps within one session.
• Export a large result set with different guarantees from interactive search.
• Report how many results matched, honestly, including when the answer is capped.

Non-functional:

Estimation that eliminates (search)

V0, the simplest thing that works (search)

Predict first. V0 is already sharded because the arithmetic forced it. So where is V0 actually simple?

+-----------+                 +-----------------+
| Catalogue | --------------> | Indexer         |
| database  |                 | batch every 5 m |
+-----------+                 +-----------------+
                                       |
                                       v
+-----------+     +-----------+     +-----------------+
| Client    | --> | Search    | --> | Index shards    |
|           | <-- | API       | <-- | 4 x 25 GB       |
+-----------+     +-----------+     +-----------------+

Caption. V0 has four components. An indexer reads the catalogue on a five minute cycle and writes four shards of 25 GB each. A search service parses and validates the request, fans out to all four shards, merges the results and returns them.

Why this is correct at the stated scale:

• Four resident shards absorb 9,000 queries per second of fan-out, because each shard sees the full query rate but only a quarter of the index.
• Depth is capped at 1,000 results, so the coordinator merges at most 4,000 candidates rather than 400,080.
• Counts are reported as capped at 10,000, with a field saying so, rather than as an exact number nobody can afford.
• The query surface is one text field plus an enumerated list of filters and sorts. Nothing about the engine appears in the request or the response.

Breaking point, then V1 (search)

Breaking point one: the result set moves while the client walks it, from the first page onwards.

What fails: 2,200 documents change between two page requests. A document that becomes newly matching and ranks above the reader's position shifts everything down by one, so an item from page one reappears on page two. A document that stops matching shifts everything up, so an item is never shown. Neither depends on how deep the reader has gone, so the depth cap does not help.

How you observe it: a synthetic client that walks a live index to the depth cap and asserts no repeated and no missing identifier. Report the duplicate rate per hundred results. A second signal is the correlation between that rate and the indexing rate, which is what tells you it is churn and not a bug in the merge.

V1 changes:

Breaking point two: one slow shard sets the request latency, and one facet can exhaust a node.

What fails: two independent things, both invisible in average latency.

• Fan-out tail amplification. With four shards, a request is only as fast as its slowest shard. For the request p99 to be 250 ms, each shard must stay under 250 ms 99.75 percent of the time. Shard-level dashboards showing a healthy p99 will look fine while the request p99 sits well above budget.
• Unbounded facet work. A single query faceting a 5 million value field allocates 40 MB per shard. A handful of such queries arriving together drives a shard into garbage collection or out of memory, which then makes it the straggler in every other request it is serving.

How you observe it: the fraction of requests whose latency is set by a single straggler shard, computed as request latency minus the median shard latency; and per-query peak allocation grouped by facet field. If the straggler is always the same shard, it is a hot shard. If it moves, it is contention from expensive queries.

+-----------+     +-----------------+     +-----------------+
| Client    | --> | Search API      | --> | Index shards    |
|           | <-- | cost budget and | <-- | 4 x 25 GB,      |
+-----------+     | hedged fan-out  |     | 2 replicas each |
                  +-----------------+     +-----------------+
                          |
                          v
                  +-----------------+
                  | View registry   |
                  | pinned readers  |
                  +-----------------+

Caption. V2 has four components. The search service estimates the cost of a request before running it, fans out with a hedge to a second replica when a shard is slow, and keeps a registry of pinned readers so a session can hold a stable view of the index for a bounded time.

V2 changes:

• Every shard has two replicas, and the coordinator sends a second copy of the request to the other replica once a shard exceeds the 95^th percentile of its own recent latency. That converts a straggler into an extra unit of load rather than a slow response.
• Requests carry an estimated cost, computed from the facet fields requested, the number of filter clauses and the depth. Requests above the budget are rejected with a machine-readable code that names the offending parameter, rather than accepted and allowed to hurt everyone.
• Facet fields come from an allow-list and always carry a top-N limit, so the 40 MB per shard case cannot be requested.
• A view registry holds pinned readers with a lifetime and a concurrency cap, derived from the 0.33 GB per view arithmetic. This is the mechanism behind the stable-paging option in deep dive two.

Deep dive one: the query surface you can still change in two years

There are three levels of query contract, and the choice is irreversible in one direction.

The rule that generates the right answer: expose intent, not implementation. sort=price_asc is an intent. sort=price:asc,_score:desc is an implementation, and the moment a client sends it you have promised that a thing called _score exists and behaves that way. Enumerate sorts by name and you can change what each one means internally.

Validate strictly, and say why. Silently ignoring an unknown filter is the worst possible behaviour: a client typo means the request returns unfiltered results, which looks like a relevance problem and gets debugged for a week. Reject unknown fields and unknown operators with a problem document that lists the valid ones, using RFC 9457, Problem Details for HTTP APIs (July 2023, accessed August 2026).

Design the evolution path before you need it.

RFC 9745, The Deprecation HTTP Response Header Field (March 2025, accessed August 2026), and RFC 8594, The Sunset HTTP Header Field (May 2019, accessed August 2026), give you a standard way to say "this is deprecated" and "this stops working on this date" without inventing headers your clients then have to learn.

Relevance exposure is part of the query surface, not a separate topic. If you return a numeric score, clients will threshold on it, and your next ranking change will move every threshold at once. Return rank order, and if callers need to know why an item ranked where it did, offer a separate explanation endpoint that is explicitly labelled as diagnostic, unstable and not for production logic.

The cost of getting this wrong is asymmetric. Adding an operator later is a day of work. Removing one, across 40 shipped clients on independent release cycles, is a year. Design as if every field you expose is permanent, because for practical purposes it is.

Deep dive two: paging a ranked set that will not hold still

Why this differs from the cursor dive in case study 2. There the sort key was a creation timestamp: immutable, owned by the row, and never recomputed. Here the sort key is a relevance score computed at query time by a model that changes, over an index that changes. A cursor that stores "the last row's sort value" is meaningless if the function producing that value is different on the next request. Same word, different problem, different mechanism.

Offer three named guarantees, and make the client pick.

The cursor needs a unique final sort key or it is broken. Ranked results routinely tie: many documents share a score. Without a unique tiebreaker appended to the sort, "everything after score 4.21" is ambiguous and the same document can appear on both sides of the boundary. Append the document identifier as the last sort key, always. Apache Lucene's IndexSearcher.searchAfter takes exactly this shape, a previous result used as the position marker, and is documented in the Lucene 10.0.0 IndexSearcher javadoc (accessed August 2026).

Put the ranking model version inside the cursor. When a model rolls out mid-session, the scores change, and a cursor issued under the old model addresses a position that no longer exists. Detect it and return a distinct error telling the client to restart the list. The alternative is silent duplicates and gaps that nobody can reproduce, because they only happen during a deploy.

Make the pinned view a resource, not a hidden mode. Create it, get an identifier and a lifetime, use it on subsequent searches, delete it when finished. Three reasons this beats an implicit session:

• The cost is visible, so you can cap concurrency and expire the oldest, using the 333 GB arithmetic from the estimation table.
• The client's intent is explicit, so you can meter or price it.
• A leaked view is a listable, expirable object rather than an invisible reference holding segments alive.

Total counts belong in the response shape, not in a number. Return a structure with a value and a relation, so "12,431 exactly" and "at least 10,000" are the same shape. That one decision lets you change how counting works later without a breaking change, and it stops clients from rendering a precise-looking number you never promised.

Deep paging gets its own door. The export interface takes the same query, returns results in an order that is cheap rather than relevant, usually the document identifier, streams them, and has its own rate limit and its own latency objective. Splitting it removes the pressure that would otherwise push you into supporting offset 100,000 on the interactive path.

Failure modes and evaluation (search)

Service level indicators to publish:

• Search latency at p50, p95 and p99, reported separately for filtered, broad and faceted queries, because they are different workloads sharing an endpoint.
• Straggler share: the fraction of requests whose latency exceeds the median shard latency by a stated margin.
• Hedge rate, and hedge win rate. A high hedge rate with a low win rate means you are adding load for nothing.
• Pagination integrity from a synthetic client walking a live index to the depth cap.
• Cursor rejection rate, split by expired, model version changed, and filter mismatch.
• Capped-count rate, the share of responses where the total was reported as a lower bound.
• Index freshness lag, measured as the age of the newest document visible to search.
• Open pinned views and the age of the oldest.

Alert on index freshness lag, because a stale index returns confident wrong answers rather than errors and no other indicator catches it. Alert on straggler share and on open pinned views. Alert on error budget burn for the search path. Do not alert on zero-result rate: it is a product metric that moves with the catalogue and with seasonality, and it belongs in a weekly review.

Re-checking against the non-functional requirements:

Where this answer is a simplification (search)

• Relevance quality is a separate discipline and a separate interview. Everything above is the contract around ranking, not the ranking. Feature engineering, training data, offline evaluation and online experimentation belong in ML System Design.
• Language handling is not a checkbox. Analysers, stemming, compound splitting and script handling differ per language, and the choice is baked into the index at build time, which means changing it is a full reindex rather than a configuration change.
• Query understanding sits in front of everything here. Spelling correction, entity recognition and intent classification rewrite the query before it reaches the index, and each rewrite is a place where the results the user sees stop corresponding to what they typed.
• Experimentation makes the contract harder. If two users get two ranking models, then results are per viewer, shared caching is gone, and cursors must carry the assignment or a user can flip models mid-list.
• Personalisation collapses the cache and the pagination story at once. Every mitigation in this design assumes many users share a result set.
• Primary sources with dates. The Lucene 10.0.0 IndexSearcher javadoc documents searchAfter as the deep paging mechanism. RFC 9457 (Problem Details, July 2023) for error bodies, RFC 8288 (Web Linking, October 2017) for next-page relations, RFC 9745 (Deprecation, March 2025) and RFC 8594 (Sunset, May 2019) for retirement signalling. All accessed August 2026. For the retrieval fundamentals underneath the index, Introduction to Information Retrieval by Manning, Raghavan and Schütze (Cambridge University Press, 2008) remains the standard free text.

Level delta (search)