Machine Learning System Design Interview

Prerequisites

Seniority is a weak readiness signal and a job title is worse. Prerequisite mastery is the signal that works. Answer every Quick check in under a minute. Two or more misses means start with the linked material rather than with Module 1.

Time budget

Learning objectives

Each objective is tested by exactly one numbered item in the mastery check, and the numbering matches: objective 1 is tested by mastery item 1.

1. Translate a stated business objective into a trainable objective plus at least one guardrail metric in under five minutes, and name the specific behaviour the model would produce if that guardrail were removed.
2. Compute, for a given prompt, the candidate pool size, a per-stage latency split that sums to the stated serving budget, and the embedding index footprint in gigabytes, showing each step, and delete every number that eliminates no design option.
3. Choose between batch precompute, online inference and streaming features for a stated freshness requirement and request rate, citing the freshness figure in minutes that decides it and the cost the rejected option would have carried.
4. Diagnose a described offline-to-online metric disagreement into exactly one named cause (training-serving skew, a join that is not point-in-time correct, position bias, delayed labels, or an evaluation slice that does not match live traffic) and name the single check that confirms it, in under three minutes.
5. Design a retrieval-then-ranking path for a stated pool size and latency budget, stating for each stage the candidate count it reduces to and the recall that stage is permitted to lose.
6. Specify an online experiment for a stated baseline rate and target lift, giving samples per arm and duration in days, and state the condition under which you would ship despite a flat primary metric.
7. Rewrite a mid-level ML answer into a staff-level one by adding label delay handling, a retrain cadence with its monthly cost, a rollback criterion and one degradation path, annotating what each addition changed about the decision.

Warm-up retrieval

Three to five minutes. Answer out loud or in writing before opening anything. Retrieval beats rereading, and a visible answer converts one into the other.

Q1. From the hub's round router: name two signals inside the first minute that tell you this is an ML design round rather than a generalist distributed-systems round.

Q2. From the trade-off atlas on the hub, several pages back: state the decision rule for batch versus stream, and name the number that decides it.

Q3. The hub states one hard rule about when a number is allowed to appear in your answer. State the rule, and state the one-line offer you make when a prompt does not turn on volume.

Module map

+--------------------------------+
| FRAME              M1 to M3    |
| scope, objective and guardrail |
| design, estimation that cuts   |
+---------------+----------------+
                |
                v
+--------------------------------+
| DATA               M4 to M5    |
| labels, label delay, features, |
| point-in-time joins, skew      |
+---------------+----------------+
                |
                v
+--------------------------------+   +----------------------+
| SERVING PATTERN    M6 to M8    |   | TRUTH                |
| retrieve, filter, rank, index, |==>| M9  calibration      |
| model choice under latency     |   | M10 offline eval     |
+---------------+----------------+   | M11 online eval      |
                |                    +-----------+----------+
                |                                |
                v                                v
+-------------------------------------------------------+
| OPERATE AND JUDGE                      M12 to M16     |
| serving and rollout, training cost, drift, patterns,  |
| level calibration                                     |
+-------------------------------------------------------+

Caption: the five clusters of this course and their reading order.
Frame feeds Data, because you cannot choose features before you
know the objective. Data feeds the Serving Pattern. The Serving
Pattern feeds Truth, which is calibration and both evaluation
layers. Both feed Operate and Judge, where mid-level and staff
answers separate. Arrows point from prerequisite to dependent.

Low ends total 600 minutes, high ends total 780. That is the 10 to 13 reading hours in the time budget above.

The delivery clock for this round

Most candidates who fail an ML design round did not lack a fact. They described a pipeline that could exist for almost any prompt, and never committed to an objective, a label, a latency budget or a way of being wrong. The clock below is a budget, not a script, and this round's budget is shaped differently from a generalist one: more minutes on the objective and the data, fewer on boxes.

The 45 minute round

The phases sum to 45 by construction: 2 plus 7 plus 4 plus 8 plus 9 plus 10 plus 5.

The 60 minute round

These sum to 60: 2 plus 9 plus 5 plus 9 plus 11 plus 16 plus 6 plus 2. The extra 15 minutes buys one more deep dive and a real evaluation and monitoring section. It does not buy more boxes.

The first two minutes

Say four sentences, in this order. Rehearse them once so they are automatic and you can spend your attention on listening.

1. "Let me restate it as a prediction: for each X we predict Y, and the consumer of that prediction is Z." (If you cannot fill in all three slots, that is your first clarifying question.)
2. "I am going to spend about seven minutes on the objective and the labels, then draw the system end to end, then break it on purpose." (You just told them your clock, which reads as control.)
3. "I will optimise for A, and I will hold B as a guardrail so we do not win A by damaging it." (Naming the guardrail is the cheapest senior signal available in this round.)
4. "Stop me if you want depth somewhere specific rather than coverage." (It costs one sentence, and it replaces your guess about what the panel wants with an answer.)

Declaring what you are skipping

Skipping silently reads as not knowing. Skipping out loud reads as prioritising. The pattern is: name it, say why it cannot change a decision in the next twenty minutes, say what you would do if it were in scope, in one sentence.

• "I am not designing the training cluster. The model fits on one machine at this data volume, so the distributed training choice cannot change anything on the serving path, which is where the latency budget is tight."
• "I am skipping the cold-start ranker for brand new users. It matters commercially, and it is a separate model with its own evaluation. If you want it, it is a popularity prior blended down as personal signal accumulates."
• "I will not size the feature store. Nothing here changes based on whether it holds 2 or 20 terabytes, because the deciding constraint is the point-in-time join, not the volume."

Checking in without sounding unsure

A weak check-in asks for approval. A strong one offers a fork.

How the split shifts by level

The rough shape: mid level spends its minutes proving a working system could be built, senior spends them proving the metric and the trade-off were chosen deliberately, staff spends them proving the system is reachable from today, operable when the labels are late, and affordable next quarter.

Module 1. Scope and honesty: which ML rounds this trains for

Time: 20 to 30 minutes reading, 10 to 15 minutes practice.

1a. Predict first

Here is a real-shaped onsite schedule, copied from the kind of confirmation email a recruiter sends. Read the five round descriptions before you read anything else on this page.

The six rounds that share the phrase "machine learning interview"

Six distinct rounds sit behind that phrase. They score different artefacts, and preparing for the wrong one is the most expensive mistake on offer because it costs weeks.

How to tell inside 90 seconds. Listen for what already exists in the prompt.

• Nothing exists yet, and the noun is a product surface ("feed", "search", "recommendations"): ML system design.
• Nothing exists yet, and the noun is a component ("feature store", "experimentation platform", "model registry"): ML platform design.
• A dataset exists: applied modelling.
• A result exists and it is confusing: ML depth, usually on evaluation.
• A function signature exists: ML coding.

Why a modelling interview is not this round

The single most common preparation error is treating an ML design round as a longer modelling round. The two rounds have different deliverables, so they have different failure modes.

The routing diagram

        first line of the prompt
                  |
     +------------+------------+
     |            |            |
 nothing      a dataset    a result
 exists yet   is given     is given
     |            |            |
     v            v            v
+----------+ +----------+ +-----------+
| design   | | applied  | | ML depth  |
| round    | | modelling| | on eval   |
+----------+ +----------+ +-----------+
     |
 product noun or component noun
     |
  +--+-------------+
  |                |
  v                v
+----------+  +-------------+
| product  |  | ML platform |
| ML design|  | design      |
+----------+  +-------------+

Figure routing the first line of an ML round to one of five rubrics

1d. Worked example

The email says: "Onsite for Machine Learning Engineer, Trust and Safety. Four rounds: ML Design (60 min), ML Depth (45 min), Coding (45 min), Behavioural (45 min). The ML Design interviewer is a tech lead on the enforcement platform."

Block 1. PURPOSE: convert the round list into a list of deliverables, because I prepare deliverables, not topics.

Four rounds, three technical. The ML Design round wants a diagram and numbers. The ML Depth round wants spoken mechanism. Coding wants working code. I write those three words at the top of my plan: diagram, mechanism, code. Everything I study has to serve one of them.

The team name is the strongest signal in the email. Trust and Safety means the domain is classification under adversarial pressure with expensive human review, which tells me the ML Design prompt is very likely harmful content, spam, or account takeover.

The error most readers make here is reading "Machine Learning Engineer" and preparing recommendation systems because that is what most study material covers. The team name overrides the job title every time.

Block 2. PURPOSE: decide what I will deliberately not prepare, because preparation is subtraction.

Given enforcement, I drop three things entirely: two-tower retrieval at billion scale, exploration and bandits, and personalisation. Enforcement is a precision and recall economics problem over a stream, not a candidate generation problem.

That frees roughly all of my ML Design time for four things: label sources including human review capacity, precision versus recall costed in currency, appeal and feedback loops, and adversarial drift.

The error most readers make here is additive preparation. They add Trust and Safety material on top of a general syllabus and arrive having half-learned everything.

Block 3. PURPOSE: buy information cheaply before the round, because two sentences of reply beat two hours of inference.

I send the recruiter one message: "For the ML Design round, is the prompt usually a greenfield product system, or an existing pipeline being extended? And does the interviewer expect the modelling layer, the platform layer, or both?"

Recruiters answer this. If the answer is "platform layer", my preparation moves almost entirely to module 5 and the labelling pipeline material, and the case study I rehearse changes.

Block 4. PURPOSE: rehearse the opening, because it is the only fully predictable 40 seconds of the round.

I write and say out loud: "I want to spend the first six or seven minutes on what we are optimising and where labels come from, because in enforcement those two decide the architecture. Then I will size the stream, draw the smallest thing that works, and break it. If you would rather I go deep on one piece instead, say so now and I will restructure."

The error most readers make here is improvising the opening, then spending minute three drawing boxes nobody asked for.

Result. The reply was: "Greenfield product prompt, but she always asks how you would operate it." That moved the last two hours of preparation from architecture drills to monitoring, alert thresholds and the reviewer queue, which is exactly where the round went.

1e. Fade the scaffold

Faded example 1. The email says: "Onsite for Machine Learning Engineer, Ads Ranking. Rounds: ML Design (60), ML Depth (60), Coding (45). The ML Depth interviewer works on auction infrastructure." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: deliverables are a diagram, spoken mechanism, and code. The team name says ads ranking, so the design prompt is click or conversion prediction.
• Block 2: I drop content moderation, recommendation diversity and cold-start-heavy material. I keep calibration, delayed conversions and cost per thousand predictions.
• Block 3: I ask whether the depth round is model-side or serving-side, because "auction infrastructure" could mean either.

Faded example 2. The email says: "Virtual onsite, Machine Learning Engineer, Search Infrastructure. Rounds: ML Design (60), Systems Design (60), Coding (45)." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: three deliverables, and note that one of the three rounds is plain distributed systems design, not ML.
• Block 2: I drop nothing from the ML side yet, because "Search Infrastructure" covers both retrieval quality and index serving. I do subtract the whole recommendation-personalisation stack.

1f. Check yourself

Q1. An interviewer opens with: "We have a model in production that flags risky transactions. Recall is 60 percent and the review team is saturated. What would you do?" Which round are you in, and what is the first thing you change about your plan?

Q2. You are 20 minutes into what you believed was an ML design round and the interviewer says "assume the model is a black box that returns a score between 0 and 1". What just happened, and what do you do?

1g. When not to use this

Round triage is a preparation tool. It is over-applied far more often than under-applied.

Module 2. Problem framing: from a business objective to a trainable one

Time: 35 to 45 minutes reading, 25 to 35 minutes practice.

2a. Predict first

A product manager says: "We want people to spend more time watching." An engineer proposes ranking videos by predicted watch time in seconds. Here are three videos with their measured completion rates.

The chain from money to loss function

Every ML design round contains one translation, done four times, and each step loses information. Naming the chain out loud in the first five minutes is one of the cheapest strong signals available.

Two rules that come out of the chain.

• Each step must be defensible in one sentence. If you cannot say why the trainable target moves the experiment metric, the round is already lost and no amount of architecture recovers it.
• The gap between layers is where gaming lives. The model optimises layer four exactly. It optimises layer one only to the extent that your translation was honest.

Choosing a trainable target, and how each one gets gamed

For any product objective there are usually four or five candidate targets. Enumerating them out loud, with their failure direction, is worth more interview credit than picking the right one silently.

Guardrail metrics, and how much leverage they carry

A guardrail is a metric you are not trying to improve, which can veto a launch. Guardrails are usually treated as a checklist item. They deserve arithmetic, because a small guardrail move can outweigh a large primary move.

Take a steady-state retention model. If a product acquires n new users per day and retains a fraction r of yesterday's users each day, the steady-state daily active user count is:

DAU_steady = n / (1 - r)

r = 0.950  ->  DAU_steady = n * 20.0
r = 0.945  ->  DAU_steady = n * 18.2

Figure the DAU multiplier is very sensitive near high retention

A drop of half a percentage point in daily retention, from 0.950 to 0.945, cuts steady-state DAU by 1 minus 18.2 over 20.0, which is 9.1 percent. That is the whole point of a guardrail: a change that improves revenue per session by 45 percent while costing half a point of retention nets out to roughly 1.455 times 0.909, or a 32 percent gain, not a 45 percent one, and the retention loss keeps compounding after the experiment ends while the revenue gain does not.

What this means for the design. Guardrails belong on the metric that has the largest multiplier, not on the metric that is easiest to compute. In a consumer product that is almost always a retention or return-frequency metric. Put session-level metrics in the primary set and cohort-level metrics in the guardrail set.

Metric cannibalisation between competing goals

Cannibalisation happens when two objectives share a fixed resource. In ranking the shared resource is nearly always slots: positions in a finite list. Every slot given to one objective is taken from another.

The design consequence is that a multi-objective system needs an explicit combination rule, and the weights in that rule are a product decision measured online, not a hyperparameter tuned offline. There is no offline dataset that tells you how much retention a dollar of ad revenue is worth.

The framing diagram

+------------------+     +--------------------+
| business object  | --> | experiment metric  |
| revenue, retain  |     | day-7 retention    |
+------------------+     +--------------------+
                              |         ^
                     must be  |         | guardrail veto
                    justified |         |
                              v         |
                     +--------------------+
                     | trainable target   |
                     | p(watch past 10s)  |
                     +--------------------+
                              |
                              v
                     +--------------------+
                     | loss and weights   |
                     +--------------------+

Figure the four-layer chain, with the guardrail as a veto edge

2d. Worked example

Prompt. "Our short-video feed shows 30 items per session. Leadership wants daily time spent up 10 percent. Frame the ML problem."

Block 1. PURPOSE: pin the objective to a decision, because an objective that decides nothing cannot be optimised.

I ask one question first: is time spent the goal, or is it a proxy for something else? The answer I get is that time spent is the proxy and the real target is retention, because ad revenue scales with DAU.

That reply is the most valuable thing I will hear in the round. It tells me time spent is a product decision metric, not the business objective, so it is legitimate to trade a little of it for retention. Without that sentence I would have optimised a proxy to the point where it hurt the thing it proxies for.

The error most readers make here is accepting the stated metric as the objective. Leadership states proxies constantly. Asking what the proxy is for takes 20 seconds and reframes the entire round.

Block 2. PURPOSE: enumerate trainable targets and eliminate on structural bias, not on taste.

I write four candidates on the board and evaluate each against the duration arithmetic from the pre-question.

The 10 second threshold is a choice with a cost, and I say so: it makes a 12 second video and a 12 minute video equivalent once both pass 10 seconds. That is why it cannot be the only head.

So the answer is not one target. It is a small set of heads with a combination rule: p(watch past 10 s), p(completion) and p(explicit negative), where the last is the signal that the user hid, skipped fast, or reported the item.

The error most readers make here is picking a single target and defending it. The interviewer is not testing whether you pick the same one they would. They are testing whether you can name the failure direction of each option.

Block 3. PURPOSE: derive guardrails from the failure directions I just named, so the guardrail set is not a generic checklist.

Each rejected or partially accepted target has a known failure direction, and each failure direction gets a guardrail that would catch it.

The error most readers make here is listing guardrails that sound responsible but are not caused by anything in the design. A guardrail exists to catch a specific failure you have already named. If you cannot name the failure, delete the guardrail.

Block 4. PURPOSE: write the combination rule and state how its weights get set, because unstated weights are where multi-objective designs quietly fail.

I propose a weighted product on the ranking score, because a product makes a near-zero on any head able to veto the item, which a weighted sum cannot do:

score = p10 ^ a  *  pcomp ^ b  *  (1 - pneg) ^ c

p10    = probability of watching past 10 seconds
pcomp  = probability of completing
pneg   = probability of hide, report or fast skip
a,b,c  = exponents set by online experiment, not offline tuning

Figure the combination rule, with one exponent per objective head

Then I say the part that earns the credit: a, b and c cannot be chosen offline. There is no offline dataset containing the exchange rate between completion and reported content. They are set by running a small grid of exponent settings as an online experiment and reading the guardrails, and they are revisited whenever the product changes.

The error most readers make here is treating the combination weights as hyperparameters and saying they would tune them on a validation set. That answer reveals the modelling-round habit: it optimises the proxy, which is exactly the thing the weights exist to trade off.

Result. The framing I deliver in six minutes is: three heads, a product combination rule, five guardrails each tied to a named failure, and one sentence saying the exponents are a product decision measured online. No architecture has been mentioned yet, and that is correct.

2e. Fade the scaffold

Faded example 1. Prompt: "We run a marketplace. Sellers list items, buyers search. Leadership wants gross merchandise value up." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: I ask what GMV is a proxy for and learn it is a proxy for take rate revenue, and that repeat purchase rate is the real long-run target.
• Block 2: candidate targets are p(click), p(add to cart), p(purchase), and E[order value]. E[order value] has a structural bias toward expensive items exactly as expected watch time biased toward long ones.
• Block 3: guardrails are median listed price of shown items, purchase rate for new sellers, return and refund rate, and 30-day repeat purchase rate.

score = p(purchase) ^ a  *  value_norm ^ b  *  (1 - p(return)) ^ c

value_norm = clip(order_value / median_price_in_category, 0.5, 2.0)

Figure a marketplace ranking rule with the value term clipped

Faded example 2. Prompt: "Design the objective for a notifications system that decides which push notifications to send and when." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the stated objective is sessions per day. On asking, it turns out to be a proxy for retention, and there is a known history of notification fatigue causing uninstalls.
• Block 2: candidate targets are p(open), p(open and then engage for 30 s), and p(disable notifications or uninstall within 7 days). The last is a rare negative event: assume roughly 1 in 2,000 notifications, which makes it 500 times rarer than an open.

send if:  p(engage30) * value  >  p(disable) * cost_disable

cost_disable = lifetime value of a user, in the same units
Figure the send decision as an expected-value threshold

2f. Check yourself

Q1. A colleague proposes training a feed ranker directly on 7-day retention as the per-impression label. Give the strongest technical objection in two sentences.

Q2. Your primary metric is up 3 percent and one guardrail, day-7 retention, is down 0.4 percentage points but is not statistically significant. What do you say?

2g. When not to use this

Multi-head objectives with combination rules are a real cost: more heads, more labels, more calibration, more experiment surface, and a weight vector that has to be maintained forever.

Also skip the whole framing exercise when the target is imposed from outside. Regulatory or contractual detection tasks, such as flagging transactions a regulator defines, come with the label definition already written. There, framing is a 60 second confirmation and the interesting work is entirely in modules 4 and 7.

Module 3. Estimation for ML systems

Time: 45 to 60 minutes reading, 35 to 50 minutes practice.

Capacity estimation in a general system design round sizes storage and bandwidth. In an ML round it does something more useful: it eliminates model families. This module derives six numbers, and every one of them rules an option out. If a number rules nothing out, delete it from your answer.

3a. Predict first

Here is the assumption block for a short-video feed. Nothing else is given.

A colleague proposes running a 6-layer transformer cross-encoder, costing about 11 GFLOPs per item, over all 500 candidates.

The six numbers, and what each one eliminates

Work them in that order. Each one consumes the previous.

Number 1: peak request rate

requests/day = A1 * A2 * A3 = 50e6 * 4 * 6 = 1.2e9
mean rps     = 1.2e9 / 86,400                = 13,900
peak rps     = 13,900 * A6                   = 34,750, call it 35,000

Figure the request-rate derivation, three lines, memorisable

What it eliminates. 35,000 requests per second is comfortably beyond one host but nowhere near the scale that forces exotic infrastructure. It rules out a design where the model is called once per user per day and cached, because the request rate says users come back within a session and expect the feed to change. It does not yet rule out any model.

Number 2: item scores per second, and the model-family cut

This is the number that does the most work in an ML round, and almost nobody computes it.

item-scores/s at peak = peak rps * A5 = 35,000 * 500 = 17,500,000

Figure the ranking stage scores 17.5 million items every second

Now cost each candidate model family in units per item. Two derivations, both from stated assumptions.

Gradient boosted trees. Assume 500 trees of depth 8, so 4,000 node visits per item. Assume 2.5 nanoseconds per node visit, on the grounds that the whole ensemble is tens of megabytes and stays resident in last-level cache, so each visit is a cache hit plus a compare and a branch.

per item      = 4,000 * 2.5 ns          = 10 microseconds
per core      = 1 / 10 us               = 100,000 items/s
cores at peak = 17.5e6 / 100,000        = 175
at 50% target utilisation               = 350 cores
hosts of 16 vCPU                        = 22

Figure the tree ensemble needs 22 ordinary hosts at peak

Transformer cross-encoder. A 6-layer model with hidden size 768 has about 12 times 768 squared parameters per layer, which is 7.1 million, so 42.5 million total. Forward-pass FLOPs are about 2 times parameters times tokens. At 128 tokens per item that is 10.9 GFLOPs, call it 11.

full slate   = 17.5e6 * 1.09e10 = 1.9e17 FLOP/s = 190 PFLOP/s
accelerators = 1.9e17 / 1e14    = 1,900
top-25 only  = 35,000 * 25 * 1.09e10 = 9.5e15 = 95 accelerators

Figure the same model is 20x cheaper when restricted to the top 25

What it eliminates. Three things at once, and this is the payoff for doing the arithmetic.

• A cross-encoder over the full 500 candidates is dead on latency (55 ms per request, shown above) and dead again on cost.
• A cross-encoder over the top 25 survives: 2.7 milliseconds of accelerator time per request and 95 accelerators. That is the arithmetic case for a re-ranking stage, derived rather than asserted.
• Trees over 500 candidates cost 22 hosts, which is small enough that ranking cost stops being an architectural constraint and the design conversation can move on.

Number 3: index footprint

Assume a retrievable catalogue of 100 million items and 256-dimensional embeddings. Compute bytes per vector for each storage choice, then multiply.

The HNSW link cost is derived, not quoted: the bottom layer of the graph stores up to 2M neighbour identifiers per node, so at M = 32 that is 64 identifiers of 4 bytes each, which is 256 bytes per vector regardless of dimension.

What it eliminates. Full-precision vectors plus a graph do not fit on a single ordinary host, so the design must either shard the index, compress the vectors, or both. Naming that fork from one multiplication is worth more than any statement about which index library you prefer. Module 7 chooses between the branches.

Number 4: training rows and bytes per day

impressions/day = requests/day * items shown = 1.2e9 * 10 = 1.2e10
positives at 4% = 4.8e8
negatives       = 1.152e10
negatives kept at 2% sampling = 2.3e8
rows/day        = 4.8e8 + 2.3e8 = 7.1e8

Figure negative downsampling turns 12 billion rows into 710 million

Bytes follow. Assume 1 kilobyte per serialized row: roughly 200 features plus identifiers, context and timestamps.

raw          = 7.1e8 * 1 KB           = 710 GB/day
columnar, 3x = 237 GB/day
90 day window                          = 21 TB

Figure the training corpus at a 90-day retention window

What it eliminates. Training on every logged impression. 12 billion rows a day is 4.4 trillion a year, and the model gains nothing from the 96 percent of negatives that are trivially negative. Downsampling is therefore not an optimisation, it is a design requirement, and it carries an obligation: the model's output probabilities are now wrong and must be corrected. Module 4 derives the correction.

It also eliminates arguments about training compute, at least for trees. Histogram-based boosting costs roughly rows times features per tree:

ops = 7.1e8 rows * 200 features * 500 trees = 7.1e13
on 32 cores at 5e9 ops/s each                = 444 seconds floor
budget 3x to 10x above the floor             = 25 to 75 minutes
cost at $0.04 per vCPU-hour                  = about $1.30 per run

Figure training cost for this model family is operationally free

At $1.30 a run, training cost cannot decide anything, so spending interview minutes on it is a mistake. For a deep model the binding constraint is usually not FLOPs either: reading 237 GB three times at 2 GB/s sustained is about 6 minutes of pure input-pipeline time, which is the number that decides whether you need a cached or sharded reader.

Number 5: retrain cadence

Cadence is usually asserted ("we retrain daily") and almost never derived. There are two honest derivations, and which one applies depends on whether the model consumes item identities or item counters.

Derivation A, from content churn. Assume the catalogue grows 1 percent per day, so 1 million new items daily, and assume items younger than 7 days take 25 percent of impressions because the product promotes new content.

cold impression share after k days = 25% * (k / 7)
tolerance: keep it under 5%
k = 7 * (5 / 25) = 1.4 days  ->  retrain daily

Figure cadence forced by how fast the catalogue turns over

Derivation B, from measured decay. Score today's holdout with the model trained d days ago and record the metric. Assume the backtest gives an AUC decay of 0.0008 per day, and assume the smallest gain your experiment platform can detect and launch is 0.002 AUC.

retrain when accumulated decay = smallest detectable gain
d = 0.002 / 0.0008 = 2.5 days  ->  round to daily

Figure cadence forced by measured staleness, not by habit

What it eliminates. A daily cadence with a batch-only feature pipeline cannot serve an item created three hours ago, because that item has no features until tomorrow's job runs. So either the cadence goes sub-hourly, which is expensive, or item-level engagement features move to a streaming pipeline while the model itself stays daily. That fork is the entire reason module 5 exists, and it was produced by one division.

Number 6: cost per thousand predictions

Define a prediction as one ranked request returning 10 items, so the unit matches the product.

cost per 1,000 requests without re-ranking
  = ($336 + $672) / (1.2e9 / 1,000) = $0.00084

cost per 1,000 requests with re-ranking
  = $5,570 / 1.2e6 = $0.0046

Figure the re-ranking stage multiplies serving cost by about 5.5x

What it eliminates. Nothing yet, and saying so is the point. A 5.5x cost multiplier is only decisive next to revenue per request. State the comparison and hand the decision back: "Re-ranking the top 25 costs about 4.6 tenths of a cent per thousand feed requests. Whether that is worth it depends on revenue per thousand requests, which you would know better than me. If it is above a few cents, this is obviously worth testing; if it is under half a cent, it is not."

That sentence is what separates cost estimation from cost theatre.

The one-page estimation sheet

+---------------------+   +----------------------+
| DAU, sessions,      |-->| peak requests/sec    |
| requests/session    |   | 35,000               |
+---------------------+   +----------------------+
                                     |
        +----------------------------+-------------+
        v                                          v
+----------------------+                +--------------------+
| item-scores/sec      |                | index footprint    |
| 17.5M -> model cut   |                | 128 GB -> shard    |
+----------------------+                +--------------------+
        |                                          |
        v                                          v
+----------------------+                +--------------------+
| rows/day 710M        |                | retrain cadence    |
| bytes/day 237 GB     |--------------->| 1.4 days -> daily  |
+----------------------+                +--------------------+
                    |
                    v
        +--------------------------+
        | cost per 1,000 preds     |
        | $0.00084 to $0.0046      |
        +--------------------------+

Figure the order the six numbers are derived, each feeding the next

3d. Worked example

Prompt. "We run product search over 400 million listings. We want semantic retrieval using 384-dimensional embeddings. Size it and tell me what infrastructure that implies."

Block 1. PURPOSE: put the assumptions on the board first, so the interviewer can correct them before I build on them.

I write four lines: 400 million vectors, 384 dimensions, a target of 25 milliseconds for the retrieval stage, and a peak of 20,000 search queries per second (assumption: e-commerce search runs at lower request volume than feed scrolling because each query is deliberate).

Then I say the sentence that buys goodwill: "If any of those four are wrong by more than a factor of two, tell me now, because the conclusion changes."

The error most readers make here is starting the arithmetic before stating assumptions. When the interviewer corrects an assumption at minute 12, all the arithmetic after it is wasted, and the correction lands as a failure rather than as collaboration.

Block 2. PURPOSE: compute the footprint and kill the options it cannot support.

float32 vector        = 384 * 4        = 1,536 bytes
HNSW links at M = 16  = 32 * 4         =   128 bytes
per vector                             = 1,664 bytes
400M vectors                           = 666 GB

float16 variant       = 768 + 128      =   896 bytes -> 358 GB
PQ at 96 bytes        = 96 + 128       =   224 bytes ->  90 GB

Figure three storage choices for the same 400 million vectors

Three conclusions, each an elimination. 666 GB does not fit any ordinary host, so full precision plus a graph is out for a single node. 358 GB in float16 still does not fit a 256 GB host, so halving precision alone does not save the single-node design. 90 GB with product quantization does fit, with room for the process and the operating system.

The error most readers make here is stopping at "we would use a vector database". The footprint is what decides how many machines that database needs and what it can promise, and a named product does not answer the question.

Block 3. PURPOSE: choose between compression and sharding by naming the price of each, not by preference.

Two viable designs remain, and they fail differently.

I choose PQ plus exact re-scoring, and I say why in one sentence: fan-out multiplies tail latency by the slowest shard, and at 20,000 queries per second I would rather pay a second local lookup than pay a four-way network fan-out on every query.

The recovery pass is concrete: retrieve 1,000 candidates by quantized distance, then re-score those 1,000 against their float16 vectors fetched from a local key-value store, and keep the top 200.

The error most readers make here is treating quantization as free. It trades recall for memory, and the trade is only acceptable when you have named the recovery mechanism and budgeted its time.

Block 4. PURPOSE: verify the choice against the latency budget, because a design that fits in memory can still miss the clock.

re-score 1,000 vectors, 384 dims, float16
  = 1,000 * 384 fused multiply-adds = 384,000 FLOPs
at 20 GFLOP/s per core with vector instructions
  = 19 microseconds of compute

fetch 1,000 vectors of 768 bytes from local memory
  = 768 KB, at 10 GB/s = 77 microseconds

total re-scoring cost: well under 0.2 ms of the 25 ms budget

Figure the recovery pass is negligible against the retrieval budget

The re-scoring pass is essentially free, which is exactly why it is the standard fix. What is not free is the quantized search itself and the network round trip, and those consume the remaining budget.

Result. The answer delivered in about five minutes: 400 million vectors at 384 dimensions is 666 GB uncompressed, which forces either sharding or compression. I compress to 96 bytes per vector, hold 90 GB on one node with three replicas for throughput and availability, retrieve 1,000 approximate candidates and re-score them exactly for under 0.2 milliseconds. Every number came from four stated assumptions.

3e. Fade the scaffold

Faded example 1. Prompt: "Size the ranking stage for a jobs marketplace. 8 million monthly active users, 30 percent visit on a given day, 2 sessions each, 3 result pages per session, 200 candidates ranked per page, peak-to-mean 3." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: assumptions are as stated, plus a ranking latency budget of 60 ms and a model of 300 trees at depth 6.
• Block 2: DAU is 8e6 times 0.30, which is 2.4 million. Requests per day are 2.4e6 times 2 times 3, which is 14.4 million. Mean rps is 14.4e6 over 86,400, which is 167. Peak is 500.
• Block 3: item scores per second at peak are 500 times 200, which is 100,000.

Faded example 2. Prompt: "A fraud system scores every card transaction. 40 million transactions per day, peak-to-mean 4, 500 features, and a hard 50 ms budget because the score sits inside the authorisation path." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: assumptions as stated, plus a fraud base rate of 0.1 percent and a 180-day training window.
• Block 2: mean rate is 40e6 over 86,400, which is 463 per second. Peak is 1,852. Each transaction is scored once, so item-scores per second equals request rate: about 1,900.

3f. Check yourself

Q1. An interviewer says "assume 1 billion items in the catalogue and 512-dimensional float32 embeddings". Give the memory footprint and the first design decision it forces, in under 20 seconds.

Q2. Your ranking stage scores 4 million items per second and you are asked whether you can add a 40-parameter-million neural ranker at 128 tokens per item. Decide, showing the arithmetic.

3g. When not to use this

Estimation is over-performed in ML rounds more often than it is skipped, because it is the part candidates have rehearsed.

One more honest limit. Every number in this module rests on assumptions you invented, and inventing plausible assumptions is a skill an interviewer can only partly assess. If they hand you real numbers, drop yours immediately and say you are doing so. Arguing for your assumption against their measurement is the fastest way to lose a round that was going well.

Module 4. Data and labels

Time: 35 to 50 minutes reading, 25 to 35 minutes practice.

Model architecture is the most rehearsed part of an ML design answer and the least decisive. Label design is the least rehearsed and the most decisive. This module is about where the target column comes from, what it costs, how late it arrives, and how wrong it is.

4a. Predict first

A card issuer wants to detect fraudulent transactions. Here are four candidate definitions of the label "this transaction was fraud".

Where labels come from, and what each one costs

Every label in an ML system comes from one of five sources. Pricing them is a two-minute exercise that reframes most design rounds.

Class imbalance, and the correction downsampling forces

Module 3 downsampled negatives to 2 percent to make the corpus tractable. That is standard, and it breaks the model's output probabilities in a way that is exactly computable.

Downsampling negatives by a factor r leaves positives untouched, so it multiplies the odds of the positive class by 1/r. Reversing it:

odds' = p' / (1 - p')          the model's odds on sampled data
odds  = r * odds'              true odds
p     = r * p' / (1 - p' + r * p')

Figure the prior correction for negative downsampling

Worked, with the fraud base rate from module 3. True positive rate 0.001, negatives kept at r = 0.01.

true odds       = 0.001 / 0.999      = 0.001001
sampled odds    = 0.001001 / 0.01    = 0.1001
model outputs   = 0.1001 / 1.1001    = 0.091

Figure a 0.1% event becomes a 9.1% model output, a 91x inflation

Why this matters beyond tidiness. If the score is only used to rank a review queue, the inflation is harmless because it is monotonic. If the score is compared against a cost threshold, fed into an expected-value calculation, or displayed to an analyst as a probability, the inflation is a production bug. Deciding which case you are in is a design decision, and saying which one out loud is the credit-earning move.

Two things are commonly confused here, so separate them explicitly:

The default answer that survives follow-up: downsample negatives for cost, correct the prior, keep the loss unweighted, and put the business cost asymmetry entirely in the threshold. Weighting the loss and then also moving the threshold applies the same asymmetry twice.

Label delay, treated as a first-class constraint

Assume the dispute maturation curve for the fraud problem: 50 percent of true fraud is reported by day 20, 90 percent by day 45, 99 percent by day 90 (assumption: disputes follow statement cycles, so the mass sits after the monthly statement arrives).

The trap is that unlabelled does not mean negative. If you build a training set from transactions 10 days old and treat every un-disputed transaction as clean, you are mislabelling the majority of positives.

share of true fraud reported by day 10   = about 30%
observed positive rate on 10-day-old data = 0.001 * 0.30 = 0.0003
true positive rate                        = 0.0010

Figure waiting 10 days measures a third of the fraud that occurred

The model trained this way learns a base rate 3.3 times too low and treats 70 percent of the positives as negatives, which is systematic label noise concentrated on exactly the hard cases.

The censored formulation is worth being able to state: a recent unlabelled example is not a negative, it is a positive-with-probability equal to the share of reports still outstanding at its age. Modelling delayed conversions this way is a known line of work in ads, going back to Chapelle's delayed-feedback modelling paper at KDD 2014, and it is the single fastest way to sound like you have shipped one of these.

Cold start, split three ways

"Cold start" is three different problems that people merge. Separating them is worth a minute in any recommendation round.

The design consequence people miss: item cold start is a supply problem, not only an accuracy problem. If new items cannot earn impressions, they never generate the interactions that would let them rank, and the catalogue ossifies. The fix is an explicit exploration budget, stated as a number of slots, which module 2's guardrail on new-item impression share is there to monitor.

Weak supervision: estimating accuracy with no ground truth

When human labels cost dollars, you write labelling functions instead: cheap noisy rules that vote on the label and abstain otherwise. The interesting question is how to combine them when you have no gold labels to score them against.

For binary labels in the range minus one to plus one, and three labelling functions whose errors are conditionally independent given the true label, the expected pairwise agreement between two functions depends only on their accuracies:

E[Li * Lj] = (2ai - 1) * (2aj - 1)

so, using all three pairs:
(2a1 - 1)^2 = E[L1 L2] * E[L1 L3] / E[L2 L3]

Figure recovering each rule's accuracy from agreements alone

Worked, on observed agreement rates from unlabelled data.

E[L1 L2] = 0.60, E[L1 L3] = 0.50, E[L2 L3] = 0.40

(2a1-1)^2 = 0.60 * 0.50 / 0.40 = 0.750 -> a1 = 0.933
(2a2-1)^2 = 0.60 * 0.40 / 0.50 = 0.480 -> a2 = 0.846
(2a3-1)^2 = 0.50 * 0.40 / 0.60 = 0.333 -> a3 = 0.789

Figure three rules ranked by accuracy without a single gold label

The load-bearing assumption is conditional independence, and it is usually false: two keyword rules written by the same person fail on the same examples. The practical answer is to test it, by checking whether observed three-way agreement matches what the pairwise estimates predict, and to hold back a small gold set purely to validate the estimates. This family of methods is what the Snorkel line of work formalised (VLDB 2017), and citing it as a named approach rather than inventing terminology is the honest move.

Active learning under a fixed review budget

Human review capacity is a hard constraint that does not scale with traffic, which makes it one of the few genuinely interesting optimisation problems in an ML design round.

reviewers                = 200
items per reviewer per day = 300
review capacity          = 60,000 labels/day
content volume           = 20,000,000 posts/day
coverage                 = 60,000 / 20e6 = 0.3%

reviewer cost at $15/hour, 300 items in 8 hours
cost per label           = $15 / 37.5 = $0.40
daily labelling cost     = 60,000 * $0.40 = $24,000

Figure the review budget, in items and in dollars

Now the allocation question, which is where the design lives. Assume a violation base rate of 0.2 percent.

The uniform random slice is the part that gets omitted, and it is the only part that makes everything above it identifiable. Its size is derivable, not arbitrary:

random labels/day    = 6,000
violations found     = 6,000 * 0.002 = 12 per day
positives needed for a stable recall estimate = about 100
days to accumulate   = 100 / 12 = 8.3 days

Figure the random slice sets how often you can trust a recall number

That derivation produces an operational fact: your recall metric has a weekly cadence, not a daily one. Anyone proposing a daily recall alert on this system has not done the arithmetic. If you need a faster read, you must enlarge the random slice, which directly reduces enforcement capacity. That trade is the answer.

The label pipeline diagram

+--------------+     +----------------+     +--------------+
| raw events   | --> | sampler        | --> | review queue |
| 20M/day      |     | 70/20/10 split |     | 60k/day      |
+--------------+     +----------------+     +--------------+
                             |                      |
                             v                      v
                     +----------------+     +--------------+
                     | weak labellers |     | gold labels  |
                     | rules, models  |     | + appeals    |
                     +----------------+     +--------------+
                             |                      |
                             +----------+-----------+
                                        v
                              +---------------------+
                              | training corpus     |
                              | with label source   |
                              | and weight per row  |
                              +---------------------+

Figure labels from three sources, tagged by origin, into one corpus

The detail that matters and is nearly always missing: every row carries its label source and a confidence weight. Without that column you cannot ever separate "the model is wrong" from "the label was wrong", and every future error analysis is guesswork.

4d. Worked example

Prompt. "Design the labelling system for harmful content classification. 20 million posts a day, 200 reviewers."

Block 1. PURPOSE: price every available label source before choosing any, because the cheapest adequate label wins.

I list five sources and put a number on each: implicit signals are free but do not measure harm; user reports are free, high volume and low precision, assume around 3 percent of reports are upheld; reviewer labels cost $0.40 each; appeal overturns are high quality but only exist on items already enforced; and prior enforcement decisions are free but are the closed loop I must not train on.

The error most readers make here is starting with the reviewer queue because it is the highest-quality source. Quality per label is the wrong ordering. Labels per dollar, times the value of that label at the decision boundary, is the right ordering.

Block 2. PURPOSE: allocate the fixed review budget, because it is the binding constraint and the split is the design.

I use the 70 / 20 / 10 split derived above and defend the 10 percent slice out loud, because it is the piece that will be challenged: "Without a uniformly random sample I can measure precision but never recall, because I only ever see the items my own model chose to show a reviewer."

Then I state the operational consequence I derived: 6,000 random reviews a day yield about 12 violations a day, so a trustworthy recall figure takes roughly eight days to accumulate. Recall is a weekly metric here. Precision, which is measured on the enforcement slice, is a daily metric.

The error most readers make here is proposing active learning as the whole answer. Pure uncertainty sampling maximises training value and destroys measurement, because the labelled set becomes a function of the current model and cannot be used to estimate anything about the population.

Block 3. PURPOSE: multiply the effective label supply with programmatic labels, and prove they are good enough without buying gold labels.

60,000 reviews a day against 20 million posts is 0.3 percent coverage, which is not enough to train a model that must generalise across dozens of policy categories. So I add labelling functions: keyword and phrase rules, a reused classifier from an adjacent policy area, a media hash match against known violating assets, and account-level reputation.

I estimate each function's accuracy from pairwise agreement using the triplet identity above, then validate the independence assumption against the small gold set, then combine into probabilistic labels with the estimated accuracies as weights.

The error most readers make here is treating weak labels as free extra training data. They are correlated noise. Used without accuracy estimation and without a gold-label validation set, they teach the model the rules' blind spots and the offline metric goes up while enforcement quality goes down.

Block 4. PURPOSE: restore the probability, because both the sampling and the weak labels broke it.

Two distortions stack. First, negatives were downsampled for corpus size, so the prior correction from earlier in this module applies. Second, and more subtly, the reviewed examples came from a biased sampler, so the model's training distribution is not the serving distribution at all.

My fix is explicit and separable:

• Correct the downsampling analytically, with the odds formula. It is exact and needs no data.
• Correct the sampling bias by fitting the final calibration layer, isotonic or Platt, only on the uniformly random slice, because that slice alone is drawn from the serving distribution.
• Recheck calibration weekly on that same slice, and alert on the gap between predicted and observed prevalence rather than on raw score distribution.

The error most readers make here is calibrating on the full labelled set. It is the largest and most tempting dataset available and it is drawn from the wrong distribution, so it produces a confidently wrong probability.

Result. The deliverable is: five label sources priced, a 70 / 20 / 10 review split with the random slice defended by a recall-precision-of-measurement argument, weak supervision with accuracies estimated from agreement, a corpus where every row is tagged with its source and weight, and calibration fitted only on the unbiased slice. Not one sentence about model architecture, and that is the correct allocation.

4e. Fade the scaffold

Faded example 1. Prompt: "Design the labels for an ads conversion model. A click is observed instantly; a purchase can happen up to 30 days later; advertisers are billed per conversion." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: sources are click logs (instant, free), advertiser-reported conversions via a pixel or server callback (delayed, mostly reliable), and advertiser-reported offline conversions (very delayed, uploaded in batches).
• Block 2: the maturation curve, assumed, is 60 percent of conversions within 1 day, 85 percent within 7 days, 99 percent within 30 days.
• Block 3: a naive model trained on data from 2 days ago sees 60 percent of the conversions, so it under-predicts conversion rate by roughly a factor of 1 over 0.6, which is 1.67.

Faded example 2. Prompt: "A new video platform launches next month. Zero users, zero interactions, 500,000 seeded videos. Design the label and data strategy for the first six months." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: at launch there are no behavioural labels at all, so this is system cold start. Content metadata exists: title, description, category, creator, thumbnail, duration.
• Block 2: the day-one ranker is not a model. It is editorial curation plus popularity, and saying so is the right answer rather than a concession.

4f. Check yourself

Q1. A teammate says "we have 400 million labelled examples from the current rules engine, so data is not our problem". Give the two-sentence objection.

Q2. You downsample negatives at 5 percent and the model outputs 0.4 for an example. What is the corrected probability, and when do you not bother?

4g. When not to use this

The machinery in this module (weak supervision, active learning, delayed-feedback modelling, sampling splits) is expensive to build and expensive to operate.

The general rule: label infrastructure earns its place when human attention is the scarce resource. If your labels are free behavioural signals and arrive in seconds, most of this module is not your problem and you should say so and move to module 5, where your problem probably is.

Module 5. Features and the feature store

Time: 35 to 50 minutes reading, 25 to 40 minutes practice.

Training and serving skew is the most common real bug in production ML and the least discussed in interview material. It is not a modelling problem. It is a problem of two pieces of code computing what is supposed to be the same number, at different times, from different data.

5a. Predict first

Here is a feature called purchases_7d, defined twice. The first is the training query. The second is the serving code.

-- TRAINING
SELECT l.user_id, l.label, COUNT(*) AS purchases_7d
FROM   labels l
JOIN   purchases p ON p.user_id = l.user_id
WHERE  p.day BETWEEN l.day - 7 AND l.day
GROUP BY l.user_id, l.label

// SERVING
count = store.count(user_id, "purchase",
                    from = request_ts - 7 days,
                    to   = request_ts)

The three places a feature can be computed

Every feature is computed in one of three modes, and the choice is forced by freshness, not by importance.

The rule that decides. Ask what the feature's value would be if it were one hour stale, then ask whether the decision changes. If it does not, use batch, because batch is one or two orders of magnitude cheaper to operate and can be recomputed from history.

Sizing batch is one multiplication. For 50 million users with 300 numeric features at 4 bytes:

50e6 * 300 * 4 bytes = 60 GB per full snapshot
recomputed nightly, 30-day history retained = 1.8 TB

Figure the entire user feature table is 60 GB, which is small

That number eliminates an argument. At 60 GB, a nightly full recompute is cheap and simple, so incremental update machinery for user features is unjustified. The same multiplication over 100 million items with a 15 minute refresh gives a completely different answer, which is why item features go to streaming and user features usually do not.

Point-in-time correctness, worked

A point-in-time correct join answers: what did we know about this entity at the exact instant the prediction was made? Anything else leaks.

Here is a single user's event log and two candidate training rows.

The third row is the one that catches experienced people out, because it looks like ordinary engineering: you have a feature table, you join it to your labels. Feature tables hold current values. Joining current values to historical labels leaks the entire future.

What the design must therefore contain, stated as three requirements:

• Every feature value is stored with the timestamp from which it was valid, not only its value.
• The training join is an as-of join on that timestamp, with a strict inequality against the prediction time.
• The join carries a deliberate lag equal to the feature's real serving delay. If a streaming feature is typically 90 seconds behind live at serving time, the training join must also cut 90 seconds early, otherwise training sees a fresher feature than serving ever will.

That third requirement is the one almost nobody states, and it converts a generic answer into a specific one.

Offline and online consistency: three strategies, priced

The third is the strongest and its cost is computable. Using module 3's numbers:

per-request context and user features
  1.2e9 requests * 200 features * 4 bytes  = 960 GB/day

per-impression item features
  1.2e10 impressions * 100 features * 4 B  = 4.8 TB/day
  compressed 3x                            = 1.6 TB/day
  7-day hot retention                      = 11 TB

Figure logging what was served costs about 1.6 TB per day compressed

The design that follows: log everything served into a short-retention store, join labels as they arrive over the following days, then downsample negatives at that point and promote the survivors into the long-retention training corpus. That ordering matters, because you cannot downsample before the labels exist without discarding the positives you have not yet learned about.

The honest limitation of log-at-serve, and the reason nobody uses it exclusively: a feature you invent today has no history. You cannot evaluate it until you have shipped it into the serving path and waited. So real systems run a hybrid: logged features for anything already in production, and a backfill path with point-in-time joins for anything new. Saying that out loud is what a practitioner answer sounds like.

The skew taxonomy, and how each type is detected

The first four are bugs and are fixable. The fifth is drift and is expected, and confusing the two is a common interview stumble: a drifting feature needs retraining, a skewed feature needs a code fix, and retraining a skewed feature just relearns the bug.

The feature layer diagram

+--------------+   +----------------+   +----------------+
| event stream | ->| stream compute | ->| online store   |
| clicks, buys |   | 15 min windows |   | key-value, ms  |
+--------------+   +----------------+   +----------------+
       |                                        ^
       v                                        |
+--------------+   +----------------+           |
| event lake   | ->| batch compute  | ----------+
| raw, dated   |   | nightly, 60 GB |
+--------------+   +----------------+
       |                    |
       |                    v
       |           +-----------------+
       +---------> | offline store   |
                   | as-of joins     |
                   +-----------------+
                            |
                            v
                   +-----------------+
                   | training corpus |
                   +-----------------+

Figure one definition, two paths, with the online store fed by both

The single property that makes this a feature store rather than two pipelines: the same registered definition drives both the streaming path and the batch path, and the offline store keeps every value with its valid-from timestamp so an as-of join is possible.

5d. Worked example

Prompt. "Design the feature layer for the short-video feed ranker from module 3."

Block 1. PURPOSE: classify features by freshness requirement, because freshness picks the pipeline and nothing else does.

I split the feature set into three groups and give each a required staleness bound, then let the bound choose the mode.

The error most readers make here is sorting features by predictive importance and then arguing about pipelines. Importance does not choose a pipeline. Staleness tolerance does, and a highly important feature that is stable over a day belongs in the cheapest pipeline available.

Block 2. PURPOSE: choose the consistency mechanism, and pay its price out loud.

I choose log-at-serve for everything already in production, with the storage cost stated: about 1.6 TB per day compressed, 11 TB of hot storage for a 7-day label window. I say the number because it is the objection an interviewer will raise, and pre-empting it costs one sentence.

For new features that have never been served, I use point-in-time backfill with the deliberate lag rule: each feature's backfill cuts off at the prediction time minus that feature's measured serving delay. The streaming features carry a 90 second lag; the batch features carry a lag equal to the age of the most recent completed job, which at serving time averages 12 hours.

The error most readers make here is choosing one mechanism for everything. Log-at-serve cannot evaluate a new feature; backfill cannot guarantee consistency. Real designs run both and say which applies when.

Block 3. PURPOSE: make the offline join correct, in a way I can state as a rule rather than as an intention.

I write the join rule as three clauses: as-of on valid_from, strictly less than prediction_ts, minus the per-feature serving lag. Then I add the test that proves it, which is the part that separates this answer from a slogan:

For a random sample of 10,000 logged predictions, recompute every feature through the offline path and compare it to the value that was actually logged at serving time. Report the fraction of features that differ by more than a rounding tolerance. Gate every model release on that fraction being under a stated threshold, for instance 0.1 percent of feature-value pairs.

The error most readers make here is describing point-in-time correctness as a principle rather than as an executable check. The principle is free to state and constrains nothing. The check either runs in the pipeline or it does not, and only the check stops the leak.

Block 4. PURPOSE: budget the online read path, because a correct feature that arrives late is a wrong feature.

The user features are one batched read per request: 120 values, one key, one round trip. Trivial.

The item features are not trivial, and the arithmetic shows why:

per request: 500 candidates * 100 item features * 4 bytes = 200 KB
at 35,000 requests/second = 7 GB/s of feature traffic

Figure fetching item features remotely needs 56 gigabits per second

7 GB per second is enough to make the feature store, not the model, the most expensive component in the system. So I use the shape of the traffic against it. Item popularity in a feed is heavily skewed, so I hold a local in-process cache of the hot items on each ranking host:

hot set: top 1,000,000 items
cache size: 1e6 * 100 features * 4 bytes = 400 MB per host
assume 90% of candidate lookups hit that set
remote traffic drops to 0.7 GB/s across 22 hosts
per host: about 0.25 gigabits per second

Figure a 400 MB local cache removes 90 percent of feature traffic

The cache carries an obligation: it is a second copy of the feature values, so it has its own staleness. I bound it by refreshing hot-item entries every 60 seconds from the streaming store and by tagging every logged feature vector with the version actually used, so the training data records what the model really saw rather than what the store currently holds.

The error most readers make here is treating feature retrieval as a detail below the level of the diagram. At 500 candidates per request it is frequently the dominant cost in the whole serving path, and the candidate who notices that is visibly ahead of the one who spends the same minute on the loss function.

Result. Four groups of features assigned to pipelines by staleness tolerance, log-at-serve plus lagged backfill as the consistency mechanism, an executable skew test gating release, and a 400 MB per-host cache derived from a 7 GB per second problem.

5e. Fade the scaffold

Faded example 1. Prompt: "Design the feature layer for the fraud scorer from module 3: 40 million transactions a day, 500 features, a hard 50 ms budget inside the authorisation path." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: three groups. Card and account features (batch, daily). Velocity features such as transactions in the last 1, 10 and 60 minutes (streaming, seconds). Transaction-intrinsic features such as amount, merchant category and country (request time, free).
• Block 2: log-at-serve is mandatory here, not optional, because the scorer's decisions are auditable and a regulator may ask what the model saw.
• Block 3: the as-of join must cut at the authorisation timestamp exactly, and velocity features must exclude the current transaction, which is the single most common leak in fraud systems.

Faded example 2. Prompt: "Two teams, search ranking and ads ranking, both compute a feature called user_category_affinity_30d. Their values disagree by 12 percent on the same users. Design the fix." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: this is a brownfield organisational problem wearing a technical costume. Both teams are correct according to their own specification, so the first task is to find out whether the two specifications differ or the two implementations differ.
• Block 2: the diagnostic is to run both implementations over one identical input snapshot. If they agree on identical inputs, the problem is source data. If they disagree, the problem is logic.

5f. Check yourself

Q1. A model gains 3 points of offline AUC from one new feature and shows no online effect. Name the three checks you run, in order.

Q2. Give one feature that must be request-time, one that must be streaming, and one that must be batch, for a food delivery ETA model. Justify each by staleness tolerance, not by importance.

5g. When not to use this

A feature store is infrastructure, and infrastructure built for one model is a liability rather than an asset.

When streaming features are the over-engineering. If no feature's staleness tolerance is under an hour, a streaming pipeline buys nothing and costs you a second execution engine, a second failure mode, and replay semantics that will not match the live run. Batch every hour is dramatically simpler than a stream, and the correct answer surprisingly often.

When request-time computation is the over-engineering. If the value is stable over a minute and the cost per request is more than a millisecond, it belongs in a store. The test is whether the value would differ between two requests one minute apart, not whether it feels dynamic.

Module 6. Retrieval then ranking, as the reusable serving pattern

Time: 30 to 40 minutes reading, 20 to 30 minutes practice.

Almost every large ML serving system has the same shape: cheaply narrow a huge pool, then spend increasing amounts of compute on an increasingly small set. This module derives that shape from a latency budget rather than presenting it as a convention, and then breaks the budget on purpose.

6a. Predict first

A candidate proposes this latency budget for a feed request with a 300 millisecond end-to-end target. Every figure is the stage's p99.

Why the funnel exists, in one multiplication

Take the module 3 numbers: 100 million retrievable items, a tree ensemble costing 10 microseconds per item, and a ranking budget of 45 milliseconds.

scoring the whole catalogue once
  = 1e8 items * 10 us = 1,000 seconds per request

Figure ranking everything is not slow, it is impossible by 4 orders

The funnel is not an optimisation. It is the only thing that fits. The design question is therefore not whether to have stages but how many items each stage sees and how much it may spend on each.

The balanced funnel principle

Useful heuristic: give each stage roughly the same total time, which means the per-item budget must rise in inverse proportion to the item count.

Two things fall out of that table and both are worth saying in a round.

• Retrieval cannot look at each item. At 0.3 nanoseconds per catalogue item, no per-item computation is possible at all, so retrieval must be a structure, not a scan. That is the entire argument for an index, derived rather than asserted.
• The ranking stage has 90 microseconds of budget per item and uses 10. Ranking is therefore not latency constrained, it is cost constrained, because the fleet is sized by throughput. That distinction changes which optimisation is worth doing: batching and caching help, and shaving microseconds off tree traversal does not.

The filtering stage, and the trap in it

Filtering removes items the user must not see: already watched, blocked creators, region restrictions, policy holds, and inventory that is out of stock. It is boring and it is where recall silently dies.

The trap is filtering after retrieval. If a filter passes a fraction f of items and you need k survivors:

candidates to retrieve = k / f

f = 0.05, k = 500  ->  retrieve 10,000, not 500
f = 0.5,  k = 500  ->  retrieve 1,000

Figure post-filtering forces over-retrieval by one over the pass rate

At f = 0.05, retrieving the usual 1,000 candidates leaves 50 survivors, so the ranking stage receives one tenth of what it was designed for and quality collapses in a way that never appears in an offline metric computed on unfiltered data.

Three responses, and which is right depends on what the filter is:

The re-ranking stage, and what it is actually for

Ranking is pointwise: each item is scored without knowing what else is on the slate. That is a modelling convenience, and it is wrong in a specific way. The value of the fourth item depends on the first three.

Everything that depends on the slate belongs in re-ranking:

• Diversity and dedup: no more than two items from one creator, no near-duplicate videos.
• Freshness and exploration quotas: at least one item under 48 hours old, stated as a slot count so it is auditable.
• Business rules: ad load, sponsored placement, editorial pins.
• Slate-level models: a sequence model that scores the ordered list rather than the items.

The design point that decides whether the rule survives contact with production: a diversity rule implemented as a penalty inside the ranking score is worse than the same rule implemented as a slot constraint at re-rank, because the penalty is opaque, needs retuning after every model change, and cannot be audited. Slot constraints are checkable by looking at the output.

The funnel diagram

+---------------------+
| catalogue 100M      |
+---------------------+
          |
          v  several sources in parallel
+---------------------+  +--------------------+
| ANN embedding       |  | recent + follows   |
| 400 items, 25 ms    |  | 600 items, 15 ms   |
+---------------------+  +--------------------+
          |                        |
          +-----------+------------+
                      v
          +---------------------+
          | filter + dedup      |
          | 1,000 -> 500        |
          +---------------------+
                      v
          +---------------------+
          | rank 500 -> 50      |
          +---------------------+
                      v
          +---------------------+
          | re-rank 50 -> 10    |
          | slate rules apply   |
          +---------------------+

Figure the funnel, with retrieval sources running in parallel

The parallel retrieval sources matter to the budget: two sources running concurrently cost the maximum of their latencies, not the sum, so the retrieval stage costs 25 ms rather than 40.

6d. Worked example

Prompt. "Give me the latency budget for the feed request, and tell me what happens when you miss it."

Block 1. PURPOSE: find the real budget by subtracting what I do not control, before allocating anything.

The product target is 300 ms end to end at p99. Client network round trip on mobile is assumed at 80 ms p99, and I do not control it. Serialisation and transfer of the response is 25 ms. That leaves 195 ms of server time to allocate.

I say the assumption out loud because it is the one most likely to be wrong: "I am assuming 80 ms of network at p99. If your clients are mostly on fixed broadband that is closer to 30 and I have another 50 ms to spend, so tell me if that is wrong."

The error most readers make here is allocating the full 300 ms across server stages, which quietly assumes an instantaneous network and produces a design that misses the product target on every mobile request.

Block 2. PURPOSE: allocate by parallelism first, because concurrent stages cost the maximum rather than the sum.

I draw the dependency order before assigning any numbers. User feature fetch and retrieval-source lookups can start together as soon as the user identifier is known. Item feature fetch cannot start until candidates exist. Ranking cannot start until item features arrive.

t=0    gateway, auth                       10 ms
t=10   [user features 25 | retrieval 25]   25 ms in parallel
t=35   filter and dedup                    10 ms
t=45   item feature fetch                  35 ms
t=80   rank 500                            45 ms
t=125  re-rank 25                          30 ms
t=155  blending and slate rules            10 ms
t=165  serialise                           25 ms
t=190  done, against a 195 ms server budget

Figure the sequential path, with one parallel section at t=10

The parallel section buys 25 ms, and that 25 ms is the entire difference between a design that fits and one that does not.

The error most readers make here is drawing the pipeline as a straight line because that is how diagrams are drawn, then never noticing that two of the stages have no dependency on each other.

Block 3. PURPOSE: convert the stage figures from p99 targets into medians with reserved slack, so the budget can actually be met.

The 190 ms above is the sum of stage p99s, which the pre-question showed is the wrong quantity to sum. So I restate every figure as a median and reserve the remainder.

81 ms of medians against a 195 ms server budget leaves 114 ms of slack, which is 58 percent. That is the number that makes the p99 target achievable: any single stage can go to its p99 and several can go well past it while the request still lands inside the budget.

The error most readers make here is treating a budget as an allocation of the target. It is an allocation of the median, with the remainder held as tail absorption. Allocate the target instead and the first stage to jitter blows the whole budget, because nothing was held back to absorb it.

Block 4. PURPOSE: design the degradation ladder, because the budget will be exceeded and the only question is what happens then.

A budget without a degradation plan is a wish. I put a deadline on the request at entry and check it at each stage boundary.

The last row is the important one and is where the parallel design pays off a second time: independent retrieval sources mean a slow source degrades the result rather than failing the request. A single retrieval source has no such property, and that is a real argument for having more than one that has nothing to do with recall.

Result. 195 ms of server budget, one parallel section worth 25 ms, stages budgeted at medians summing to 81 ms with 58 percent slack, and a four-rung degradation ladder tied to elapsed-time checkpoints rather than to per-stage timeouts.

6e. Fade the scaffold

Faded example 1. Prompt: "Design the funnel for product search over 400 million listings. Target 200 ms end to end, desktop and mobile mixed, so assume 50 ms of network." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: 200 minus 50 network minus 20 serialisation leaves 130 ms of server budget.
• Block 2: query understanding and user feature fetch run in parallel with nothing else, because retrieval depends on the parsed query. Lexical and vector retrieval then run in parallel with each other.
• Block 3: medians are query parse 5, retrieval 15, fusion and filter 5, item features 10, rank 400 items 15, re-rank 40 with a cross-encoder 20, serialise 8. Sum of medians is 78 against 130, which is 40 percent slack, tighter than the feed but workable.

Faded example 2. Prompt: "A notifications system decides, once per hour, which of 40 million users to send a push to, from a pool of 10,000 candidate messages. There is no request latency budget at all, because it is a batch job." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: there is no user-facing latency, so the binding constraint is the batch window and total compute cost, not milliseconds.
• Block 2: naive full scoring is 40e6 users times 10,000 messages, which is 4e11 pairs. At 10 microseconds per pair that is 4e6 core-seconds, or 1,111 core-hours per run. At hourly cadence that is 1,111 cores running continuously, at $0.04 per vCPU-hour, or about $32,000 per month.

6f. Check yourself

Q1. Your filter passes 20 percent of retrieved items and the ranker needs 500 candidates. How many do you retrieve, and what is the second-order effect of that decision?

Q2. Someone proposes moving the diversity rule from re-ranking into the ranking model as a feature. Give one argument for and one against.

6g. When not to use this

The multi-stage funnel is the default in large systems and is over-applied in small ones, where it triples the number of components for no benefit.

When a re-ranking stage specifically is not justified. If your ranking model already fits in the per-item budget for the whole candidate set, a second stage adds latency and cost for nothing. The re-ranker exists only because a better model is too expensive to run on all candidates. Measure that first: if the heavy model costs less than the per-item budget from the balanced funnel table, use it everywhere and delete a stage.

When even the ranking model is not justified. If the catalogue is under a few thousand items and engagement is dominated by recency or popularity, a sorted query is competitive with a model and needs no training data, no feature pipeline and no monitoring. The correct interview answer is sometimes to build the funnel's structure now and populate it with rules until there is enough data to train on. Saying that costs nothing and demonstrates the restraint that the case-study modules keep scoring.

Module 7. Retrieval infrastructure

Time: 35 to 50 minutes reading, 25 to 35 minutes practice.

Module 6 derived that retrieval must be a structure rather than a scan. This module chooses the structure, on build cost, query cost and memory, and then handles the part that decides whether the system survives contact with production: keeping the index fresh.

7a. Predict first

Three index configurations for the same corpus of 100 million 256-dimensional vectors. The corpus grows by 1 million vectors per day.

The families, compared on the three costs that matter

Locality-sensitive hashing appears in a lot of older material. It is worth naming and dismissing in one sentence: for the same recall it generally needs far more memory than a graph index, and it lost the practical argument once graph and quantization methods matured, so proposing it today invites a follow-up you do not want.

Query cost, derived rather than quoted

Both major families have a query cost you can compute from their parameters, which means you can defend a configuration instead of naming a product.

IVF plus PQ. With N = 100 million, nlist = 4,096 and nprobe = 32:

vectors per list      = 1e8 / 4096          = 24,414
vectors scanned       = 24,414 * 32         = 781,250
per distance, PQ, 64 subquantizers          = about 128 ops
ops per query         = 781,250 * 128       = 1.0e8
at 2e10 ops/s per core                      = 5 ms

Figure IVF plus PQ costs about 5 ms of one core per query

HNSW. With efSearch = 128 and M = 32, a search visits on the order of efSearch times M candidate nodes:

distance computations = 128 * 32            = 4,096
per distance, 256 dims float16              = 512 flops
flops per query       = 4,096 * 512         = 2.1e6
at 2e10 flop/s per core                     = 0.1 ms

Figure the graph index costs about 0.1 ms of one core per query

So the graph is roughly 50 times faster per query and costs roughly 11 times the memory. Neither wins in the abstract. What decides is query rate, and that break-even is computable.

The break-even between memory and CPU

Assume $0.04 per vCPU-hour and $0.005 per gigabyte of RAM per hour (assumption: these are the ratios implied by ordinary general-purpose cloud instance pricing, where memory and cores are sold together in fixed ratios; substitute your own and redo the two lines).

HNSW    per replica: 77 GB and Q * 0.0001 core-seconds per second
IVF-PQ  per replica:  7 GB and Q * 0.005  core-seconds per second

daily cost, HNSW   = 9.6e-5 * Q  + 9.24
daily cost, IVF-PQ = 4.8e-3 * Q  + 0.77

equal when  4.704e-3 * Q = 8.47  ->  Q = 1,800 queries/second

Figure below 1,800 queries per second, quantize; above it, use a graph

That single number is the answer to "which index would you use", and it is derived from four inputs the reader can change. Below roughly 1,800 queries per second per replica, memory dominates the bill and quantization is right. Above it, CPU dominates and the graph is right. At the 20,000 queries per second of the product search example, the graph wins by roughly a factor of eight on total cost.

The caveat that keeps this honest: the two configurations do not deliver identical recall. The comparison is only valid once both have been tuned to the same measured recall at the same k, which is what the next section is for.

Build cost: measure small, extrapolate, never quote

Graph construction is roughly N log N in distance computations, and the constant depends on your hardware, dimensionality and parameters far too much to quote. So measure it.

build 1% of the corpus, 1,000,000 vectors, and time it
scale factor to 100,000,000 vectors:

  (100e6 / 1e6) * (log2(100e6) / log2(1e6))
  = 100 * (26.6 / 19.9)
  = 133

Figure extrapolating a 1 percent build to the full corpus

If the 1 million vector build takes 4 minutes on your machine, budget roughly 9 hours for the full one. That estimate is worth an order of magnitude more than any published benchmark, because it is measured on your data with your parameters.

The number matters because it decides your refresh strategy. A 9 hour build cannot run hourly. It can run nightly, which means everything fresher than one day has to come from somewhere else.

Measuring recall without a labelled dataset

The recall of an approximate index is measurable exactly, and cheaply, and almost nobody in an interview says how. Take a sample of queries, compute their true nearest neighbours by brute force once, and store that as ground truth.

1,000 sampled queries against 100,000,000 vectors
distance computations = 1e3 * 1e8 = 1e11
flops                 = 1e11 * 512 = 5.1e13
at 2e10 flop/s per core = 2,560 core-seconds
on 32 cores             = 80 seconds

Figure exact ground truth for a recall curve costs 80 seconds

With that ground truth fixed, sweep efSearch or nprobe and plot recall at k against p99 latency. Pick the point where the recall curve flattens. Re-run it after every index rebuild and after every embedding model change, and alert if the operating point moves.

Why this earns credit: "we would tune for recall" is what everybody says. "Ground truth for a thousand queries costs 80 core-seconds, so I would regenerate it on every rebuild and gate the deploy on the recall-latency curve" is a design, not an intention.

Freshness: the two-tier index

The build cost derivation says the main index refreshes nightly. The product usually needs new items searchable in minutes. Both are satisfied by splitting the index in two.

+------------------------+     +---------------------+
| base index, frozen     |     | hot index           |
| 100M vectors, 77 GB    |     | 1M vectors, 1.3 GB  |
| rebuilt nightly, 9 hrs |     | rebuilt every 10 min|
+------------------------+     +---------------------+
            |                            |
            +-------------+--------------+
                          v
              +------------------------+
              | merge top-k from both  |
              | apply tombstone filter |
              +------------------------+
                          v
              +------------------------+
              | candidates to filter   |
              +------------------------+

Figure a frozen base index plus a small hot index, merged at query

The hot index sizing is derived: 1 million new items per day at 1,280 bytes each is 1.3 GB, which builds in seconds and can be rebuilt from scratch every ten minutes rather than updated in place. Rebuilding from scratch is dramatically simpler than incremental maintenance, and at 1.3 GB it is affordable, so simplicity wins.

Deletion is handled by a tombstone set applied after the merge, not inside either index. The set is small: at an assumed 2 percent deletion per week, it holds 2 million identifiers, which as a hash set of 8 byte keys is 16 MB.

When the base index must be rebuilt is also derivable rather than a matter of taste:

The last row is the one that causes real incidents. Changing the embedding model requires re-embedding the entire corpus and swapping indexes atomically. Serving a mixed index during a partial migration produces results that are not wrong in an obvious way, which is the worst kind of wrong.

Hot swap and the memory cliff

Swapping a 77 GB index means holding the old and new copies simultaneously, which needs 154 GB plus working memory. Two options:

The second is usually correct and its cost must be stated: for the duration of the roll, two identical queries can return different results. If the product requires stable pagination or reproducible results within a session, pin a session to an index version, which is one more thing to build.

Filtered search, and where graph indexes break

Module 6 covered over-retrieval for post-filtering. Graph indexes have an additional failure that quantized list-based indexes do not.

A graph search walks from an entry point toward the query. If eligibility is checked during traversal and only 1 percent of nodes are eligible, the walk spends its budget on ineligible nodes and can terminate in a region with no eligible neighbours at all. Recall does not degrade gracefully; it collapses.

That last row is the one worth remembering. When a filter is very selective, the filter itself is the better retrieval structure, and insisting on vector search is the mistake.

7d. Worked example

Prompt. "Pick and configure the retrieval index for the 400 million listing product search from module 3, and tell me how it stays fresh."

Block 1. PURPOSE: reduce the requirements to the three numbers that select a family, before naming any technology.

I write three numbers on the board: 400 million vectors at 384 dimensions, 20,000 queries per second at peak, and a 25 millisecond retrieval budget. Then I add a fourth that nobody gave me and that decides the refresh design: how often does the catalogue change? I ask, and assume 2 percent of listings change price or availability per hour, and 1 percent of listings are new per day.

The error most readers make here is naming a vector database in the first sentence. The product does not choose itself; the three numbers do, and a named product without the numbers cannot be defended when the interviewer changes one of them.

Block 2. PURPOSE: apply the memory-versus-CPU break-even, because at this query rate it decides the family.

At 20,000 queries per second, spread over replicas, the per-replica rate is far above the 1,800 queries per second break-even derived above, so CPU dominates the bill and the graph index wins. But module 3 showed the graph in float16 is 358 GB, which does not fit one host.

So I combine, rather than choose: HNSW graph over product-quantized vectors, with exact re-scoring of the top 1,000 from a local float16 store.

per vector: 96 B quantized + 128 B links (M = 16)  = 224 B
400M vectors                                        =  90 GB
plus float16 vectors for re-scoring, on local SSD   = 307 GB on disk
three replicas of 128 GB RAM

Figure the graph over quantized vectors fits one host per replica

The error most readers make here is treating the choice as graph versus quantization. They are orthogonal: the graph decides how you navigate, the quantization decides how much each vector costs. Combining them is standard and saying so distinguishes a designed answer from a recalled one.

Block 3. PURPOSE: design the refresh path, because catalogue churn, not query volume, is what will actually break this.

Three distinct kinds of change, three distinct mechanisms, and conflating them is the common failure:

The third row is the one that matters most, and it is a modelling decision disguised as an infrastructure one: putting a fast-changing attribute into the embedding forces an index update on every change. Keeping it out of the embedding turns 8 million hourly index updates into zero.

The error most readers make here is designing an index that must be updated whenever anything about an item changes. The embedding should carry only slow-moving semantics.

Block 4. PURPOSE: define the measurement that tells me it is working, because an approximate index degrades silently.

Three measurements, each with a cadence and a trigger:

• Recall at 100 against exact ground truth, on 1,000 fixed probe queries. Ground truth costs about 80 core-seconds to recompute. Run after every nightly rebuild, alert on a drop of 2 points.
• p99 retrieval latency, per replica, per index version. Alert on a 20 percent increase, since a rising latency at fixed efSearch usually means tombstone accumulation.
• Hot index coverage: the fraction of results in the final slate that came from the hot index. If that goes to zero, the hot path has silently broken, and no other metric would show it.

The third is the one people forget. A merge path that silently stops contributing looks perfectly healthy on latency and error rate, and the only symptom is that new listings never appear, which the business notices weeks later through a complaint.

Result. A quantized graph index at 224 bytes per vector, 90 GB per replica, three replicas, a 10 minute hot index, a tombstone set, price and stock deliberately excluded from the embedding, and three measurements with cadences and triggers.

7e. Fade the scaffold

Faded example 1. Prompt: "Semantic search over 3 million internal documents. 5 queries per second. Documents change constantly, and a change must be searchable within a minute." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: the numbers are 3 million vectors, 5 queries per second, and a 60 second freshness requirement. The query rate is four orders of magnitude below the break-even, so CPU cost is irrelevant and simplicity should win.
• Block 2: at 768 dimensions and float32, 3 million vectors is 9.2 GB, which fits comfortably in the memory of one ordinary host, so no quantization and no sharding are needed. A flat index would cost 3e6 times 768 times 2 flops, which is 4.6 GFLOPs per query, or roughly 230 ms per core, which is too slow. So a graph index it is, at 9.2 GB plus 128 MB of links.
• Block 3: freshness within 60 seconds with a 3 million vector corpus means the whole index rebuilds in well under a minute, so a hot index is unnecessary complexity.

Faded example 2. Prompt: "You are migrating from a 768-dimensional embedding model to a 1024-dimensional one, over 200 million vectors, with no allowed downtime and no allowed quality regression." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: the two vector spaces are not comparable, so a mixed index is invalid. This is a full re-embedding plus a full rebuild, not an update.
• Block 2: the cost is dominated by re-embedding, not by index building. 200 million documents through an encoder is an offline batch job whose duration should be computed from the encoder's throughput before anything else is decided.

7f. Check yourself

Q1. Your graph index has accumulated tombstones for 30 percent of its vectors. Describe the two symptoms you would see and what you would do.

Q2. An interviewer asks why you would not just use exact search. Give the threshold and the arithmetic.

7g. When not to use this

When a dedicated vector service is the over-engineering, even at scale. If your corpus already lives in a relational or search engine that supports vector indexing, adding a second datastore introduces a consistency problem: two systems that must agree about which documents exist. That consistency problem is frequently more expensive than the performance you gained. The measurement that justifies separating them is when the vector workload's memory or query pattern actively harms the primary store's performance, which is observable, not hypothetical.

When approximate retrieval is the wrong tool entirely. If the query is a filter with high selectivity, if results must be exactly reproducible, or if the corpus is small and highly dynamic, an inverted index or a plain database query with exact scoring beats vector search on every axis. Module 6's filtering table gives the selectivity threshold: below roughly 1 percent pass rate on a high-cardinality attribute, stop reaching for the embedding.

Module 8. Model choice as a design decision

Time: 35 to 45 minutes reading, 25 to 35 minutes practice.

Model architecture deserves about five minutes of a 45 minute round, and this module is what those five minutes should contain. The claim being defended: at production scale the model family is chosen by the serving pattern, the data shape and the cost per item, and almost never by an accuracy argument.

8a. Predict first

Three ranking or retrieval problems, described only by their data and their constraints.

Gradient boosted trees against neural networks

The honest default: for tabular data under roughly 10 million rows, boosted trees are the baseline that must be beaten, and they frequently are not beaten. Saying that in an interview is a strength, not a hedge, provided you also name the conditions under which you would switch.

Where the switch actually comes from, with arithmetic. Representing a 200 million value item identifier:

one-hot in a tree: 2e8 candidate splits evaluated per node.
                   Not a tuning problem, an impossibility.

embedding, dim 32: 2e8 * 32 * 4 bytes = 25.6 GB of table

frequency cutoff, keep the top 5 million identifiers,
assume they cover 95 percent of impressions:
                   5e6 * 32 * 4 bytes = 640 MB

Figure the identifier problem, and the cutoff that makes it tractable

That 640 MB number is the interesting one. It says the network's advantage is affordable, and it says exactly what you gave up: the 195 million rare items now share a hashed or out-of-vocabulary embedding, so the model is deliberately worse on the long tail. Whether that is acceptable is a product question, and the guardrail from module 2 on new-item impression share is the thing that would detect it going wrong.

Two-tower retrieval, and why its shape is forced

The item tower's output must be computable offline, because it goes into the index built in module 7. That single requirement forces the architecture:

+--------------+                +---------------+
| user tower   |                | item tower    |
| online, 5 ms |                | offline batch |
+--------------+                +---------------+
        |                               |
        v                               v
   user vector                     item vectors
    256 dims                    100M x 256 dims
        |                               |
        +-------------> dot <-----------+
                   product / ANN

Figure the towers never meet before the dot product, by construction

What this costs you, stated plainly: no feature can depend on both the user and the item until after retrieval. "How many of this creator's videos has this user watched" is exactly the feature you want and exactly the feature this architecture forbids. That is why retrieval is followed by ranking: the ranker is where cross features live, and the funnel exists partly to make cross features affordable.

Three specifics that separate a designed answer from a recited one.

The popularity correction is the part worth being precise about, because without it the sampling scheme quietly encodes a popularity prior into the model. The mechanism is simple enough to state: if an item appears as a negative with probability proportional to its frequency Q(j), subtracting log Q(j) from its logit removes that bias. The sampled-softmax formulation with this correction is described in Yi and colleagues at RecSys 2019, and naming the source is better than presenting it as folklore.

The cost of keeping item vectors current connects straight back to module 7:

item tower forward pass, assume 5 MFLOPs per item
100e6 items * 5e6 FLOPs = 5e14 FLOPs
at 1e14 usable FLOP/s   = 5,000 seconds = 1.4 accelerator-hours

Figure re-embedding the entire catalogue costs under two GPU hours

That number matters because it says a full daily re-embedding is affordable, which in turn means the nightly index rebuild from module 7 can use fresh vectors rather than months-old ones. If the number had come out at 200 hours, the design would need incremental embedding and a version-mixing problem, and the whole retrieval section would change.

Multi-task and multi-objective ranking

Module 2 produced three heads and a combination rule. There are two ways to build that, and the choice is usually made on cost rather than accuracy.

The serving arithmetic is decisive at scale: 33 GFLOPs against 11.3 is a factor of 2.9, which at module 3's re-ranking cost of $4,560 per day is a difference of about $8,900 per day. Multi-task learning at that scale is frequently a cost decision that gets narrated afterwards as an accuracy decision.

The second real reason is label sparsity. If clicks occur at 4 percent and purchases at 0.1 percent, the purchase head has 40 times less signal. A shared trunk trained mostly by the click task gives the purchase head a representation it could never have learned alone.

The counter-risk is negative transfer, where an unrelated task's gradients degrade the shared representation, and the mitigation is expert gating so each task can select which parts of the trunk it uses. The multi-gate mixture-of-experts formulation from KDD 2018 is the standard reference to name here.

What the heads do not decide: the combination weights. Those still come from module 2 and are set online. A multi-task model produces three calibrated probabilities; it does not tell you what a completion is worth relative to a report.

Distillation, as a latency and cost decision

Distillation trains a small student to reproduce a large teacher's outputs, including the teacher's probabilities on examples that have no label.

teacher: 11 GFLOPs/item, 95 accelerators, $4,560/day (module 3)
student:  0.35 GFLOPs/item, 31x cheaper
          95 / 31 = 3 accelerators, about $150/day

Figure distillation converts a 31x compute reduction into $4,400/day

The trade is quality, and the honest way to present it is as a rate: if the student gives up 0.3 points of AUC and the cost falls by 97 percent, whether that is right depends on how much 0.3 points of AUC is worth in your product, which is measurable from past experiments. That framing, cost per unit of metric, is what a senior answer sounds like.

Two things distillation buys that are easy to miss. The student can be trained on unlabelled traffic, because the teacher supplies targets, which is useful when labels are the scarce resource. And the student inherits the teacher's calibration behaviour if trained on soft targets, which a separately trained small model would not.

Where it fails: when the teacher is only better because of features the student cannot afford to fetch. Distillation compresses computation, not information. If the teacher's advantage came from a 200 millisecond feature lookup, no student will reproduce it.

8d. Worked example

Prompt. "Pick the models for the whole short-video feed stack and justify each one."

Block 1. PURPOSE: choose the retrieval model from the index's constraint, because that constraint is not negotiable.

Module 7's index stores one vector per item and searches by dot product, so the item representation cannot depend on the user. That eliminates every architecture with early user-item interaction, regardless of its accuracy, and it does so before any experiment is run.

So: a two-tower model, 256 dimensions to match the index sizing, trained with sampled softmax over in-batch negatives mixed with uniform catalogue negatives, with the logQ popularity correction. The user tower runs online inside the 25 millisecond retrieval budget; the item tower runs nightly at a cost of about 1.4 accelerator-hours for the full catalogue.

I add one thing that is not a model decision but decides the design: retrieval is not a single tower. I run the embedding source alongside a follows-and-recency source and a trending source, because module 6's degradation ladder needs independent sources to degrade gracefully, and because a single learned retriever has no diversity floor.

The error most readers make here is designing retrieval for accuracy and only afterwards asking whether it can be indexed. The index requirement comes first and removes most of the option space for free.

Block 2. PURPOSE: choose the ranking model from data shape and per-item cost, since those two are what actually differ between families here.

The per-item budget from module 6 is 90 microseconds and the corpus is 20 billion impressions with a 200 million value item identifier and a 50-item history sequence. Data shape says a network; the budget says the network must be small.

I choose a moderate network: embeddings for the identifiers with a frequency cutoff at the top 5 million items (640 MB of table, covering an assumed 95 percent of impressions), a short sequence encoder over the last 50 interactions, and a dense trunk of a few hidden layers. I size it to roughly 300 MFLOPs per item, which is 3 microseconds at 100 TFLOP/s and comfortably inside the budget.

Then I do the comparison that earns credit rather than asserting the choice: I would ship boosted trees first. They train in under an hour, they need no embedding infrastructure, and they establish the baseline. The network has to beat that baseline on an online experiment before any of its infrastructure is justified.

The error most readers make here is proposing the sophisticated model as the starting point. The sophisticated model is the second model, and the first one is the thing that tells you whether the second is worth building at all.

Block 3. PURPOSE: decide multi-task against separate models on cost and label sparsity, not on elegance.

Module 2 requires three heads: p(watch past 10 s) at roughly 30 percent positive rate, p(completion) at roughly 12 percent, and p(negative reaction) at roughly 0.3 percent.

The negative-reaction head is 100 times sparser than the first, which is the label-sparsity argument for sharing a trunk. The serving argument points the same way: three separate networks would triple the ranking cost, taking the ranking stage from a rounding error to a real line item.

So: one shared trunk, three heads, with gated experts if and only if a measured regression appears. I state the fallback condition explicitly, because "we will add expert gating" without a trigger is decoration: if adding the negative-reaction head degrades the watch-past-10s head by more than its experiment noise band, that is negative transfer, and gating is the response.

The error most readers make here is treating multi-task learning as free accuracy. It couples three release cadences, three regression surfaces and three calibration curves into one artefact, and that coupling is a real operational cost that has to be paid for by a real benefit.

Block 4. PURPOSE: decide whether the heavy re-ranker earns its place, and state what happens if it does not.

Module 3 costed the top-25 cross-encoder at 95 accelerators and $4,560 per day, about 5.5 times the rest of the online path. That is the number, and the number does not by itself decide anything.

So I frame it as an experiment with a stated decision rule, which is the correct answer to a cost question you cannot resolve from the whiteboard: run the cross-encoder on 1 percent of traffic, measure the primary metric, and convert the observed lift into revenue using the product's existing revenue per session. Launch if the lift pays for $4,560 per day with margin; otherwise distil the cross-encoder into a 0.35 GFLOP student at about $150 per day and re-measure.

The distillation branch is the interesting one, because it usually keeps most of the gain at 3 percent of the cost, and because it changes the design: a cheap student can score all 500 candidates rather than 25, which may recover more quality than the teacher lost.

The error most readers make here is adding the heavy re-ranker to the diagram because sophisticated systems have one. Every stage must be paid for by a measured lift, and being willing to delete a stage is scored.

Result. Two-tower retrieval forced by the index, boosted trees as the shipped baseline for ranking, a moderate embedding network as the challenger with a 640 MB table from a stated frequency cutoff, one shared trunk with three heads justified by a 100-fold sparsity difference and a 2.9-fold serving saving, and a re-ranking stage that only exists if an experiment pays for it.

8e. Fade the scaffold

Faded example 1. Prompt: "Choose the model for a delivery time estimate. 2 million completed deliveries per month, 80 features, all tabular, served in 30 ms, and the estimate is shown to customers so it must not be wildly wrong on any segment." Blocks 1 to 3 are given. Produce block 4 yourself.

• Block 1: no retrieval stage exists. There is one prediction per order, so the funnel does not apply and the whole model choice is a single decision.
• Block 2: 2 million rows per month, 80 tabular features. That is squarely in tree territory: a network would need feature engineering effort to match and gains nothing from an embedding.
• Block 3: the target is a duration, so this is regression. Because the estimate is shown to customers, the loss should not be plain squared error: being 10 minutes late is far worse than being 10 minutes early, which is an asymmetric cost.

Faded example 2. Prompt: "You have a ranking model at 11 GFLOPs per item that you cannot afford. Design the plan to cut the cost by 10 times without losing more than 0.5 points of AUC." Blocks 1 and 2 are given. Produce blocks 3 and 4.

• Block 1: enumerate the levers rather than picking one. Fewer candidates scored, a smaller model, cheaper arithmetic, caching, and moving work offline are five independent axes.
• Block 2: order them by cost per engineer-week. Reducing candidates scored is a configuration change; quantizing to lower precision is days; distillation is weeks; a new architecture is months.

8f. Check yourself

Q1. An interviewer asks why you would not use a transformer for a 300,000 row tabular fraud problem. Answer in three sentences.

Q2. Your two-tower retrieval model returns almost exclusively popular items. Give the most likely cause and the fix.

8g. When not to use this

The general principle, stated once. Every model choice above is downstream of a constraint that already exists in the system: the index shape, the per-item budget, the row count, the feature shape, the label density. If you can name the constraint, the model choice is nearly automatic and takes a minute. If you cannot name the constraint, no amount of architecture discussion will produce a defensible answer, and the minutes are better spent on modules 2, 4 and 5.

Module 9. Calibration and bias

Time: 40 to 50 minutes reading, 30 to 40 minutes practice.

This module is about the gap between a score that sorts well and a score that means something. Almost every ML design round eventually walks into that gap, and almost every recited pipeline answer walks straight past it.

9a. Predict first

Read this artefact and commit to an answer before continuing.

+---------------------------------------------------------------+
| Ad ranker. Slot is sold if eCPM passes a floor.               |
|   eCPM = bid_per_click x pCTR x 1000                          |
|   floor = 10.00 per 1000 impressions                          |
| Offline, held out on the next day of traffic:                 |
|   AUC 0.812     log loss 0.089                                |
| Predicted vs observed click rate by predicted decile:         |
|   decile 1    pred 0.004    obs 0.002                         |
|   decile 5    pred 0.031    obs 0.016                         |
|   decile 10   pred 0.142    obs 0.070                         |
| The ad in question bids 0.50 per click and lands in decile 5. |
+---------------------------------------------------------------+
Caption: an ad ranker's discrimination and its reliability table,
plus the auction rule that consumes the score.

Question. The model sorts ads well. Name the decision it still gets wrong, and compute what that decision costs per million impressions.

Two properties, one number. A score has a discrimination property (does a higher score mean a more likely positive) and a calibration property (does the number equal the rate). They are independent, and different consumers of the score need different ones.

The one-line rule. Calibration matters exactly when the score is combined with a number the model did not produce. If you can state that sentence in an interview and then apply it, you are already above the recited-pipeline answer.

Defining calibration so it can be measured. A model is calibrated on a set if, for every band of predictions near p, the observed positive rate is p.

Use the ratio for monitoring, the reliability table for debugging, and never report a single global expected calibration error as evidence that an auction is safe.

A defensible tolerance band. Treat a calibration ratio inside 0.95 to 1.05 as healthy. That is an operating convention, not a law, and here is the reason it is reasonable: bid increments in most auction interfaces are coarser than 5 percent, so an error smaller than that rarely flips which ad wins. Set your own band from the smallest decision your system can actually take.

The four things that break calibration, and their inverses.

The downsampling correction, derived. Keep every positive and keep each negative with probability w. Sampling does not change the positives, so the odds in the sampled set are the true odds divided by w.

• True odds: o = p / (1 - p)
• Sampled odds: o' = o / w, so the model learns p' = o / (w + o)
• To invert: o' = p' / (1 - p'), then o = w x o', then p = o / (1 + o)

Worked once, with w = 0.05 and a model output of 0.28: o' = 0.28 / 0.72 = 0.3889, o = 0.05 x 0.3889 = 0.019444, p = 0.019444 / 1.019444 = 0.01907. The model said 28 percent. The truth is 1.9 percent. Facebook engineers published this correction with their ad ranking system in 2014 (Practical Lessons from Predicting Clicks on Ads at Facebook, ADKDD 2014, link checked 1 August 2026).

Post-hoc calibration methods, compared honestly.

Every one of these is a monotone map, so none of them can change AUC by a single point. That is the sentence that settles the argument with whoever asks why you bothered. Guo and colleagues documented the shrink-the-logits failure in modern networks and the one-parameter fix in 2017 (On Calibration of Modern Neural Networks, accessed August 2026).

Position bias, stated as a model. Under the examination hypothesis, a click happens when the user examines the slot and the item is relevant:

P(click at slot k) = P(examine slot k) x P(relevant)

The first factor belongs to your old ranker's layout. The second is what you want to learn. Training on raw clicks teaches the new model to reproduce the old model's slot assignment.

How badly, with numbers. Take these examination propensities, measured on a randomised slice (they are an assumption here, and the section below says how to measure your own):

The genuinely best item, at relevance 0.20, produces the lowest click rate in the log. Fitted naively, your model learns that item is the worst of the three. That is not a subtle statistical concern, it is an inverted label.

Inverse propensity weighting, and what it costs. Weight each logged event by 1 / propensity. The estimator becomes unbiased for the relevance term, and the price is variance. Measure the price with effective sample size:

ESS = (sum of weights)^2 / (sum of squared weights)

Worked: one million clicks, half at slot 1 (weight 1) and half at slot 10 (weight 10, propensity 0.10).

• Sum of weights = 500,000 x 1 + 500,000 x 10 = 5,500,000
• Sum of squares = 500,000 x 1 + 500,000 x 100 = 50,500,000
• ESS = 5.5e6 squared / 5.05e7 = 3.025e13 / 5.05e7 = 599,000

You paid 40 percent of your data for unbiasedness. Now clip weights at 5:

• Sum of weights = 500,000 + 2,500,000 = 3,000,000
• Sum of squares = 500,000 + 12,500,000 = 13,000,000
• ESS = 9e12 / 1.3e7 = 692,000

Clipping bought back 93,000 effective rows and reintroduced a bounded bias against deep slots. That is the whole trade, and being able to compute both sides of it in an interview is a senior signal. Joachims and colleagues gave the unbiased learning-to-rank formulation in 2017 (Unbiased Learning-to-Rank with Biased Feedback, accessed August 2026).

Three ways to get propensities, priced.

Agarwal and colleagues published the intervention-harvesting approach in 2019 (Estimating Position Bias without Intrusive Interventions, accessed August 2026).

The operational instruction that matters more than any of this. Log the propensity at serve time, in the same record as the impression. It is the probability your serving policy assigned to the action it took. You cannot reconstruct it later, because the model has been retrained, the features have moved, and the code has changed. Teams that skip this line lose the ability to do off-policy evaluation forever, and they usually discover it during the quarter they most need it.

Other named biases in the same log.

Delayed feedback, stated as arithmetic. Let F(W) be the fraction of conversions that arrive within window W of the click. If you label with a window W, the observed rate is the true rate times F(W).

• Measured on matured cohorts: F(1 day) = 0.62, F(7 days) = 0.88, F(30) = 1.00
• A model trained on 1-day labels predicts 0.62 of the true 30-day rate
• Multiply by 1 / 0.62 = 1.613 to recover the level

The trap is that F is not one curve. App installs mature in hours, insurance quotes in weeks. A single global uplift over-corrects the fast segment and under-corrects the slow one, which is worse than no correction for the segments that matter.

Four ways to handle it, with the architectural consequence.

Chapelle published the joint conversion-and-delay formulation in 2014 (Modeling Delayed Feedback in Display Advertising, KDD 2014, accessed August 2026).

+-----------------+     +------------------+     +-----------------+
| RANKER v1       | ==> | IMPRESSION LOG   | ==> | LABEL STREAM    |
| chooses slot k  |     | slot k, p_serve  |     | click now,      |
| writes p_serve  |     | features frozen  |     | convert later   |
+-----------------+     +------------------+     +--------+--------+
                                                          |
   +------------------------------------------------------+
   |
   v
+---------------------------------------------------------------+
| TRAINING SET                                                   |
|   weight = 1 / propensity(slot)     undoes position bias       |
|   uplift = 1 / F(W) per segment     undoes delay bias          |
|   odds   = w x odds_model           undoes downsampling        |
+-------------------------------+-------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| RANKER v2 > CALIBRATION LAYER > eCPM = bid x pCTR x 1000       |
+---------------------------------------------------------------+
Caption: three distinct distortions enter between the serving
ranker and the training set, each with its own inverse. Fixing one
and skipping the others still leaves the auction mispriced.

9d. Worked example

The prompt. "We predict click probability for an ad auction. Training on the full log is too slow, the old ranker chose the slots, and conversions arrive days later. Make the number trustworthy."

Inputs, stated before I touch anything:

Requests are not impressions, and the gap is a design fact rather than a rounding error. The ranker scores all 2.5 billion requests, but only the 2 billion that clear the floor and the filters become impressions carrying a label. The other 500 million are scored and discarded.

Two consequences follow immediately, and both are propagated below.

• Row counts are impression counts, so they are requests x fill, not requests. The 2e9 in Block B is 2.5e9 x 0.80.
• Per-mille money moves with the same factor. The 2,000 currency units of under-floor inventory per million served impressions in 9a is 0.80 x 2,000, so 1,600 per million ad requests. Quote the wrong denominator and you are 25 percent out.

Block A. Purpose: decide which properties the score must have

I start by asking what consumes the score. It is multiplied by a bid and compared against a floor. That single fact tells me calibration is a hard requirement, not a nicety, and it tells me AUC cannot be my ship criterion.

I say this out loud in the first two minutes, because it reframes everything that follows: I am not building a better sorter, I am building an estimator of a rate that other people will do arithmetic with.

The error most readers make here. Opening with model architecture. The consumer of the score decides the requirements, and the requirements decide whether a gradient boosted tree with a calibration layer beats a larger network without one. Most of the time it does.

Block B. Purpose: make training affordable without lying about the level

The impression log is 2.5e9 requests times the 0.80 fill rate, so 2e9 rows per day. At 2 percent click rate that is 40 M positives and 1.96e9 negatives per day. I keep all positives and keep negatives with probability w = 0.05, giving 98 M negatives and a training set of 138 M rows. That is 14 times smaller, which is the difference between a 90 minute run and a 21 hour run on the same hardware.

The price is a scale distortion in the level, and the price is exactly zero because the inverse is closed form:

+-------------------------------------------------------------+
| p_model  = model output on the sampled distribution         |
| o_sample = p_model / (1 - p_model)                          |
| o_true   = w x o_sample                     w = 0.05        |
| p_true   = o_true / (1 + o_true)                            |
+-------------------------------------------------------------+
Caption: the four-line inverse of negative downsampling, applied
at serve time before any auction arithmetic.

I apply this at serving time, not in training, so that the model file stays independent of the sampling rate I happened to use. When someone changes w next quarter they change one constant in the serving config.

The error most readers make here. Sampling negatives and then reporting log loss on the sampled set as if it measured production quality. It does not. Log loss on a set with an artificial base rate is not comparable to anything.

Block C. Purpose: remove the bias the old ranker wrote into the logs

Every training row carries the slot it was shown in, so I weight by the inverse propensity: slot 1 weight 1.00, slot 2 weight 1.82, slot 3 weight 2.63.

I compute what this costs before deciding to do it. Suppose slots are used evenly, so one third of rows at each weight.

• Sum of weights per 3 rows = 1.00 + 1.82 + 2.63 = 5.45
• Sum of squares per 3 rows = 1.00 + 3.31 + 6.92 = 11.23
• ESS per 3 rows = 5.45 squared / 11.23 = 29.70 / 11.23 = 2.64

So 3 rows are worth 2.64, a 12 percent loss of effective data. With 138 M rows that is still 121 M effective rows, which is plenty. I do not clip, because the maximum weight is 2.63 and clipping only matters when weights reach the tens.

That last sentence is the point of the whole block. With three slots, position bias correction is nearly free. With twenty slots and a propensity of 0.05 at the bottom, the same correction would cost most of my data and I would have to clip, or restrict training to the top few slots, or run a small randomised slice to get cleaner estimates.

The fill rate hides a second selection that propensity weighting cannot touch. Slot weights repair which of three positions an ad landed in. They do nothing about the 500 million requests per day that produced no impression at all, because a request with no served ad has no outcome to reweight.

So the honest claim is narrower than it looks: this model is calibrated on filled inventory. If you want to know what the unfilled 20 percent would have done, the only source is a small forced-fill slice that serves below the floor on a capped budget and logs the result.

The error most readers make here. Applying inverse propensity weighting with propensities estimated from the very logs the ranker produced, without any randomisation or natural variation. The estimate then encodes the ranker's own behaviour and the correction quietly does nothing.

Block D. Purpose: pay for conversions that have not arrived yet

Clicks are immediate, so pCTR needs no delay treatment. Advertisers billed per acquisition need pCVR, and that label matures over days.

I choose a 1-day label plus a per-segment uplift, and I say why I rejected the alternatives:

The segment table holds F_s(1 day) per (vertical, device), refreshed monthly from cohorts at least 45 days old. Any segment with under 1,000 matured conversions falls back to the parent vertical, because an uplift estimated on 50 conversions has a relative standard error of 1 / sqrt(50) = 14 percent and would inject more error than it removes.

The error most readers make here. Refreshing the delay curve from recent cohorts. A cohort from last week is itself immature, so it reports a smaller F(W) and inflates the uplift, which then over-predicts, which then raises bids. The curve must come from cohorts older than the longest delay you believe in.

Block E. Purpose: pick the monitor that catches this the day it starts

One number, per slice, daily: calibration ratio = sum of predictions divided by sum of realised labels.

Slices: vertical, slot, device, and new-advertiser-under-14-days. That is four dimensions, not the full cross product, because the cross product produces thousands of thin slices and a permanent alert storm.

I set the alert band from the noise, not from taste. For a slice with n impressions and click rate p, the count of positives is about n x p and the relative standard error of a count is 1 / sqrt(n x p).

• n = 10,000 and p = 0.02 gives 200 positives, relative error 1 / sqrt(200) = 7.1 percent
• So a plus or minus 7 percent daily wobble on a 10,000 impression slice is pure noise
• The alert therefore fires on two consecutive days outside 0.93 to 1.07, and only for slices with more than 10,000 impressions

Two consecutive independent breaches at roughly a 5 percent one-day rate is about 0.25 percent per slice, which across 40 monitored slices gives roughly one false page every ten days. That is the number I would defend in a design review.

The error most readers make here. Alerting on the global calibration ratio. It is the average of everything and it is the last number to move. A vertical can be 40 percent miscalibrated for a month while the global number sits at 1.01.

9e. Fade the scaffold

Fade 1: a fraud model whose score multiplies transaction amount. Expected loss = pFraud x amount, and a transaction is held for review when expected loss exceeds 25 currency units. Same five-block shape, last block blank.

• Block A, which properties the score needs: the score multiplies an amount and is compared to a currency threshold, so calibration is required. Ranking alone would let the level drift and silently move the review rate.
• Block B, affordable training: fraud rate is 0.3 percent, so negatives are downsampled at w = 0.02 and the closed-form inverse is applied at serve time.
• Block C, bias from the old policy: transactions the old model blocked have no outcome label at all. This is selection bias, not position bias, and inverse propensity weighting cannot fix a missing outcome. A small random-release slice of high-score transactions is the only source of labels in that region.
• Block D, delayed labels: chargebacks arrive up to 120 days later, so the 30-day label captures a fraction F(30) that must be measured on matured cohorts and inverted per segment.
• Block E, the monitor: you write this block.

Fade 2: a notification ranker. The score decides whether to send, subject to a cap of 3 sends per user per day, and the label is "opened within 24 hours". Last two blocks blank.

• Block A, which properties the score needs: the threshold is fixed and the cap makes this a per-user top-3 selection, so both ranking within a user and calibration across users are needed. Calibration decides how many users get zero sends.
• Block B, affordable training: sends are already sparse, so no downsampling is needed. The scale distortion here comes from class weighting, which someone added to fix a recall complaint. Remove it and let the threshold do that job.
• Block C, bias from the old policy: you write this block.
• Block D, delayed labels: you write this block.

9f. Check yourself

Q1. You add isotonic regression to a ranker and offline AUC does not move by a single point. Your product manager asks why you spent a sprint on it. Answer in three sentences, with one number.

Q2. A colleague fits position propensities using a click model trained on the same logs your production ranker generated, with no randomisation anywhere in the system. Name the circularity and say what visibly breaks first.

9g. When not to use this

A post-hoc calibration layer.

Inverse propensity weighting for position bias.

A joint delayed-feedback model.

A permanent randomised exploration slice.

Module 10. Offline evaluation

Time: 35 to 45 minutes reading, 25 to 35 minutes practice.

Offline evaluation is where most candidates are weakest and most confident. The recited answer names three metrics. The strong answer picks one metric because of what the system does with the score, then spends its time on the split, the slices and the errors.

10a. Predict first

+---------------------------------------------------------------+
| System: retrieve 500 of 8 M items, then rank, then show 10.   |
| Change under test: the RANKER only. Retrieval is untouched.   |
| Offline results, new model versus incumbent:                  |
|   AUC          0.781  ->  0.802     (+2.7 percent)            |
|   PR-AUC       0.061  ->  0.058     (-4.9 percent)            |
|   NDCG at 10   0.312  ->  0.319     (+2.2 percent)            |
|   recall at 100  0.71 ->  0.70      (-1.4 percent)            |
|   log loss     0.214  ->  0.219     (worse)                   |
| Evaluation set: random 10 percent of the last 90 days.        |
+---------------------------------------------------------------+
Caption: five offline numbers for a ranker change in a two-stage
funnel, with the split that produced them.

Question. Which single number should decide whether this ships, and which line of the artefact makes all five numbers untrustworthy?

Metric selection starts from the decision, not the task type. Write down the sentence "the system uses this score to ..." and finish it. The metric follows.

Why AUC lies under imbalance, derived. Take 1,000,000 events with 1,000 positives, a prevalence of 0.1 percent. Suppose the model achieves AUC 0.9, and at the operating point you would actually pick, true positive rate is 0.8 and false positive rate is 0.1.

• True positives = 0.8 x 1,000 = 800
• False positives = 0.1 x 999,000 = 99,900
• Precision = 800 / 100,700 = 0.79 percent

A reviewer would work 100,700 cases to catch 800. To reach 10 percent precision you need false positives under 7,200, so a false positive rate under 0.0072. Whether the model can do that is a question about one point on the curve, and AUC never tells you. Ask for precision at the recall you need, or the partial area over the region you would operate in.

Ranking metrics, defined before they are used.

The structural limit nobody states. All of these are computed on logged impressions, so they measure re-ordering of what the incumbent already retrieved. A new model whose whole value is surfacing items the incumbent never showed will score zero improvement, or worse, on every one of them. That is not a metric bug, it is the reason online evaluation exists.

Splits, and the leakage arithmetic. Split by time for anything with temporal features, by entity for anything with grouped rows, and by both when both apply.

Concrete leakage: a feature "user click rate over the trailing 30 days" computed once over the whole window, on a user with 20 events. The scored event contributes 1 of 20 observations to its own feature, so 5 percent of the feature is the label. For a user with 2 events it is 50 percent. Sparse users are exactly the cohort you care about, and they are exactly where the leak is worst, which is why leaked evaluations flatter new-user performance most.

The evaluation split ladder.

Slice-based evaluation, and the arithmetic that makes it mandatory. Define your slices before the run, five to eight of them, each tied to a named risk.

Suppose overall click rate improves 1.02 percent while new users regress:

• New users are 15 percent of traffic, effect minus 3.4 percent
• Returning users are 85 percent, effect plus 1.8 percent
• Weighted: 0.15 x (-3.4) + 0.85 x (1.8) = -0.51 + 1.53 = +1.02

The aggregate is positive and the cohort that drives growth is down 3.4 percent. This is not an exotic case. It is what happens whenever a model gets better at exploiting history, because the cohort with no history loses.

A default slice set for a ranking system.

Error analysis as a design skill. Sample 100 errors stratified across your slices, read them, and tag each by cause. The output is not a model idea, it is a budget allocation.

Those counts are illustrative, not measured, and you should produce your own. The structural point survives whatever numbers you get: in most first passes, the majority of errors are not model problems. A candidate who says "I would pull 100 errors and count causes" ranks above one who says "I would try a larger model", because the first is a decision procedure and the second is a guess.

How precise is a count of 100? A cause seen 22 times has a standard error of about sqrt(22) = 4.7, so roughly 22 plus or minus 9 at 95 percent confidence. That means 100 samples can separate a 22 percent cause from a 5 percent cause and cannot separate it from a 15 percent cause. Sample 400 if you need that resolution, and do not build a roadmap on a difference of five errors.

What an offline win does not tell you. Say this list out loud in the interview and you will sound like someone who has shipped.

Measure your own offline-to-online agreement. Take your last ten experiments, record the sign of the offline delta and the sign of the online primary metric, and count agreements. If they agree 6 times out of 10, that is not evidence of anything: under a coin flip, 6 or more agreements happens about 38 percent of the time. Agreement of 9 or 10 out of 10 is a signal. Teams that have never done this count are guessing about the value of their whole offline stack.

Off-policy estimation, at interview depth. If you logged the serving propensity (Module 9), you can estimate a new policy's value from old data:

V = (1 / n) x sum over logged events of (pi_new(a) / mu_logged(a)) x reward

Two rules make it honest. First, report effective sample size alongside the estimate; if ESS is under about 10 percent of n, report a direction and refuse to report a number. Second, the estimate is only valid where the logging policy had non-zero probability, so a new policy that promotes items the old one never showed cannot be evaluated this way at all.

+----------------------------------------------------------+
| STEP 1  LEAK CHECK      time split, entity split,         |
|                         point in time features            |
+----------------------------------------------------------+
                          |
                          v
+----------------------------------------------------------+
| STEP 2  MATCHED METRIC  one metric chosen from what the   |
|                         system does with the score        |
+----------------------------------------------------------+
                          |
                          v
+----------------------------------------------------------+
| STEP 3  SLICES          5 to 8 named slices, each tied    |
|                         to a risk, all reported           |
+----------------------------------------------------------+
                          |
                          v
+----------------------------------------------------------+
| STEP 4  ERROR SAMPLE    100 errors, tagged by cause,      |
|                         counts drive the next sprint      |
+----------------------------------------------------------+
                          |
                          v
+----------------------------------------------------------+
| STEP 5  DECISION        ship to online test, or fix, or   |
|                         stop. Offline never ships alone   |
+----------------------------------------------------------+
Caption: the offline evaluation ladder. A step skipped early
invalidates every step after it, so the order is the method.

10d. Worked example

The prompt. "Your team has a new ranker with plus 2.2 percent NDCG at 10. Decide whether it goes to an online test, and defend the decision."

Inputs:

Block A. Purpose: destroy the result before trusting it

I do not start by asking whether 2.2 percent is a lot. I start by trying to break the measurement, because a wrong number is worse than no number.

1. Was the split temporal? If it was random, stop, re-run, and expect the gain to shrink. I have never seen it grow.
2. Were features computed as of the request time, or with any aggregate that spans the label? This is the point-in-time join question from the feature store module, and it is the single most common real bug in this space.
3. Was the eval set built from the incumbent's own impressions? Then the metric measures re-ordering only, and a model that finds new items is penalised.
4. Are the two models scored on identical candidate sets? If the new ranker also changed filtering, NDCG is comparing two different problems.

The error most readers make here. Treating the offline number as the finding and the split as a detail. The split determines whether the number exists.

Block B. Purpose: size the gain against its own noise

A delta with no interval is a rumour. I compute a paired estimate: for each query in the eval set, NDCG under the new model minus NDCG under the incumbent, then the mean and standard error of that per-query difference.

With 200,000 eval queries and a per-query difference standard deviation of 0.18 (this is measured, not assumed, and 0.18 is typical only in the sense that per-query NDCG is highly variable):

• Standard error = 0.18 / sqrt(200,000) = 0.18 / 447 = 0.0004
• Observed delta = 0.319 minus 0.312 = 0.007
• Ratio = 0.007 / 0.0004 = 17.4 standard errors

So the offline gain is real as a measurement. That says nothing about whether it is real as a product effect, and I say both halves of that sentence out loud, because candidates who say only the first sound naive and candidates who say only the second sound obstructive.

The error most readers make here. Reporting the aggregate delta with no variance at all, then being unable to answer "is that within noise?".

Block C. Purpose: find out who paid for the average

I break the delta by the six slices above. Suppose the table comes back:

Now the decision has changed shape. The aggregate is a growth-negative trade: better for users with history, worse for users without, and it concentrates impressions on popular items. That last row is a feedback-loop risk, not a metric quibble, because next month's training data will contain even fewer tail impressions.

The error most readers make here. Reporting slices and then shipping anyway because the aggregate is positive. The slice table is only useful if it can change the decision.

Block D. Purpose: read 100 errors before proposing a fix

I pull 100 new-user queries where the incumbent put a relevant item in the top 10 and the candidate did not. Tagging them:

• 41 have almost no user features, and the candidate leans harder on personalisation features that are null for these users
• 26 are ambiguous queries where both orderings look defensible
• 18 have a stale item-popularity feature computed on a daily batch
• 15 are genuine quality losses

That distribution names the fix precisely: the candidate degrades badly under null features. The cheap repair is not a new architecture, it is a null-handling change plus a cold-start path, and I can estimate its value at up to 41 percent of the regression before writing any code.

The error most readers make here. Jumping from "new users regressed" to "add a new-user model". Two thirds of this regression is null handling and stale features, both of which are cheaper and land sooner.

Block E. Purpose: state the decision and the condition to revisit it

My decision: do not take the experiment slot yet. Fix null handling, re-run the slice table, and require the new-user slice to be no worse than minus 0.5 percent before the online test.

I write the ship rule now, before the next result exists, because a threshold chosen after seeing the number is not a threshold. I also write what the online test will measure that offline cannot: tail item coverage over time, and the new-user retention effect, neither of which appears in NDCG.

The error most readers make here. Saying "needs more work" without a numeric re-entry condition. A decision with no threshold attached becomes a negotiation next week, and the person with the most patience wins it rather than the person with the evidence.

10e. Fade the scaffold

Fade 1: a moderation classifier. New model shows PR-AUC 0.61 to 0.66 on a held-out month. It runs at a fixed threshold that currently sends 2 percent of content to human review. Last block blank.

• Block A, destroy the result first: check that the eval month is after the training window, that appeals-driven relabels have not been folded back into the test set, and that the test set is not the rebalanced 50-50 set used for training.
• Block B, size the gain: PR-AUC moved, but the decision runs at one threshold, so recompute precision and recall at the threshold that holds review volume at 2 percent, with a bootstrap interval over the positive class.
• Block C, who paid: slice by language, content type and reporter type. A gain concentrated in the majority language while a minority language regresses is a shipping blocker even at a positive aggregate.
• Block D, read the errors: sample 100 false negatives at the operating threshold and tag by policy category.
• Block E, decision and revisit condition: you write this block.

Fade 2: an ETA model. New model shows RMSE 4.8 minutes to 4.4 minutes on 30 days of held-out trips. The number shown to users is a promise, and being late generates support contacts. Last two blocks blank.

• Block A, destroy the result first: split by time and by city, confirm no trip appears in both sides through a shared driver-level feature, and confirm the features are as of dispatch time rather than as of trip completion.
• Block B, size the gain: you write this block.
• Block C, who paid: you write this block.

10f. Check yourself

Q1. A teammate reports AUC 0.94 on a fraud model with a 0.2 percent base rate and proposes shipping. What two numbers do you ask for, and what will you do with them?

Q2. Your offline metric and your online metric have agreed in sign on 6 of your last 10 experiments. What does that tell you, and what would you do next?

10g. When not to use this

A full slice matrix on every evaluation.

Off-policy evaluation with importance weights.

Formal error analysis on a sample of 100.

A curated golden evaluation set maintained by hand.

Module 11. Online evaluation

Time: 40 to 50 minutes reading, 30 to 40 minutes practice.

Offline evaluation asks whether the model is better. Online evaluation asks whether the system with the model in it is better, for real users, at a cost you can measure. They are different questions and they disagree often enough that handling the disagreement well is a senior-level signal by itself.

11a. Predict first

+---------------------------------------------------------------+
| Experiment readout, 14 days, 1 percent of users per arm.      |
| Randomisation unit: user. Analysis unit: impression.          |
| Primary metric declared before launch: none                   |
| Metrics reported: 14                                          |
|   click rate         +0.9 percent   p = 0.04                  |
|   sessions per user  +0.2 percent   p = 0.61                  |
|   revenue per user   -0.4 percent   p = 0.33                  |
|   ... 11 more, all p above 0.10                               |
| Assignment counts: control 501,900   treatment 499,100        |
+---------------------------------------------------------------+
Caption: a typical experiment readout with no declared primary
metric, a unit mismatch, and its assignment counts.

Question. Compute the probability that at least one of 14 metrics with no true effect shows p below 0.05, and name the second thing in this readout that would invalidate it even if only one metric had been reported.

Pick the randomisation unit from where interference lives. Interference means one unit's treatment affects another unit's outcome. Randomise at or above the level where it happens.

Randomising a marketplace change by user is the classic error: treatment users take the supply that control users then cannot have, so the measured effect contains a transfer as well as a creation.

Power, derived once so you can redo it at the whiteboard. For a two-arm test with equal allocation, 80 percent power and a two-sided 5 percent level, the required sample per arm is:

n = 2 x (1.96 + 0.84)^2 x variance / effect^2, which is about 16 x variance / effect^2

For a binary metric at a 2 percent base rate, variance is p(1 - p) = 0.0196.

Read the table backwards, which is how it is actually used. If the team can only run for a week at this traffic, the smallest honest question is "is this better by at least 2 percent?". Running a 1 percent question for a week and reporting "no significant difference" is a statement about your sample size, not about the feature.

The correction most readouts skip. That table assumes the metric's variance is p(1 - p), which holds for a single Bernoulli draw. Your metric is usually a per-user rate, whose variance across users is larger because users differ. In practice, multiply the required sample by the design effect measured from an A/A test on your own traffic. Do not assume the factor; measure it once and reuse it, and re-measure when the product changes.

Variance reduction buys time, and the arithmetic is exact. CUPED adjusts each user's metric using their pre-period value:

Y_adjusted = Y - theta x (X_pre - mean of X_pre), with theta = covariance(Y, X_pre) / variance(X_pre)

The adjusted variance is (1 - correlation squared) times the original. Sample size scales with variance, so:

Deng and colleagues published this method in 2013 (Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM 2013, accessed August 2026). The catch is structural: new users have no pre-period, so the technique helps least in the cohort where you most often need power.

One primary, a few guardrails, everything else is a diagnostic.

A guardrail needs a margin, not just a direction. "Latency p95 must not increase by more than 5 ms" is testable. "Latency must not regress" is not, because noise guarantees some regression somewhere.

Sample ratio mismatch is the cheapest invalidity check you own. Compare observed assignment counts to intended, with a chi-square test.

Worked, from the artefact: total 1,001,000 with 501,900 and 499,100 gives expected 500,500 each. Deviation 1,400 each, so chi-square = 2 x (1,400^2 / 500,500) = 2 x 3.915 = 7.83, and for one degree of freedom that is about p = 0.005. That is a red flag, not a rounding artefact.

Common causes are assignment-time filtering that differs by arm, a treatment that crashes for some users before logging, and bot traffic landing unevenly. Fabijan and colleagues catalogued the causes in 2019 (Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD 2019, accessed August 2026). A failed ratio check voids the readout. Do not analyse further, find the bug.

Interleaving, and its four hard limits. Interleaving merges two rankers' results into one list and attributes each click to whichever ranker contributed the item. Because each user sees both rankers, between-user variance cancels and the comparison needs far less traffic than a split test.

Chapelle and colleagues published a large-scale validation of interleaved evaluation in 2012 (Large-Scale Validation and Analysis of Interleaved Search Evaluation, TOIS 2012, accessed August 2026). Their sensitivity results are theirs, measured on their traffic; run your own comparison before quoting a factor.

Switchbacks, for systems where users are not independent. Randomise time-region cells to treatment or control, then alternate.

Design rules, each with its reason:

The arithmetic that surprises people: 5 regions, 14 days, 4-hour periods gives 5 x 14 x 6 = 420 periods, so 210 per arm. If period-level metric variation has a coefficient of variation of 8 percent, the minimum detectable relative effect is about 2.8 x 0.08 x sqrt(2 / 210) = 2.8 x 0.08 x 0.0976 = 2.2 percent. Two million trips, and you can only see a 2.2 percent effect, because your sample is 420 periods.

Bandits, and the honest scope.

The regret argument is weaker than it sounds for two arms. A two-week split test at 50 percent allocation with a true 2 percent effect wastes 2 percent of half the traffic for two weeks. If the decision it informs runs for a year, that is about 0.2 percent of the annual metric. With 200 arms, the same argument reverses completely, because a split test never reaches power and the bandit is the only mechanism that converges.

When online disagrees with offline, work this table top to bottom.

Novelty, quantified. Suppose the daily treatment effect runs plus 4.1, plus 2.6, plus 1.8, plus 1.4, plus 1.2, plus 1.1, plus 0.9 percent over seven days. The asymptote near 1 percent is the number to plan with, not the 2.2 percent two-week average. If your readout averages a decaying series and reports the mean, you will systematically over-forecast every launch.

+---------------------------------------------------------------+
| Does one user's treatment change another user's outcome?      |
+---------------------------------------------------------------+
       | no                                    | yes
       v                                       v
+---------------------------+   +-------------------------------+
| Do you need a business    |   | Is the shared resource        |
| effect size in currency   |   | bounded by geography or time? |
| or retention?             |   +-------------------------------+
+---------------------------+        | yes            | no
   | yes         | no                v                v
   v             v          +----------------+  +----------------+
+---------+  +-----------+  | SWITCHBACK     |  | CLUSTER TEST   |
| A/B     |  | INTERLEAVE|  | unit is a      |  | unit is a      |
| test    |  | ranker vs |  | region-time    |  | network        |
| by user |  | ranker    |  | cell           |  | component      |
+---------+  +-----------+  +----------------+  +----------------+
Caption: choosing the online method from the interference question
first and the metric question second. Arm count and reward delay
then decide whether a bandit replaces the chosen design.

11d. Worked example

The prompt. "The ranker from the offline module passed its gate. Design the online test, then tell me what you do if click rate is up and revenue is flat."

Inputs:

Block A. Purpose: choose the unit and the primary metric before anything else

Interference: this is feed ranking with no shared inventory, so one user's ranking does not consume another's. Unit is the user.

Primary metric: I do not pick click rate. Clicks are the model's objective, so a model trained to maximise clicks will raise clicks almost by construction, and the interesting question is whether anything else moves. I declare revenue per user as primary and click rate as a diagnostic, with two guardrails: p95 latency must not rise by more than 15 ms, and 7-day return rate must not fall by more than 0.3 percentage points.

I write this in the experiment document before launch and I say so, because a primary metric chosen after the readout is not a primary metric.

The error most readers make here. Choosing the metric the model optimises as the primary metric of the experiment. That converts the test into a check that the training loop works.

Block B. Purpose: size it, then negotiate the question to fit the traffic

Revenue per user has higher variance than a click indicator, so I use a measured coefficient of variation from history rather than p(1 - p). Assume it is 2.2, which I would take from the last quarter of data.

For a relative effect d on a metric with coefficient of variation c:

n per arm = 16 x c^2 / d^2

• Detect 2 percent: n = 16 x 4.84 / 0.0004 = 193,600 users per arm
• Detect 1 percent: n = 16 x 4.84 / 0.0001 = 774,400 users per arm

With 5 M daily users, a 50-50 split reaches 774,400 users per arm in under a day of exposure, but users must accumulate their full in-period metric, so the binding constraint is the 14-day window, not the user count. That is the usual situation for revenue-style metrics and it is worth saying plainly: for per-user metrics, duration is driven by the metric's accumulation window, and extra traffic beyond a point buys nothing.

Applying CUPED at correlation 0.65 removes 1 - 0.65^2 = 58 percent of the variance, so the same question needs 42 percent of the sample. I use it to add guardrail precision rather than to shorten the test, because the 14 days are set by the metric window.

The error most readers make here. Computing power on the metric they can compute and then reporting on a different metric. Power belongs to the primary metric only.

Block C. Purpose: build the invalidity checks into the launch

Before I look at any result, three checks run:

1. Sample ratio: chi-square on assignment counts, with the readout blocked if p is below 0.001.
2. A/A period: the first day runs both arms with the same model, and any metric showing a significant difference is flagged as having an instrumentation problem.
3. Guardrail telemetry: exposure logging fires at assignment time, not after the model returns, because logging after a treatment-only code path is one of the commonest causes of ratio mismatch.

The error most readers make here. Running the ratio check after the analysis and treating it as a footnote. It is a gate, not a caveat.

Block D. Purpose: read the disagreement without guessing

Result: click rate plus 3.1 percent with a tight interval, revenue per user minus 0.2 percent with an interval spanning zero. I work the table:

The finding is now precise: the model increased engagement with cheaper inventory and moved no revenue, and the test could not have detected a 0.9 percent revenue loss. That is a very different sentence from "click rate is up, ship it".

The error most readers make here. Reading a non-significant guardrail as "no harm". A non-significant result on an underpowered metric means you did not look hard enough.

Block E. Purpose: decide, and say what would change the decision

Decision: do not ship as is. Two paths, priced:

I also name the number that would flip me: if a re-run with 4 weeks and CUPED shows revenue per user non-inferior within a 0.3 percent margin while clicks hold plus 2 percent, I ship, because the engagement gain then comes without a revenue transfer.

The error most readers make here. Shipping anyway because the primary metric was not significantly negative. Non-inferiority is a claim that needs a stated margin and the power to support it, and "not significantly worse" on an underpowered metric is not that claim.

11e. Fade the scaffold

Fade 1: a dispatch change in a delivery marketplace. The change assigns couriers in batches instead of one at a time. Last block blank.

• Block A, unit and primary metric: couriers are a shared bounded resource, so user-level randomisation would let treatment orders take couriers from control orders. Use a switchback on region and time. Primary metric is orders delivered per courier hour. Guardrails are late-delivery rate and courier earnings per hour.
• Block B, sizing: 6 regions, 21 days, 3-hour periods gives 6 x 21 x 8 = 1,008 periods, 504 per arm. At a period-level coefficient of variation of 10 percent, the minimum detectable effect is 2.8 x 0.10 x sqrt(2 / 504) = 1.8 percent.
• Block C, invalidity checks: verify that period assignment is balanced across hour of day, that the burn-in exclusion is applied symmetrically, and that region boundaries do not leak couriers.
• Block D, reading the result: you write this block.

Fade 2: a two-model comparison for search relevance with no revenue effect expected. Last two blocks blank.

• Block A, unit and primary metric: no interference between users, and the question is purely which ranker users prefer. Interleaving on the query is available and needs far less traffic than a user-split test.
• Block B, sizing: you write this block.
• Block C, invalidity checks: you write this block.

11f. Check yourself

Q1. A team reports "no significant difference" on a guardrail after a one-week test and ships. What do you ask for, and how do you phrase the risk in one sentence a product manager will act on?

Q2. Your experiment shows a 3 percent lift that decays from 5.2 percent on day one to 2.4 percent on day fourteen. Two colleagues disagree: one says novelty, the other says the treatment is degrading. How do you tell them apart with data you already have?

11g. When not to use this

A full A/B test with a declared primary metric.

Interleaving.

Switchback experiments.

Bandits in place of an experiment.

Module 12. Serving and deployment

Time: 40 to 50 minutes reading, 30 to 40 minutes practice.

This is the module where an ML design becomes a system. Everything before it decides what to compute. This decides where it runs, what it costs, how fast it answers, and how you take it back out when it is wrong.

12a. Predict first

+---------------------------------------------------------------+
| SLO: end to end p99 of 250 ms. Measured p99 by stage:         |
|   feature fetch, parallel to 8 stores          90 ms          |
|   retrieval, parallel to 12 index shards       60 ms          |
|   ranking, one model over 600 candidates       45 ms          |
|   reranking and policy filter                  20 ms          |
|   gateway, serialisation, network              25 ms          |
|   sum of the stage p99 values                 240 ms          |
| Measured end to end p99                       310 ms          |
+---------------------------------------------------------------+
Caption: per-stage and end-to-end tail latency for a ranking
service, with the stage values summing to less than the whole.

Question. The stages sum to 240 ms and the whole is 310 ms. Explain the gap, and say which stage you attack first.

Three serving modes, and the number that chooses between them.

Compute all three volumes before choosing. Take 100 M users, 8 sessions per user per day, one ranked request per session:

• Batch, top 500 per user, once a day: 100 M inferences per day, 1,157 per second averaged, no tail requirement
• Online: 800 M inferences per day, 9,259 per second average, roughly 27,800 per second at a 3x peak, with a hard tail requirement
• Near-line, recomputing only when a user takes a meaningful action, assume 5 percent of events: 40 M per day

Online costs 8 times the inference volume of daily precompute and adds a tail SLO. It buys freshness. If the answer would be identical for 24 hours, you are paying 8 times for nothing.

The hybrid is usually the right answer and is the one candidates skip. Precompute the candidate list nightly, then rerank the top 200 online with features that changed since. Cost is the batch cost plus a 200-candidate online pass, and freshness is request-time for anything that matters. When an interviewer asks for fresher results without doubling cost, this is the shape of the answer.

Latency budget, built from the SLO downward. Write the budget before the architecture, then let the budget reject boxes.

The reserve is not padding. A budget with no reserve is a budget that is exceeded on the first bad Tuesday.

Fan-out multiplies the tail, and the arithmetic is exact. If a request waits on k independent shards and each shard exceeds x with probability q, the request exceeds x with probability 1 minus (1 - q)^k.

To hold a request-level p99 across 12 shards you need (1 - q)^12 = 0.99, so q = 1 minus 0.99^(1/12) = 0.00084. Each shard must meet the target at its 99.92^nd percentile. That is the single most useful number in this module, because it explains why fan-out systems feel unfixable: you are tuning a percentile two orders of magnitude deeper than the one you report.

Three responses, in cost order:

Dean and Barroso set out the mechanisms and the hedging technique in 2013 (The Tail at Scale, CACM 2013, accessed August 2026).

Where the cost actually is, derived. Assume a ranker costing 2 MFLOP per candidate, 600 candidates, so 1.2 GFLOP per request. Assume an accelerator delivering 100 TFLOPS peak and, at online batch sizes with a latency constraint, 10 percent of that in practice. That utilisation assumption is the one to challenge first, and it is low because online serving cannot batch the way training can.

• Effective throughput: 1e13 / 1.2e9 = 8,333 requests per second per device
• At a 27,800 per second peak: 4 devices, call it 8 with headroom
• At an assumed 3.00 per device-hour: 24.00 per hour, 576 per day
• Per thousand requests: 576 / 800,000 = 0.00072

Now the same request's feature traffic. Assume 600 candidates each carrying 200 float features:

• 600 x 200 x 4 bytes = 480 KB per request
• At 27,800 requests per second: 13.3 GB per second, about 107 Gbps

The model is fractions of a cent per thousand requests. The feature movement is a hundred gigabits per second. This is the reason experienced answers put item features and embeddings in the ranker's own memory, precompute item vectors, and prune features aggressively, while inexperienced answers spend the whole deep dive on model architecture.

The cascade, priced. Run a cheap model over all candidates, then the expensive one over the survivors.

• One stage: 600 x 2 MFLOP = 1,200 MFLOP
• Two stages: 600 x 0.05 MFLOP + 60 x 2 MFLOP = 30 + 120 = 150 MFLOP
• Eight times cheaper

The quality question has one honest measurement: what fraction of the items that the full model would place in the final top 10 survive the cheap stage's top 60. Measure it offline on logged requests. If it is 97 percent, ship the cascade. If it is 80 percent, the cheap stage is not cheap, it is a different product.

Caching, and what is actually cacheable.

Query caching, sized with a distribution you can defend. Assume query frequency follows a Zipf distribution with exponent 1, which you should verify by plotting log frequency against log rank on one day of logs. Then the share of traffic covered by the top k of N distinct queries is the ratio of harmonic numbers, approximately (ln k + 0.577) / (ln N + 0.577).

• N = 10 M distinct queries, k = 100,000
• (11.51 + 0.58) / (16.12 + 0.58) = 12.09 / 16.70 = 0.72

So caching 100,000 query results covers about 72 percent of traffic. Memory: 100 k entries x 500 candidate identifiers x 8 bytes = 400 MB, which fits in one process. The cost is freshness: with a 10 minute time to live, a new item waits up to 10 minutes to appear on a cached query, and you should say that number out loud rather than letting the interviewer find it.

Sharding: replicate for throughput, shard for size.

• An index of 500 M vectors at 768 dimensions in 4-byte floats is 500e6 x 768 x 4 = 1.54 TB
• At 128 GB usable per host: 12 shards
• 12 shards means the fan-out tail arithmetic above, so per-shard p99.92
• Quantising vectors to 1 byte per dimension cuts this to 384 GB and 3 shards, at a recall cost you must measure

Notice that the memory decision and the tail latency decision are the same decision. Compressing the index is a latency intervention, not just a cost one.

Model optimisation, in the order that pays.

The ordering matters more than the list. Teams reach for quantisation and distillation because they are interesting, and then discover the model was 12 percent of the request.

+-------------+   +--------------+   +----------------+
| GATEWAY     |=>>| RETRIEVAL    |=>>| RANKER         |
| 20 ms       |   | 12 shards    |   | 600 cands      |
| auth, parse |   | 40 ms        |   | 60 ms          |
+------+------+   +------+-------+   +--------+-------+
       |                 |                    |
       |                 v                    v
       |          +--------------+   +----------------+
       |          | FEATURE CACHE|   | POLICY LAYER   |
       |          | item vectors |   | filters, caps  |
       |          | in process   |   | 20 ms          |
       |          +--------------+   +--------+-------+
       |                                      |
       v                                      v
+---------------+                    +----------------+
| SHADOW COPY   |                    | RESPONSE       |
| mirrored, no  |                    | 50 ms reserve  |
| user impact   |                    | unspent        |
+---------------+                    +----------------+
Caption: the serving path with the latency budget written inside
each box, plus the shadow copy that receives mirrored traffic and
returns nothing to the user.

12d. Worked example

The prompt. "Our recommender recomputes everything nightly. Users complain results are stale. Make it fresher without doubling the bill, and tell me how you would roll it out."

Inputs:

Block A. Purpose: locate the staleness before repricing anything

Staleness is not one thing. I split it into three and ask which one users actually feel:

The second row is the complaint. That is decisive, because reflecting within-session behaviour does not require recomputing the whole catalogue, and recomputing the whole catalogue does not fix it.

The error most readers make here. Answering "make the batch run hourly". That multiplies the batch bill by up to 24 and still leaves a session-level lag that users experience as the system ignoring them.

Block B. Purpose: choose the hybrid split and price it

The design: keep the nightly batch producing 500 candidates per user, add an online reranking pass over the top 200 using request-time features, and add a near-line path that injects new and trending items into the candidate pool within minutes.

Pricing the online rerank against the numbers above:

• 200 candidates x 2 MFLOP = 400 MFLOP per request
• At 1e13 effective FLOPS per device: 25,000 requests per second per device
• Peak 27,800 needs 2 devices; provision 6 for headroom and failure domains
• 6 devices x 3.00 per hour x 24 = 432 per day

Against a nightly batch of 200 machines for 6 hours, the online layer is a small addition, and the number that proves it is the ratio of the two compute bills, which I would compute from the actual machine prices rather than asserting.

The near-line path is cheaper still: 50,000 new items per day need embedding once each, which is 50,000 inferences, a rounding error against 800 M.

The error most readers make here. Pricing only the model and forgetting the feature movement. 200 candidates x 200 features x 4 bytes = 160 KB per request, and at 27,800 per second that is 4.4 GB per second. That number, not the FLOPs, decides whether item features live in the ranker's memory or behind a network call.

Block C. Purpose: protect the tail while adding a stage

I have added a stage to a path that already meets 250 ms, so I must take the time from somewhere. The budget:

The hybrid is faster than the pure online design would be, because the nightly batch already eliminated the 12-way index fan-out. That is the trade to say out loud: precompute buys tail latency as well as cost, and it pays for both with freshness, which the near-line path then buys back for the subset of items where freshness matters.

The error most readers make here. Adding the online stage on top of the existing budget and declaring the SLO unchanged. Every new box spends time that belonged to something else.

Block D. Purpose: roll it out so that being wrong is cheap

Three stages, each catching a different class of failure:

Canary detection time, derived: at 1 percent of 800 M requests per day, the canary sees 8 M requests per day, which is 333,000 per hour. To detect a 5 percent relative drop in a 2 percent click rate, n = 16 x 0.0196 / (0.001)^2 = 313,600 in the small arm. So roughly one hour of canary gives power to see a 5 percent quality regression, while an error-rate spike is visible in seconds.

That asymmetry sets the schedule: hold each canary step for the slowest signal you intend to gate on, not the fastest one you can see.

The error most readers make here. Treating shadow traffic as a quality test. Shadow has no labels, because nobody saw the output. It tests the machinery, not the recommendation.

Block E. Purpose: write the rollback rules before launch

Rollback is a configuration flip to a model version that is already loaded and warm, not a redeploy. If your rollback path requires a build, your mean time to recovery is your build time, and you should say that number in the interview because it is usually the honest weak point of the design.

The error most readers make here. Writing rollback criteria that all require a human judgement. If every threshold ends in "then investigate", your rollback time is the time it takes to wake someone up, and the fast signals should have been automatic.

12e. Fade the scaffold

Fade 1: a fraud scorer at checkout with a 40 ms budget. Last block blank.

• Block A, locate the real constraint: the budget is 40 ms end to end inside a payment flow, so the constraint is the tail, not the cost. Precompute is impossible because the request itself is the entity being scored.
• Block B, choose the shape and price it: a small model in process, features restricted to those available within the request plus a cached account aggregate. Any feature requiring a network call to a second service is either cached with a stale-but-bounded value or dropped.
• Block C, protect the tail: a hard deadline with a defined degraded answer. If the account aggregate lookup exceeds 8 ms, score without it and set a flag, because a slow answer here is a failed checkout.
• Block D, rollout: you write this block.

Fade 2: a semantic search service, 200 M documents, p99 budget 150 ms. Last two blocks blank.

• Block A, locate the constraint: index size. 200 M vectors at 768 dimensions and 4 bytes is 614 GB, which does not fit one host, so fan-out and its tail arithmetic are forced unless the vectors are compressed or the corpus is partitioned by a filter that queries always carry.
• Block B, choose the shape and price it: you write this block.
• Block C, protect the tail: you write this block.

12f. Check yourself

Q1. A colleague proposes moving from a nightly batch recommender to fully online scoring, arguing that it is simpler because there is only one code path. Give the strongest version of their argument, then the number that decides it.

Q2. Your ranking service fans out to 12 shards and reports a p99 of 60 ms while each shard reports a p99 of 45 ms. Someone concludes the shards are fine. What do you tell them?

12g. When not to use this

Online scoring rather than batch precompute.

A cache in front of the model.

Sharding the index.

Shadow deployment.

Module 13. Training infrastructure and cost

Time: 35 to 45 minutes reading, 25 to 35 minutes practice.

Training infrastructure is missing from most interview preparation, which is strange, because retrain cadence constrains architecture more tightly than model choice does. If a full run takes 22 hours, a daily cadence is a promise you cannot keep, and every design decision downstream inherits that.

13a. Predict first

+---------------------------------------------------------------+
| Ranking model retrain, as it runs today                       |
|   training examples per day          10,000,000,000           |
|   example size on disk               2 KB                     |
|   dense parameters                   100 M                    |
|   embedding table                    1e9 rows x 64 dims       |
|   feature backfill before training   5 hours                  |
|   full training run, 16 devices      14 hours wall clock      |
|   validation and canary gate         3 hours                  |
|   target cadence                     daily                    |
+---------------------------------------------------------------+
Caption: the wall-clock stages of one retrain, with the model
sizes that determine how it must be distributed.

Question. Does the daily cadence hold? Compute two numbers before answering, and name the first change you would make.

Distribution modes, and what forces each one.

For classic ranking and recommendation systems, the honest answer is that you almost always need the third row and almost never need the last two. Say that, and then say what would change your mind.

Data-parallel communication, derived. Ring all-reduce moves about 2 x P bytes of gradient per worker per step, where P is the parameter payload in bytes, and this is nearly independent of worker count.

• 100 M dense parameters in 2-byte precision: P = 200 MB
• Traffic per worker per step: about 400 MB
• Interconnect at 100 Gbps, which is 12.5 GB per second: 32 ms per step

If a step's compute is 100 ms:

• Scaling efficiency = 100 / (100 + 32) = 0.76
• 16 workers give an effective speedup of 16 x 0.76 = 12.1

That number belongs in your answer. It converts "we will distribute it" into "16 devices buy a 12x speedup at 16x the hourly cost, so we pay 32 percent more money for the wall-clock win". That is a decision, not a slogan.

Larger batches move the ratio, and the trade is explicit. Doubling the batch roughly doubles compute per step while leaving communication per step unchanged, so efficiency rises. The cost is that very large batches often need a learning rate schedule change and can reduce final quality, so treat batch size as a tuned parameter with a quality measurement attached, not as a free lever.

Sparse embedding traffic, derived separately. Embedding gradients are sparse, so they do not go through the dense all-reduce.

• Batch 8,192, 50 active identifiers per example, 64 dimensions, 4 bytes
• 8,192 x 50 x 64 x 4 = 105 MB gathered per step
• At 12.5 GB per second: 8.4 ms

So embedding lookup traffic is a quarter of the dense gradient traffic here. Compute both numbers before deciding which one to optimise, because teams routinely optimise the one they find more interesting.

Checkpointing, with the optimal interval derived rather than guessed. Let C be the cost of writing one checkpoint and M the mean time between failures for the job. Daly's approximation gives an optimal interval of

T = sqrt(2 x C x M)

and a total overhead fraction of about sqrt(2 x C / M). Daly published the derivation in 2006 (A higher order estimate of the optimum checkpoint interval, Future Generation Computer Systems, accessed August 2026).

Worked, on interruptible capacity: 64 instances, each with an assumed 5 percent chance of preemption per hour, so the job's failure rate is 64 x 0.05 = 3.2 per hour and M = 1,125 seconds.

Now price the interruptible capacity. Assume a 65 percent discount, so you pay 0.35 of the on-demand rate:

• At C = 60 s: 0.35 x 1.327 = 0.464, so 54 percent cheaper. Take it
• At C = 300 s: 0.35 x 1.730 = 0.606, so 39 percent cheaper, with far more operational noise. Marginal

The conclusion is not "use interruptible capacity". It is that checkpoint write speed decides whether interruptible capacity is worth it, so asynchronous or incremental checkpointing is the enabling investment, not a nicety.

Checkpoint size, sized. Adam-style optimisers carry two moments plus the parameters. At 100 M parameters in 4-byte precision that is 100 M x 4 x 3 = 1.2 GB for the dense part, before the 256 GB embedding table. That contrast is the whole story: for recommendation models the checkpoint is dominated by embeddings, so incremental checkpointing of only the touched rows is what turns C from minutes into seconds.

GPU economics, derived end to end. Take these inputs and recompute them with your own numbers, because hardware and prices change faster than this page.

• Total: 10e9 x 50e6 x 3 = 1.5e18 FLOP per epoch
• Effective throughput: 300e12 x 0.40 = 1.2e14 FLOPS
• Single device: 1.5e18 / 1.2e14 = 12,500 s = 3.47 hours, costing 10.40
• On 16 devices at 0.76 efficiency: 3.47 / (16 x 0.76) = 0.285 hours wall clock, costing 16 x 0.285 x 3.00 = 13.70

Now compute the other two bills, because one of them usually wins.

Inference costs 40 times what a training epoch costs here, and the data path costs slightly more than the accelerators. Both facts are common and both are routinely missed, because candidates compute the one number they were taught to compute. Compute all three and say which one you would attack.

Retrain cadence constrains architecture. This table is the module in one place.

Read the last column as a constraint on the rest of your design. A team that promises hourly retraining has promised that no feature will ever require a nightly join, and that promise is usually broken within two quarters.

Cost levers, ordered by what they typically return.

+---------------------------------------------------------------+
| DAY BUDGET 24 h.  Rule: pipeline must fit in half the cadence |
+---------------------------------------------------------------+
   +----------+   +----------+   +----------+   +-------------+
   | BACKFILL |=>>| TRAIN    |=>>| VALIDATE |=>>| PROMOTE     |
   | 5 h      |   | 14 h     |   | 3 h      |   | canary 1 h  |
   +----------+   +----------+   +----------+   +-------------+
        |              |              |
        | stream       | warm start   | gate on a fixed
        | instead      | from         | golden set, then
        | of stage     | yesterday    | on canary metrics
        v              v              v
   +----------+   +----------+   +----------+
   | 0 h      |   | 4 h      |   | 1 h      |
   +----------+   +----------+   +----------+
   Total after the three changes: 6 h, inside the 12 h rule
Caption: the retrain pipeline as a wall-clock budget, with the
three changes that bring a 22 hour pipeline inside a daily cadence.

13d. Worked example

The prompt. "Design the training infrastructure for the ranking model in the previous module. Daily retrain, and finance has asked why the bill grew 40 percent last quarter."

Inputs are those from the pre-question artefact, plus a stated budget ceiling of the current spend plus 10 percent.

Block A. Purpose: find out what the money is actually buying

Before optimising anything I split the bill into the three buckets and get an order of magnitude for each. Using the derivations above: training around 14 per epoch, data movement around 14 per epoch, inference around 576 per day.

At one epoch per day, training plus data is under 30 per day and inference is 576. So the 40 percent growth is almost certainly not in training. I would check inference volume growth and feature count growth before touching the trainer.

I say this out loud, because the prompt invites me to optimise training and the correct first move is to refuse until the bill is decomposed.

The error most readers make here. Optimising the thing the question pointed at. The question pointed at training infrastructure, and the money is in serving.

Block B. Purpose: make the cadence achievable at all

Cadence is a correctness problem before it is a cost problem, because a pipeline that takes 22 of 24 hours fails permanently the first time anything goes wrong.

Three changes, each with its number:

Total: 6 hours, comfortably inside the 12 hour rule. Now a failed run has room to retry inside the same day, which is the property that makes the cadence real.

The risk I take on with warm start is drift accumulation: the model can carry forward a pathology that a cold start would have cleared. My mitigation is a scheduled cold retrain, weekly, compared against the warm chain on the same golden set. If the warm chain drifts, the comparison catches it in a week rather than a quarter.

The error most readers make here. Warm starting forever with no cold-start comparison. The failure is silent, slow, and shows up as an unexplained decline that nobody can attribute.

Block C. Purpose: choose the distribution strategy from the sizes

The embedding table is 256 GB, which decides the shape: shard embeddings across hosts, replicate the dense part, all-reduce the dense gradients, and gather the sparse rows.

I check both communication numbers before sizing the cluster:

• Dense all-reduce: 400 MB per worker per step at 12.5 GB/s is 32 ms
• Sparse gather: 105 MB per step is 8.4 ms
• Compute per step: 100 ms
• Efficiency: 100 / (100 + 32 + 8.4) = 0.71

At 16 workers the effective speedup is 11.4. Going to 32 workers, the communication terms stay roughly constant per worker while compute per worker halves, so efficiency falls to 50 / (50 + 40.4) = 0.55 and effective speedup is 17.7 for twice the money. The marginal 16 devices buy 6.3x of speedup at 16x of cost.

That table is the argument against the reflexive answer of adding hardware, and it took four multiplications.

The error most readers make here. Quoting a scaling efficiency without naming the interconnect. The same model on a slower fabric has a completely different answer, and the interviewer usually knows that.

Block D. Purpose: survive interruption and price the capacity choice

With a 4 hour run on 16 devices, and an assumed 5 percent preemption per device per hour, failures arrive at 16 x 0.05 = 0.8 per hour, so M = 4,500 s.

• With incremental embedding checkpointing at C = 45 s: T = sqrt(2 x 45 x 4500) = 636 s, and overhead is sqrt(90 / 4500) = 14.1 percent
• Effective cost on interruptible capacity: 0.35 x 1.141 = 0.40, so 60 percent cheaper

I take the interruptible capacity, and I state the dependency plainly: this only works because the checkpoint writes in 45 seconds, which is only true because we checkpoint the touched embedding rows rather than the full 256 GB. If someone later replaces incremental checkpointing with a full write, C goes to several minutes and the economics invert.

The error most readers make here. Treating interruptible capacity as a procurement decision. It is an engineering decision whose viability is set by checkpoint write time.

Block E. Purpose: state what the cadence forbids

Daily retraining is now real, and I close by naming what it forbids, because that is the constraint the rest of the design must respect:

• No feature may require a join that is only available in a weekly batch
• No validation step may require a human in the loop on the critical path
• Any label with a window longer than a day must be handled by the delayed feedback machinery from Module 9, not by waiting
• The rollback path must be a version pin in a registry, warm and loaded, since a bad daily model reaches production within hours

The error most readers make here. Treating the cadence as a scheduling detail rather than a contract. Every one of those four lines is a promise the rest of the system must keep, and the team that adds a weekly feature join in month three has broken the daily cadence without anyone filing a bug.

13e. Fade the scaffold

Fade 1: a fraud model that must retrain hourly because attackers adapt within a day. Last block blank.

• Block A, find the money: hourly retraining multiplies data pipeline reads by 24 unless the trainer is incremental, so the data path, not the accelerators, is where the bill moves. Compute the read volume per hour before anything else.
• Block B, make the cadence achievable: incremental training on the last hour plus a reservoir sample of history, with the full cold retrain nightly as the reference.
• Block C, distribution: the model is small and the feature space is modest, so a single device with a large batch is likely sufficient. Check it before distributing anything.
• Block D, interruption: an hourly job that takes 12 minutes has an expected preemption cost far below its checkpoint cost, so run it on on-demand capacity and skip checkpointing entirely inside a run.
• Block E, what the cadence forbids: you write this block.

Fade 2: a demand forecasting model, retrained weekly, that has grown from 4 hours to 30 hours on the same hardware over a year. Last two blocks blank.

• Block A, find the money: growth in wall clock with fixed hardware means either data volume growth, feature count growth, or a change in model size. Get the three time series before forming a hypothesis.
• Block B, make the cadence achievable: you write this block.
• Block C, distribution: you write this block.

13f. Check yourself

Q1. A team says distributed training gave them a 16x speedup on 16 devices. What single question exposes whether that number is real, and what is the most likely true figure?

Q2. Your training job runs on interruptible capacity at a 65 percent discount, and someone proposes replacing incremental checkpointing with a simple full checkpoint because the code is easier to maintain. Price the proposal.

13g. When not to use this

Distributed data-parallel training.

Sharded embedding tables and a parameter server style topology.

Interruptible or spot capacity.

Hourly or continuous retraining.

Module 14. Monitoring and drift

Time: 30 to 40 minutes reading, 20 to 30 minutes practice.

A model that is not monitored is a model whose quality is unknown, and unknown quality decays. The interview version of this topic is usually three drift definitions recited without a detection plan. The useful version is a ladder of signals ordered by how fast they arrive, and an alert budget computed from arithmetic rather than taste.

14a. Predict first

+---------------------------------------------------------------+
| Dashboard, yesterday against the trailing 28 days             |
|   feature 17 two-sample KS p-value          0.0001            |
|   feature 17 population stability index     0.02              |
|   mean predicted score            0.0208 => 0.0209            |
|   null rate, feature 42              0.3% => 4.1%             |
|   realised click rate               2.01% => 2.00%            |
|   labels arriving within 24 hours     98% => 71%              |
| Volume: 40,000,000 scored requests per day                    |
+---------------------------------------------------------------+
Caption: six monitored signals for one model on one day, with the
daily volume that determines what a p-value means here.

Question. Rank these by how urgently you would act, and name the one that is not evidence of anything at all.

Three shifts, defined before use, and which are detectable without labels.

The uncomfortable consequence: the shift that hurts most is the one no input-distribution monitor can see. Anyone who proposes feature drift monitoring as the answer to model decay has answered the easy third of the question.

Effect sizes that survive large samples.

Calibrate any threshold you use by replaying the last year: compute the metric day over day on days when nothing happened, take the 99^th percentile of that distribution, and set the threshold above it. A threshold derived from your own quiet days is defensible. A threshold copied from a blog post is not.

The signal ladder, ordered by arrival time. This is the structure to draw when asked how you would monitor a model.

Most teams build tier 4 first because it is the one that sounds like machine learning, and then discover that tier 1 catches most of their incidents.

Detection under delayed labels, made concrete. If your label window is 30 days, tier 4 is 30 days late. Build the bridge with the same arithmetic from Module 9.

• Measured maturity: F(1 day) = 0.62, F(7) = 0.88, F(30) = 1.00
• A partial 1-day conversion count divided by 0.62 gives an unbiased estimate of the eventual 30-day count for that cohort
• Its noise is the noise of the partial count, so a cohort with 200 observed conversions has a relative standard error of 1 / sqrt(200) = 7.1 percent
• So you can detect a 15 percent regression within a day, and a 5 percent regression takes about nine times the volume or nine times the days

That converts "labels are delayed so we cannot monitor" into "we can detect a 15 percent break tomorrow and a 5 percent break in a week", which is a design, not an excuse.

Alert design as an arithmetic problem. Decide the alert budget first, then derive the thresholds.

• Suppose you monitor 40 signals, each checked daily
• You will tolerate one false page per week
• Allowed false positive rate per check per day: 1 / (7 x 40) = 0.0036

A single-day threshold at 0.0036 corresponds to roughly 2.9 standard deviations for a symmetric signal. Alternatively, require two consecutive days outside a 2 standard deviation band: the per-day rate is about 0.046, and two independent consecutive breaches give 0.046 x 0.046 = 0.0021, which meets the budget with a tighter band and one extra day of delay.

That is the entire trade, and it fits in two lines:

False-alarm economics, stated plainly. Alert precision is real incidents divided by all pages. If you have 2 real incidents a month and 13 false pages a week, precision is 2 / 58, or 3 percent. At that precision the on-call engineer stops reading the pages, and your monitoring is now worse than nothing, because it produces false confidence and real fatigue.

Set a target precision, for example above 30 percent, and treat every false page as a bug with an owner. The fix is usually not a higher threshold. It is a better signal: a page on realised click rate is noisy, a page on the null rate of a top-10 feature almost never fires falsely.

Training-serving skew is the highest-yield monitor nobody builds. Log the exact feature vector used at serving time, then recompute those same features offline from the same sources and compare row by row.

• Alert on the fraction of rows where any feature differs beyond a tolerance
• A healthy system sits near zero; anything above about 0.1 percent of rows is a real bug
• This single check catches the class of failure that offline evaluation is structurally incapable of detecting

What the dashboard must contain, in one screen.

+---------------------------------------------------------------+
| TIER 0  traffic, errors, latency            seconds           |
+---------------------------------------------------------------+
| TIER 1  feature nulls, ranges, schema       minutes           |
+---------------------------------------------------------------+
| TIER 2  prediction distribution             minutes           |
+---------------------------------------------------------------+
| TIER 3  fast proxy labels                   hours             |
+---------------------------------------------------------------+
| TIER 4  matured labels, calibration ratio   days to weeks     |
+---------------------------------------------------------------+
| TIER 5  permanent holdout, cumulative gap   months            |
+---------------------------------------------------------------+
Caption: the monitoring ladder ordered by how fast each signal
arrives. Build upward from tier 0; a team that starts at tier 4
is blind for the first three days of every incident.

14d. Worked example

The prompt. "Design monitoring for the ad ranking system. Conversions take up to 30 days. You get one on-call rotation, so do not page them into uselessness."

Inputs:

Block A. Purpose: decide what a page is for before choosing any signal

I split the space into three response classes, because a signal with no distinct response should not be a page.

That mapping settles most arguments about thresholds. A drift metric that nobody would act on at 3 a.m. is a ticket, not a page, and it should be on a weekly review rather than in the alerting system.

The error most readers make here. Building an alert per metric. The right unit is an alert per response, and multiple metrics can feed one response.

Block B. Purpose: build the fast tiers, which catch most incidents

Tier 0 and tier 1 get automatic responses with no human:

• Error rate above 0.5 percent for 2 minutes: roll back
• p99 above the SLO for 5 minutes: roll back
• Fallback path serving above 1 percent of requests for 10 minutes: page, since a silent fallback is invisible in every quality metric
• Null rate on any top-20 feature above 3x its trailing 28-day median: page

I size the null-rate threshold from noise rather than taste. A feature with a 0.3 percent null rate over 2 B requests has 6 M nulls, and the relative standard error of that count is 1 / sqrt(6e6) = 0.04 percent. Day-to-day variation is therefore tiny, and a 3x jump is unambiguous. This is the rare signal where a huge sample makes the monitor better rather than useless.

The error most readers make here. Applying the same statistical machinery to every signal. Count-based integrity signals at high volume are nearly noise-free, while rate-based quality signals on thin slices are nearly all noise. Different signals, different thresholds, derived separately.

Block C. Purpose: bridge the 30-day label gap

Three quality signals at three latencies:

The middle row is the one that makes this design work, and it exists only because we measured the maturity curve in Module 9. I say that dependency out loud: the delayed feedback work and the monitoring work are the same work.

The error most readers make here. Treating the maturity curve as a constant. It is a measured quantity with its own drift, so it needs its own monthly refresh and its own comparison against the previous value.

Block D. Purpose: fit the page budget

I count the checks: 4 infrastructure, 20 feature integrity, 3 prediction distribution, and 12 quality checks across slices, so 39 checks.

Budget of one page per week over 39 daily checks gives an allowed per-check daily false positive rate of 1 / (7 x 39) = 0.0037.

Applying it:

• Integrity checks are near noise-free, so they use a fixed multiple rather than a statistical band and contribute almost no false pages
• Quality checks use two consecutive days outside a 2 standard deviation band, giving roughly 0.002 per check per day and about 0.6 pages per week across all 12
• Prediction distribution uses a single-day 3 standard deviation band because its response is a rollback and delay is expensive there

Total expected false pages: under one per week, with the fast signals kept fast. I write this arithmetic into the runbook, because the next engineer who adds 20 checks needs to see that the budget is a shared resource.

The error most readers make here. Adding monitors without a budget. Alerting systems degrade by accretion, and the failure is gradual until the rotation stops reading pages entirely.

Block E. Purpose: catch the decay that no page will ever fire for

A permanent 0.5 percent holdout served by a fixed non-personalised policy, never changed, never optimised.

Its purpose is not to detect an incident. It is to answer a question no experiment can: what has the sum of two years of launches actually delivered against doing nothing. Individual tests each measured a small win against the then-current baseline, and those wins do not add up cleanly, because each one shifted the population the next was measured on.

The cost is honest and I state it: 0.5 percent of users get a worse product permanently. At 2 B requests a day that is 10 M requests, and the design should rotate which users are in the holdout only if the metric being measured does not require a persistent cohort, which for cumulative effects it does. That is a real trade with a real cost, and it is the trade that lets you answer the question at all.

The error most readers make here. Proposing the holdout and then letting it be rotated, resized or quietly switched off during a capacity incident. A holdout that changes composition measures nothing, so its protection has to be written into the deploy process, not into a team's good intentions.

14e. Fade the scaffold

Fade 1: a content moderation model. Labels come from human reviewers with a 2 to 5 day queue, and enforcement changes attacker behaviour. Last block blank.

• Block A, what a page is for: automatic rollback on serving failures, page on review-queue volume breaching capacity, ticket for drift metrics.
• Block B, fast tiers: flagged rate per hour by content type, model score distribution, and the fraction of items hitting the default action because a feature was missing.
• Block C, bridging the label gap: reviewer decisions arrive in days, so use reviewer agreement rate on a small sample of already-actioned items as the fast proxy, and reserve full precision and recall for the weekly readout.
• Block D, page budget: you write this block.

Fade 2: an ETA model in a delivery product. Labels arrive within an hour of each trip, so there is no label delay problem at all. Last two blocks blank.

• Block A, what a page is for: rollback on serving failures, page when realised coverage of the promised time drops, ticket for feature drift.
• Block B, fast tiers: prediction distribution by city, null rate on traffic and courier features, and the rate at which the fallback estimate is served.
• Block C, quality signals: you write this block.
• Block D, page budget: you write this block.

14f. Check yourself

Q1. A colleague reports that a Kolmogorov-Smirnov test on 12 features shows significant drift on 9 of them, at a daily volume of 50 M rows, and proposes an emergency retrain. What do you say?

Q2. Your model's realised metric has been flat for six months and every one of the last eight experiments showed a positive lift. Reconcile these two facts, and name the single instrument that would have told you which explanation is right.

14g. When not to use this

Feature drift monitoring across the whole feature set.

A permanent non-personalised holdout.

Sequential or change-point detection methods.

Automatic retraining triggered by a drift alert.

Module 15. Pattern and anti-pattern catalogue

Time: 30 to 40 minutes reading, 25 to 35 minutes practice.

Distributed systems interviews have a shared vocabulary of failure: retry storm, cache stampede, hot key, split brain. Machine learning systems have the same recurring failures and almost no shared names for them, so each one gets described from scratch every time and the description carries no evidence that the failure was ever seen before. This module gives them names, symptoms and repairs.

15a. Predict first

+---------------------------------------------------------------+
| A three year old recommender. Six observations, one day:      |
| 1. Offline AUC has risen every quarter for two years          |
| 2. The realised engagement metric is flat over the same time  |
| 3. 11 percent of requests are served by the heuristic path    |
| 4. New items take 6 days to reach 1,000 impressions           |
| 5. The moderation threshold has not been changed since launch |
| 6. Nobody can name the owner of the second-stage ranker       |
+---------------------------------------------------------------+
Caption: six independent symptoms observed on one day in a mature
recommendation system, with no incident open.

Question. Each line is a named failure. Name as many as you can, then say which one you would repair first and why.

How to use a catalogue in an interview. Not as a recital. The value is recognition speed: when the interviewer describes a symptom, you name the class, state the mechanism in one sentence, and propose the detection before the repair. That sequence is what separates someone who has operated a system from someone who has read about one.

Serving patterns.

Training patterns.

Operations patterns.

Sculley and colleagues named the general problem of accumulating machine learning system debt in 2015 (Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015, accessed August 2026). Their paper names categories. What follows adds what an operator needs and what an interview rewards: the symptom you would observe, and the detection that separates one cause from another.

The anti-pattern catalogue.

Three that cost the most, with their arithmetic.

Feature time travel. The commonest single reason an offline win vanishes. The detection is cheap and specific: pick the feature with the highest gain, recompute it strictly as of the prediction timestamp, and re-run the evaluation. If the metric drops materially, you have found it. Teams skip this because it takes a day, and then spend a quarter on an experiment that was never going to work.

The silent fallback. Suppose 11 percent of traffic is served by a heuristic whose click rate is 1.2 percent while the model's is 2.1 percent. The reported blended rate is 0.89 x 2.1 + 0.11 x 1.2 = 1.87 + 0.13 = 2.00 percent.

The model looks 5 percent worse than it is, and every experiment run on blended traffic is diluted by the same fraction, so an 11 percent fallback rate silently reduces every measured effect by about 11 percent. That is often the difference between a significant result and a flat one.

Metric ratchet. Eight launches, each measured at plus 1.5 percent against its own baseline. Naive compounding predicts 1.015 to the power 8, which is plus 12.6 percent. The holdout says plus 2 percent. The gap is not fraud, it is arithmetic: effects measured against a moving baseline do not compose, novelty decays, and launches cannibalise each other. Without a holdout you cannot tell a team that ships well from a team that ships noise.

+-------------------+   +--------------------+   +----------------+
| DATA              |   | TRAINING           |   | SERVING        |
| feature time      |   | precision theatre  |   | silent         |
| travel            |   | champion eval set  |   | fallback       |
| two pipelines     |   | label lag denial   |   | frozen         |
| unbounded feature |   | warm chain drift   |   | threshold      |
+---------+---------+   +----------+---------+   +--------+-------+
          |                        |                      |
          +------------------------+----------------------+
                                   |
                                   v
                    +---------------------------+
                    | OPERATIONS                |
                    | metric ratchet            |
                    | alert accretion           |
                    | zombie model              |
                    | retrain on your own shadow|
                    +---------------------------+
Caption: the four zones where ML systems rot, with the named
failures that live in each. Failures in the operations zone are
the ones no single launch review will ever catch.

15d. Worked example

The prompt. "Here is a running recommender with the six symptoms from the artefact above. You have one quarter and two engineers. What do you do, in what order, and what do you deliberately not do?"

Block A. Purpose: separate what corrupts measurement from what costs money

I sort the six symptoms by whether they break my ability to measure anything else, because repairs made on corrupted measurements are guesses.

Ordering falls out: fix the measurement corruptions first, cheapest first.

The error most readers make here. Starting with the most intellectually interesting item, which is always the feedback loop. It is the slowest to repair and the hardest to measure, and it should be third.

Block B. Purpose: repair the cheap corruptions in the first three weeks

Week 1: fallback rate becomes a tier 0 metric with an alert. Then find out why 11 percent. In my experience the answer is usually one of three: a timeout that is too tight for a slow dependency, a null feature triggering the guard path, or a class of requests the model was never given inputs for. Each is a days-long fix, and each removes a chunk of the 11 percent.

Week 2: the frozen threshold becomes a target-rate threshold. Instead of "block above 0.83", the rule becomes "block the top 0.4 percent of scored items", with the score cut recomputed after every promotion. The target rate is set by review capacity, which is a real constraint, rather than by a number someone picked in 2023.

Week 3: launch the permanent holdout at 0.5 percent, and state honestly that it gives no answer for a quarter. That is the reason to do it in week 3 rather than week 12.

The error most readers make here. Rewriting the fallback path rather than first measuring which of the three causes it is. The fix is cheap once the cause is known and expensive when guessed.

Block C. Purpose: attack the feedback loop with a measurement, not a redesign

I do not propose a new architecture for popularity collapse. I propose an instrument and a small budget.

• Measure: impression share of the top 1 percent of items, monthly, over the last two years. This turns an argument into a trend.
• Instrument: log the serving propensity, which costs one field and unlocks every off-policy method later.
• Budget: 1 percent of requests get a slot filled by an item sampled from outside the model's top 100, with the sampling probability logged.

Then measure the cost of that 1 percent directly, so the exploration budget is defended by a number rather than by principle. If the exposed users lose 3 percent of engagement on 1 percent of traffic, the cost is 0.03 percent of the metric, which is a rounding error against the value of being able to evaluate anything counterfactually.

The error most readers make here. Proposing a diversity term in the loss as the first move. It changes the objective before anyone has measured the problem, and it makes the next quarter's comparisons harder.

Block D. Purpose: state what I will not do, and why

Closing sentence I would actually say: "Two engineers, one quarter, and the plan is measurement repair, an operating-point fix, a permanent holdout and a 1 percent exploration budget. No new model. I expect the first honest read on whether this system is improving in about four months, and that is the main deliverable."

The error most readers make here. Producing the not-doing list and then apologising for it. The list is the strongest part of the answer, because a plan that fits two engineers into a quarter is only credible once you have said what it excludes.

15e. Fade the scaffold

Fade 1: a fraud system. Symptoms: precision reported at 94 percent offline and 31 percent in production; the review queue has doubled with no rule change; the model was last retrained 8 months ago; blocked transactions have no outcome labels at all. Last block blank.

• Block A, separate corruption from cost: the offline-production precision gap is a measurement corruption, and the doubled queue is a direct cost. The missing labels on blocked transactions corrupt everything downstream.
• Block B, repair the cheap corruptions: the 94 versus 31 gap is precision theatre, so re-evaluate on the production base rate at the production threshold. Then convert the fixed threshold to a target review rate matched to team capacity, which fixes the doubled queue immediately.
• Block C, attack the structural problem: blocked transactions have no labels, so add a small random-release slice with logged probability. Without it, the model can never acquire data on the region it currently blocks.
• Block D, what I will not do: you write this block.

Fade 2: a search ranking system. Symptoms: every retrieval change loses offline and wins online; the offline harness is trusted by leadership; a cross-encoder reranker added last year serves 100 percent of traffic and nobody has measured its contribution. Last two blocks blank.

• Block A, separate corruption from cost: retrieval changes losing offline while winning online is the champion's eval set, a pure measurement corruption. The unmeasured reranker is a cost of unknown size.
• Block B, repair the cheap corruptions: you write this block.
• Block C, attack the structural problem: you write this block.

15f. Check yourself

Q1. An interviewer says: "Our offline metric has improved for six straight quarters and our engagement metric has not moved. What is going on?" Give three candidate explanations and the single instrument that separates them.

Q2. You inherit a model whose block threshold has not changed in two years. Nothing appears wrong. Name the specific measurement you would take, and the two possible results and what each implies.

15g. When not to use this

A named pattern catalogue as an answer structure.

A permanent exploration budget.

A permanent holdout for cumulative effect.

Rewriting a legacy component you have identified as a zombie.

Module 16. Level calibration

Time: 25 to 35 minutes reading, 25 to 35 minutes practice.

Every course tells you what to say. Almost none tells you what a mid, senior or staff version of the same answer sounds like, which is the thing candidates actually ask for. This module answers one prompt three times and annotates every difference.

16a. Predict first

+---------------------------------------------------------------+
| Three answers to: "how would you know this model is working?" |
| A: "I would track AUC offline and click-through rate online,  |
|     and set up dashboards for both."                          |
| B: "Click rate is the model's own objective, so I would make  |
|     revenue per user primary and click rate a diagnostic,     |
|     with unsubscribe rate as a guardrail at a stated margin." |
| C: "Before picking metrics I would ask what decision this     |
|     model's output changes, because our current metric moved  |
|     3 percent last year while the holdout moved 0.4, and I    |
|     want the measurement fixed before we optimise against it."|
+---------------------------------------------------------------+
Caption: three answers to one follow-up question, differing not in
knowledge but in what each treats as already settled.

Question. Assign each answer to mid, senior or staff, and state the single structural difference that made the assignment, without using the word experience.

The mistake this module exists to correct. The instinct is to level up by adding vocabulary. But the three answers above differ in scope, prioritisation and self-correction, not in vocabulary, and none of them names a technique the others could not. The additions that change the level are structural.

The eight behaviours that move a rating, in rough order of value.

What reads as over-reaching. Staff behaviours performed without the underlying design are worse than an honest mid answer.

+---------------------------------------------------------------+
| MID     builds the system as described, correctly              |
|         objective given, metrics given, scope given            |
+---------------------------------------------------------------+
                            | adds: chooses between designs,
                            v        names what breaks, deletes
+---------------------------------------------------------------+
| SENIOR  derives the design from constraints, with numbers      |
|         and states what it does NOT build                      |
+---------------------------------------------------------------+
                            | adds: questions the objective,
                            v        makes it measurable, sequences
+---------------------------------------------------------------+
| STAFF   chooses the problem, fixes the measurement, and        |
|         leaves a reversible path for whoever inherits it       |
+---------------------------------------------------------------+
Caption: each level contains the one below it. A staff answer that
skips the senior layer reads as evasion, not as seniority.

16d. Worked example

The prompt. "Design the system that decides which push notifications each user receives, and how many."

Shared inputs for all three answers, so the comparison is fair:

Block A. Purpose: the mid answer, in full, so the deltas are visible

"I would build a ranking system. Candidate notifications are generated by upstream producers, then a model scores each one for the probability the user opens it. I would use gradient boosted trees on user features, notification type, time of day and recent activity. Train on the last 30 days of send and open logs.

Serving is online, since candidates are event-driven. I would rank the candidates and send the ones above a threshold, up to the cap of 5.

For evaluation I would use AUC offline and open rate online, with an A/B test. For monitoring I would track open rate and model latency, and retrain weekly."

What is right about this. The pipeline is correct, the model choice is defensible, the serving mode matches the trigger, and it would work.

What caps it at mid. Every given was accepted. The objective is open rate, so the system will learn to send whatever gets opened, which is notification-shaped clickbait. The cap of 5 was inherited, not derived. There is no notion of what a notification costs the user, so the system has no way to choose zero.

The error most readers make here. Believing the mid answer is wrong. It is not wrong, it is unbounded: nothing in it can ever conclude "send nothing".

Block B. Purpose: what the senior answer adds, item by item

The senior version keeps the pipeline and adds five things:

1. It separates the goal from the trainable proxy. "Open rate is a proxy, and it is the wrong one on its own, because a notification that is opened and regretted is a loss. I would train on open, then evaluate against downstream session length and next-day return, and I would add unsubscribe and uninstall as guardrails with stated margins."

2. It makes the cap a decision, not an inheritance. "The cap of 5 is a constant nobody derived. I would measure the marginal effect of the nth notification of the day by capping at n for a random slice, and set the cap where the marginal notification stops paying."

3. It notices that a threshold requires calibration. "The send decision is a fixed threshold, so the score must be calibrated, not just ranked. If the score distribution shifts after a retrain, the send volume moves with no decision, which is the frozen threshold failure. I would set the cut by target send rate and recalibrate on every promotion."

4. It handles the delayed label. "The open label matures over about a day. That caps retrain cadence at daily unless I use a shorter window with an uplift factor, and I would only do that if the content half-life is under a day."

5. It states what it will not build. "No streaming trainer, no per-user reinforcement learning loop, no separate model per notification type. Daily retraining with a type feature fits inside the cadence, and I would revisit if the per-type quality gap exceeds a measured threshold."

The error most readers make here. Adding the guardrails but never stating a margin. "I would watch unsubscribes" is a mid sentence. "Unsubscribe rate must not rise by more than 0.05 percentage points, one-sided" is a senior sentence.

Block C. Purpose: what the staff answer adds, item by item

The staff version keeps everything above and adds four things, each of which is a scope change rather than a technique.

1. It prices the send in a common currency, and derives the reserve. "Every producer wants to send, so the real system is an allocation problem under a shared user attention budget. I would give every candidate a value in one currency: expected incremental sessions minus the expected cost in churn risk.

Take the measured numbers we would need: if the marginal notification produces 0.006 additional sessions at an assumed session value of 0.05, the gain is 0.0003. If it adds 0.0004 to the probability of uninstall against an assumed 30-day user value of 20, the cost is 0.008. Cost exceeds gain by more than an order of magnitude, so at that margin the correct action is to send nothing, and the reserve price is whatever value makes those two terms equal."

That derivation is the whole staff move: it converts "how many notifications" from a product argument into a break-even calculation with named, measurable inputs.

2. It names the organisational failure that will defeat the design. "Every producing team is measured on the opens their own notifications get, so each will lobby for exemptions from the budget. The system will not survive unless the budget is enforced centrally and producers are measured on their share of the user-level outcome rather than on their own open rate. I would propose that change alongside the design, because the design fails without it."

3. It designs the measurement to survive two years. "Per-launch A/B tests will each show a small win, and the sum will not appear. I would launch a permanent 0.5 percent no-notification holdout on day one, accept that it gives no answer for a quarter, and report cumulative effect against it quarterly. That is the only number that can settle whether this product is net positive."

4. It leaves a reversible path. "Migration order: shadow the value model against the current policy for two weeks, then run it as an advisory reserve on 1 percent, then enforce. Kill switch reverts to the current fixed threshold and the cap of 5, and it stays in place for two quarters. Blast radius of a bad value model is bounded by the daily cap, which is the reason I would keep a hard cap even after the budget is live."

The error most readers make here. Delivering item 2 first. Organisational framing before a design is on the board reads as avoidance, and interviewers have heard it from candidates who could not do item 1.

Block D. Purpose: the same follow-up, answered at three levels

Follow-up: "You ship it and open rate is up 6 percent, but daily active users are flat. What now?"

The error most readers make here. At every level, treating "flat" as "no effect". Compute the minimum detectable effect first, as in Module 11.

Block E. Purpose: signal one level up without over-reaching

Three moves that are cheap, safe and read one level higher than they cost:

And one move to avoid: do not question the prompt in your first two minutes unless you can then design under your own reframing. An interviewer who hears the reframe and no design has learned only that you can reframe.

The error most readers make here. Deploying all three moves in the first five minutes. Delivered together and early they read as a rehearsed opening. Delivered one each at the point where the design actually forces the choice, they read as judgement.

16e. Fade the scaffold

Fade 1: "design fraud detection for a payments product." Mid and senior blocks given, staff block blank.

• Mid: a supervised classifier on transaction and account features, threshold chosen for precision, online serving inside the payment path, weekly retrain, precision and recall monitored.
• Senior: adds that labels arrive as chargebacks weeks later, so the training set is systematically incomplete; that blocked transactions produce no labels at all, so a random-release slice is required; that the threshold must be set by review capacity rather than by score value; and that the latency budget inside checkout forces a small in-process model with a degraded path when a feature lookup is slow.
• Staff: you write this block.

Fade 2: "design ranking for a marketplace search page." Mid block given, senior and staff blank.

• Mid: two-stage retrieval and ranking, a gradient boosted ranker on query, item and user features, NDCG offline, click rate online, daily retrain, cached item features.
• Senior: you write this block.
• Staff: you write this block.

16f. Check yourself

Q1. You are interviewing for a senior role and you have 40 minutes. You notice the prompt's stated objective is the wrong thing to optimise. What exactly do you do, and in what order?

Q2. An interviewer says "that is over-engineered" about your design. Give the mid, senior and staff response, and say which one is a trap.

16g. When not to use this

Answering at a level above the one you are interviewing for.

Level deltas as a study device.

The marginal-value derivation shown in Block C.

Case study 1: Video recommendation

Two assumptions run through every case study on this page, because they turn latency and cost claims into arithmetic you can redo with your own numbers.

Say both out loud in the room. A number labelled as a placeholder that you offer to re-measure reads as engineering. The same number stated as fact reads as recall.

The prompt (video recommendation)

"Design the system that decides which videos to show on the home screen of a video app."

Questions that fork the design (video recommendation)

Requirements (video recommendation)

Functional:

• Return a page of 20 ranked videos for the home screen of a signed in user.
• Exclude items the user has already watched, and items the policy service marks ineligible for that user.
• Give newly published videos a path to their first impressions.
• Log every impression with its position, its retrieval source and the probability it was selected with.
• Refresh the page on a pull, without repeating what the user saw on the previous pull of the same session.

Non-functional:

Estimation that eliminates (video recommendation)

The sample size of 5,400 is derived in deep dive one rather than asserted here, because it needs two lines of arithmetic and one stated confidence level.

V0, the simplest thing that works (video recommendation)

Predict first: at what daily active user count does a nightly precomputed home screen stop being the right answer?

+--------+     +------------------+     +------------------+
| Client | --> | Home service     | --> | Precomputed rows |
+--------+     | read one row     |     | user id -> ids   |
               +------------------+     +------------------+
                                                 ^
                                        +-----------------+
                                        | Nightly batch   |
                                        | score and write |
                                        +-----------------+

Caption. V0 has a nightly batch job that scores a candidate set per user and writes one row of video identifiers per user; a home request is a single key lookup plus metadata hydration, with no model in the request path at all.

Why this is correct at a real company's scale:

• No online inference means no latency budget to split and no accelerator fleet to operate.
• The batch job can use a model that is far too slow to serve, which often beats a fast online model on quality.
• Debugging is trivial: the recommendation for any user is a row you can read.
• Retrieval and ranking still exist, they just both run offline, so the concepts transfer when you move online.

Breaking point, then V1 (video recommendation)

What fails: intra-session staleness. A user watches three cooking videos at 9am and the home screen still shows the list computed at 3am. How you observe it: bucket sessions by position within the day, then plot engagement rate against hours since the batch ran. A downward slope that is steep in the first hour and flat afterwards is a context signal problem. A slope that is flat all day is not, and means you should leave V0 alone.

The second failure is coverage. A video published at 10am gets zero impressions until the next batch, so the median time to first impression is about 12 hours. With a million new videos per day, that is a permanent 12 hour handicap on every new item.

V1 puts a staged pipeline in the request path:

+--------+    +--------------+    +-----------------+
| Client |--> | Home service |--> | Retrieval, 4    |
+--------+    +--------------+    | sources, 250    |
                                  | candidates each |
                                  +-----------------+
                                           |
                                           v
+--------------+  +---------------+  +-----------------+
| Re-rank and  |<-| Ranker over   |<-| Filter, dedup,  |
| explore mix  |  | 1000, batched |  | policy, seen    |
+--------------+  +---------------+  +-----------------+
       |                  ^
       v                  |
+--------------+  +---------------+
| 20 item page |  | Feature store |
+--------------+  +---------------+

Caption. V1 fans out to four retrieval sources in parallel for about 250 candidates each, removes ineligible, seen and duplicate items, scores the surviving 1,000 in one batched forward pass using features read from a feature store, then re-ranks for diversity and mixes in an exploration lane before returning 20 items.

The latency budget, which is where the design becomes real:

The headroom is the point. A budget that sums exactly to the objective has no room for a slow feature store call, and p99 is where slow calls live.

Breaking point, then V2 (video recommendation)

What fails: new items never accumulate the impressions they need to be rankable. The ranker's strongest features are the item's own historical engagement rates. A video with zero impressions has no such features, so it scores near the prior, so it stays below the 20 item cut, so it keeps zero impressions. That loop is self sealing and it does not resolve on its own.

How you observe it: plot the share of daily impressions going to items first published in the last seven days against the share of the eligible corpus those items represent. With 1 million new items per day and 200 million eligible, seven days of new items are 3.5 percent of the corpus. If they receive far under 3.5 percent of impressions, you have a starvation problem rather than a quality signal.

V2 adds a dedicated exploration lane and a cold start scorer, and it changes the logging contract:

Deep dive one: the exploration budget, priced in watch time

Exploration is usually discussed as a philosophy. It is a budget, and the budget is derivable.

Step one: how many impressions does one new item need? To distinguish a click rate of 3.5 percent from 4.5 percent at 95 percent confidence and 80 percent power, the standard two proportion sample size is approximately:

n = 16 * p * (1 - p) / d^2

The 16 comes from 2 times (1.96 + 0.84) squared, which is 15.7 rounded. With p = 0.035 (ASSUMED base click rate) and d = 0.01:

• n = 16 times 0.035 times 0.965 divided by 0.0001
• n = 0.5404 divided by 0.0001
• n = 5,404, call it 5,400 impressions per item

Step two: what does exploring everything cost? 1 million new items per day times 5,400 impressions is 5.4e9 impressions. You have 8e9 per day. Uniform exploration over new items consumes 67 percent of your entire inventory. That is not a budget, it is the product.

Step three: gate the candidates, not the budget. The cold start scorer ranks the day's new uploads before any of them is shown. Admit the top 5 percent:

Step four: price it. Assume exploration slots earn 40 percent of the engagement of exploited slots. That number is not a guess you carry in; it is directly measurable as the ratio of engagement on the exploration lane to the exploitation lane, and you will have it within a day of launching. The cost is 3.4 percent of impressions times a 60 percent shortfall, which is about 2.0 percent of total watch time, permanently.

Saying "we spend two percent of watch time on exploration, here is what it buys" is a different conversation from "we should explore more".

Step five: choose the mechanism, and notice this is not a classic bandit.

The last row is the senior point. A home screen is not a one armed decision. You observe outcomes for every slot you filled, which is far more information than a bandit formulation assumes, and the price of that information is that slot 1 and slot 20 are not comparable until you correct for position.

Step six: log propensities or lose the ability to evaluate. Off policy estimation, meaning "what would this new policy have earned on last week's traffic", needs the probability that the logged policy chose the item it chose. That float exists only at serving time. If you skip it, the only way to compare policies is a live experiment, which costs weeks per comparison. Eight billion floats per day at 4 bytes is 32 GB per day, which is a rounding error against your impression logs.

Deep dive two: recall accounting across the retrieval and ranking split

The split saved you nine orders of magnitude. The bill arrives as a question nobody asks until quality stalls: how much of the ranker's potential is the retrieval stage throwing away before the ranker ever sees it?

Define the metric precisely. Retrieval recall at k is the fraction of the items the ranker would have placed in its top k, had it scored the whole corpus, that actually appeared in the retrieved candidate set. It is a property of the pair (retrieval stage, current ranker), so it changes every time you retrain the ranker, which is the part teams miss.

Measure it without scoring 200 million items per request. Sample the traffic and sample the corpus:

Eight hundredths of one percent. There is no version of this where you cannot afford to measure it. Note the honest caveat: recall measured against a uniform corpus sample is optimistic, because the hardest misses are correlated with the items your retrieval already favours. Report it as a trend line, not as an absolute.

The four sources, and what each one uniquely covers.

The quotas are the design. Fixed quotas are predictable, debuggable and slightly wasteful. Learned quotas earn a few percent and cost you the ability to answer "why did this item not appear" in an incident. Start fixed; move only one quota at a time, and only when the recall metric above says that source is starved.

Where the filter goes, and why it is not a detail. Three placements, three different failure shapes:

The measurement that decides it is the distribution of drop rate by user, not its mean. Plot drop rate at p50 and p99. If p99 drop rate is under about 20 percent, over-fetch and move on. If it is over 50 percent, your heaviest users are being served from a quarter of the candidates everyone else gets, and no amount of ranker improvement will show up for them.

Seen suppression is a different problem from eligibility. Eligibility is a correctness constraint and must be exact, because showing a blocked item is a policy incident. Already watched is a quality preference where a rare mistake costs one repeated video. That asymmetry is the entire design: eligibility gets an exact check against the policy service, seen suppression gets an approximate structure sized for memory rather than exactness.

Failure modes and evaluation (video recommendation)

Service level indicators:

• Home request latency at p50 and p99, segmented by whether the exploration lane fired, because the exploration path has a different candidate mix and can hide a regression.
• Retrieval recall at 100 against the current ranker, computed daily from the 1,000 request sample.
• Impression share of items published in the last seven days, against their corpus share.
• Gini coefficient over per item daily impressions.
• Feature freshness lag at p99, measured as the age of the newest event reflected in a served feature.
• Empty or short page rate, which catches retrieval failures that latency metrics never see.

Alert on retrieval recall dropping more than a stated amount day over day (it moves whenever the ranker retrains, so alert on the delta, not the level), on feature freshness lag crossing the window the model was trained with, and on empty page rate deviating from baseline.

Re-check against the requirements:

Where this answer is a simplification (video recommendation)

All links in this list were checked on 1 August 2026.

• Covington, Adams and Sargin, "Deep Neural Networks for YouTube Recommendations" (RecSys, 2016) is the primary published description of the candidate generation plus ranking split used here, including the argument for training on all watches rather than only recommended watches.
• Chen and colleagues, "Top-K Off-Policy Correction for a REINFORCE Recommender System" (WSDM, 2019) is the clearest primary treatment of logged propensities and off policy correction for slate recommenders, which is the machinery the propensity logging above exists to enable.
• Steck, "Calibrated Recommendations" (RecSys, 2018) argues that a recommender should preserve the proportions of a user's interests rather than collapse onto their strongest one, which is the re-rank stage this case study treats too briefly.
• Chandrashekar, Amat, Basilico and Jebara, "Artwork Personalization at Netflix" (Netflix Technology Blog, 7 December 2017) is a readable primary account of running contextual bandits in production, including the offline replay evaluation problem.
• McInerney and colleagues, "Explore, Exploit, and Explain" (RecSys, 2018) connects bandit exploration to explanation, which matters because unexplained exploration reads to users as a broken product.
• The real objective is not watch time. Every serious version of this system predicts several outcomes and combines them, which is case study 2 on this page. Treating watch time as the target is a teaching simplification that a good interviewer will challenge within five minutes.
• Multi-surface reality. Home, next-video, search, subscriptions and notifications all want candidates. Building five retrieval stacks is the actual architectural problem inside a large company, and the answer is usually a shared candidate service with per surface quotas.

Level delta (video recommendation)

What each level added:

• Mid to senior: the architecture stops being a remembered pipeline and starts being derived from two divisions, and the first design offered is the small one.
• Senior to staff: every knob becomes a priced lever with a measurement attached, the measurement itself gets costed to prove it is affordable, and the weakest part of the design is volunteered before the interviewer finds it.

Case study 2: Social feed ranking

Case study 1 optimised one outcome. This prompt has six, they disagree, and one of them is negative. The interesting work is not the model, it is deciding what the number the model produces is worth.

The prompt (feed ranking)

"Design the ranking system for a social feed, where a user opening the app sees a mix of posts from accounts they follow, groups they belong to, and recommended content."

Questions that fork the design (feed ranking)

Requirements (feed ranking)

Functional:

• Return a page of 20 posts drawn from follow, group and recommendation inventory.
• Predict several outcomes per candidate and combine them into one ordering.
• Apply integrity actions produced by a separate policy system, before the ordering is returned.
• Guarantee that a post the policy system has removed cannot be served, at any latency.
• Log every impression with position, predicted values per head, and the integrity actions that applied.
• Support changing the combination weights without retraining the model.

Non-functional:

Estimation that eliminates (feed ranking)

V0, the simplest thing that works (feed ranking)

Predict first: what does a feed ranked purely by predicted click probability start doing within a few weeks, and which metric shows it first?

+--------+   +--------------+   +----------------+
| Client |-->| Feed service |-->| Candidates     |
+--------+   +--------------+   | recent, unseen |
                    |           +----------------+
                    v
             +--------------+   +----------------+
             | One model    |-->| Sort by score  |
             | p(click)     |   | take top 20    |
             +--------------+   +----------------+

Caption. V0 fetches recent unseen candidates, scores each with one model predicting click probability, sorts, and returns the top 20; there is no pre-ranker, no multi-head combination and no integrity layer.

Why this is correct at a real company's scale:

• One model, one label, one metric. Every regression has one place to look.
• The value model, the weight governance process and the experiment queue that comes with them do not exist yet, and they are organisational costs as much as technical ones.
• With a small community, integrity is handled by people reading reports, which is both cheaper and better than a classifier trained on 40 positive examples.
• A single head that is well calibrated is worth more than six heads that are not, and calibration is easier to get right once.

Breaking point, then V1 (feed ranking)

What fails: 657 accelerators, and the fact that the single objective is provably wrong.

The cost failure is arithmetic: one heavy model over 3,000 candidates at 43,800 requests per second is 1.314e8 scorings per second. How you observe it is trivially, on the invoice, but the design signal arrives earlier as p99 latency climbing linearly with candidate count, which is the signature of an unstaged ranker.

The quality failure is the one from the pre-question. How you observe it: pick one negative outcome you can label cheaply, such as hide, and plot it against your positive metric over successive model launches. If clicks rise 4 percent and hides rise 9 percent across three launches, the objective is selecting for something you did not intend.

V1 splits the ranking stage and the model head:

+--------+  +--------------+  +----------------+
| Client |->| Feed service |->| Retrieval      |
+--------+  +--------------+  | 3000 candidates|
                   |          +----------------+
                   v                   |
            +--------------+           v
            | Pre-ranker   |<--+----------------+
            | cheap, 3000  |   | Feature store  |
            +--------------+   +----------------+
                   |
                   v
            +--------------+  +----------------+
            | Heavy ranker |->| Value model    |
            | 600, 6 heads |  | weights served |
            +--------------+  +----------------+
                                       |
                                       v
                              +----------------+
                              | Top 20         |
                              +----------------+

Caption. V1 narrows 3,000 candidates to 600 with a cheap pre-ranker, scores the 600 with one shared trunk producing six calibrated probabilities, then combines them using weights served from configuration rather than baked into the model, and returns the top 20.

The two changes are independent and worth stating separately:

Breaking point, then V2 (feed ranking)

What fails: a policy decision cannot wait for a retrain.

A category of content becomes harmful on a Tuesday afternoon. Your options in V1 are to retrain the ranker, which is a day, or to hand edit a weight, which is blunt and affects everything. Neither meets the 60 second removal requirement, and neither produces a record of what was done to which post and why.

How you observe it: measure the time from a policy decision being made to the last impression of the affected content. In V1 that number is bounded by your retrain cadence and it will be measured in hours or days.

V2 adds an integrity action layer between the value model and the response:

The rule layer is not a fallback for a weak model. It exists because policy changes on a different clock from model training, and because a demotion you cannot attribute to a rule is a demotion you cannot explain, appeal or audit.

Deep dive one: combining six predictions into one ordering

The model outputs six calibrated probabilities. Ranking needs one number. Everything hard about this system lives in that reduction.

Start with the shape. The default is a weighted sum of calibrated probabilities:

value = w1*p(click) + w2*p(long dwell) + w3*p(like)
      + w4*p(comment) + w5*p(share) - w6*p(hide or report)

Two properties make this the right default and one makes it dangerous:

That last row is the cleanest test for whether something is a weight or a rule. If there is any value of the other predictions that should not be allowed to override it, it is a rule.

Now the hard part: what is a weight? A weight is an exchange rate. Setting w3 to 8 says one like is worth eight times one unit of whatever w1 is denominated in. Nobody can defend that number in the abstract, so do not try. Two workable methods:

The honest answer combines them: search offline, confirm the top two or three candidates online. This is forced by the estimate above, where a naive grid needs 7.2 times your user base.

The cannibalisation table is the artefact to draw. For each weight you consider raising, name what falls:

A candidate who lists the metric that falls for each weight they raise is doing the actual job. A candidate who says "we optimise for meaningful interactions" has restated the problem.

Calibration is a prerequisite, not a refinement. A weighted sum of six numbers is meaningless if the numbers are not on a common scale. If p(share) is systematically 3x too high in a slice, w5 is effectively 3x larger for that slice, and you have a per slice value model you never designed. That is why calibration gets a full deep dive in case study 3 rather than a mention here: in ads it decides money, and here it silently decides your exchange rates.

Rare heads need care. Report and hide have perhaps three orders of magnitude fewer positives than click. Three consequences:

• Loss weighting or the head contributes nothing to the shared trunk's gradients.
• Calibration on a rare head is noisy, so bin by decile and accept wider error bars.
• The confidence interval on the rare head's contribution to value should be checked before the weight is raised, or you are ranking on noise.

Deep dive two: integrity as a constraint layer, not a feature

The moderation system that produces integrity scores is its own design problem, with its own labelling pipeline, appeal loop and precision economics, and it is a separate case study on this page. This dive is narrower and more often botched: given those scores, how do they enter ranking?

Three actions, three mechanisms, three different correctness requirements.

Why not just feed the integrity score into the ranker as a feature? It is a fair question and the interviewer will ask it. Five reasons, in the order they will actually bite you:

Note what this does not say. Integrity signals can also be features, and usually are, for the many cases where a soft nudge is right. The argument is that a policy with consequences must not be only a feature.

Sizing the human loop, because thresholds are staffing decisions.

The threshold that routes to humans is where precision and recall become dollars. Lower it to 0.8 percent and you need 160 reviewers. Raise it to 0.2 percent and you halve the staff and double the harm that reaches the feed unreviewed. That trade is a budget conversation with a precision-recall curve attached, and framing it that way is a senior move.

Propagating a removal in under 60 seconds. The requirement is not met by the classifier, it is met by the distribution path:

Failure modes and evaluation (feed ranking)

Service level indicators:

• Feed request latency at p50 and p99, split by pre-ranker and heavy ranker stages, so a regression has an owner.
• Pre-ranker recall at 600 against the heavy ranker.
• Per head expected calibration error, by slice, refreshed daily.
• Time from policy decision to last impression, at p99. This is the removal requirement, measured directly.
• Deny set staleness at p99 across ranking nodes.
• Guardrail panel: hides per impression, reports per impression, and the share of sessions ending immediately after one click.
• Weight configuration change log, joined to metric movements, so the exchange rates have a history.

Alert on deny set staleness, on p99 removal propagation crossing 60 seconds, on any per head calibration error breaching its threshold, and on the guardrail panel rather than on the north star, because the north star is what a bad launch improves.

Re-check against the requirements:

Where this answer is a simplification (feed ranking)

All links in this list were checked on 1 August 2026.

• Zhao and colleagues, "Recommending What Video to Watch Next: A Multitask Ranking System" (RecSys, 2019) is the primary published description of a multi task ranker with a separate shallow tower for position bias, which is the mechanism referenced above.
• Meta's engineering post "How machine learning powers Facebook's News Feed ranking algorithm" (Meta Engineering, 26 January 2021) describes a three pass design with a lightweight pass 0 that selects roughly 500 posts before the heavier scoring pass, and a weighted combination of several predicted actions, which is the shape of V1 here.
• Meta's Transparency Center page "Types of content we demote" states that some content is reduced in distribution rather than removed, and groups the demoted categories by rationale, including clickbait, engagement bait and fact checked misinformation. It is a useful existence proof that the removal against demotion split can be written down and published, which is the property the audit trail above depends on.
• X, formerly Twitter, published the source of its recommendation pipeline in March 2023 at github.com/twitter/the-algorithm, which is an unusually direct primary source for the stage structure of a production feed: candidate sources, a heavy ranker and a set of post ranking filters, readable as code rather than described in prose. The trained weights are not in the repository, so read it for the shape and not for the numbers.
• Mosseri, "Instagram Ranking Explained" (Instagram Blog, 31 May 2023) is a primary account of per surface ranking signals from the product owner, with different signal orderings named for Feed, Stories, Explore and Reels.
• The value model is not the whole ordering. Diversity constraints, author fatigue rules, advertisement insertion and inventory pacing all sit between the value model and the rendered page, and each of them can dominate the ordering in practice.
• Off policy evaluation is harder than described. Estimates degrade where the logging policy rarely went, and the proposals worth testing are exactly the ones that go there. Treat the offline search as a candidate generator for experiments, never as a substitute for them.

Level delta (feed ranking)

What each level added:

• Mid to senior: the ordering stops being one number and becomes a combination with a stated shape, and the ranking stack is staged for a reason with a number behind it.
• Senior to staff: thresholds become budget and staffing decisions with named owners, the weight tuning method is derived from a power calculation rather than asserted, and the boundary between what the system can and cannot explain is volunteered.

Case study 3: Ad click prediction

In case studies 1 and 2 the model's output was consumed by a sort, and a sort is invariant to any monotone transformation of the scores. Here the output is consumed by an auction that charges money, and an auction is not. That one sentence is the whole reason this problem is different.

The prompt (ad click prediction)

"Design the system that predicts whether a user will click an ad, so the ad server can decide which ad to show and what to charge."

Questions that fork the design (ad click prediction)

Requirements (ad click prediction)

Functional:

• For each ad request, return the winning ad and the price to charge, within the exchange timeout.
• Predict click probability per candidate creative, calibrated to absolute probability, not just ordered correctly.
• Predict conversion probability for campaigns bidding on conversions.
• Attribute conversions to impressions within the advertiser's stated attribution window.
• Log every auction with the predicted probabilities, the bid, the clearing price and the model version.
• Support daily model retraining without a calibration regression.

Non-functional:

Estimation that eliminates (ad click prediction)

V0, the simplest thing that works (ad click prediction)

Predict first: you double every predicted click probability. What breaks, and what does not?

+---------+   +---------------+   +----------------+
| Request |-->| Targeting     |-->| Sparse model   |
+---------+   | index, 2000   |   | p(click)       |
              +---------------+   +----------------+
                                          |
                                          v
                                  +----------------+
                                  | Auction        |
                                  | rank by bid    |
                                  | times p(click) |
                                  +----------------+

Caption. V0 reduces 50 million creatives to 2,000 with a targeting index, scores each with one sparse logistic model, ranks by bid multiplied by predicted click probability, and runs the auction; there is no calibration layer, no conversion model and no delayed feedback handling.

Why this is correct at a real network's scale:

• A sparse linear or factorisation model over hashed features trains in minutes and serves in microseconds, and for click prediction it is a genuinely strong baseline rather than a placeholder.
• The whole system is one model, one label that arrives in seconds, and one metric.
• With a second price auction and no reserve, ranking is all you need, so calibration can wait.
• It costs 0.028 percent of revenue, which means you can spend all your attention on the targeting index and the logging, both of which you will need forever.

Breaking point, then V1 (ad click prediction)

What fails: the model outputs 33 percent where the truth is 0.5 percent.

You downsampled negatives at 1 in 100 to make daily training affordable, which was correct. The consequence is that the sampled dataset has a click rate of:

p_s = p / (p + r * (1 - p))
p_s = 0.005 / (0.005 + 0.01 * 0.995) = 0.005 / 0.01495 = 0.334

The model faithfully learns 0.334. Ranking is untouched. Pacing, reserve comparison and reporting are all wrong by a factor of 67.

How you observe it: plot predicted probability against observed click rate in deciles of prediction. A model with correct ordering and broken calibration produces a curve that is monotone and nowhere near the diagonal. The single number version is expected calibration error, the average absolute gap between predicted and observed rate across bins, weighted by bin population.

V1 adds an explicit calibration stage and a per slice calibration monitor:

+---------+   +---------------+   +----------------+
| Request |-->| Targeting     |-->| Sparse model   |
+---------+   | index, 2000   |   | p_s sampled    |
              +---------------+   +----------------+
                                          |
                                          v
              +---------------+   +----------------+
              | Auction and   |<--| Recalibrate    |
              | pacing        |   | invert sample  |
              +---------------+   | then isotonic  |
                     ^            +----------------+
                     |                    ^
              +---------------+   +----------------+
              | Reserve and   |   | Calibration    |
              | price rules   |   | monitor, slices|
              +---------------+   +----------------+

Caption. V1 inserts a two step recalibration between the model and the auction: an exact algebraic inversion of the negative downsampling, then a fitted monotone correction, with a monitor that reports calibration error per slice rather than only in aggregate.

Breaking point, then V2 (ad click prediction)

What fails: a third of your conversions have not happened yet when you train.

For campaigns bidding on conversions, the label is not "did they click" but "did they convert", and conversions arrive over days. Train nightly on yesterday's clicks and label everything unconverted as a negative, and you are training on a systematically incomplete label.

How you observe it: take a cohort of clicks from a fixed hour, then plot the cumulative conversion count for that cohort against days elapsed. If the curve is still rising at day seven, every model trained with a shorter window is biased low, and the bias is a number you can read straight off the curve.

V2 adds a conversion path with explicit delay handling:

Deep dive one: calibration, because the auction spends money on the third decimal

Why ordering is not enough, stated once and precisely. A sort is invariant under any strictly increasing transformation of scores. An auction is not, because at least four downstream consumers read the number itself: