What you need to know
Offline evaluations tell you whether a change meets criteria on a test set. They cannot tell you how real users and real traffic respond. For that you run a controlled experiment on live traffic, or, when exposure is too risky, a shadow test.
The release ladder
- Offline evaluation on the golden dataset: does the change meet the thresholds and keep the regression suite green?
- Shadow test where risk is high: the new version processes copies of real requests; only the current version's output reaches users.
- A/B test (or a small canary) where some exposure is acceptable: a random share of users gets the new version.
- Staged rollout with monitoring and a rollback trigger.
Each step answers a different question. Skipping straight to an A/B test for a high-risk change, or treating an offline score as proof of user impact, are both mistakes.
Five parts of a sound A/B test
| Part | What to get right |
|---|---|
| Hypothesis | Specific and falsifiable: the change, the expected effect, the primary metric, and the constraints that must hold |
| Assignment | Random, and sticky per user or session, so groups are comparable and no one sees both versions |
| Primary metric | One metric, chosen before the test. Guardrail metrics (latency, cost, safety) must not get worse |
| Sample size | Enough traffic to detect the smallest effect worth acting on, at the confidence the stakes demand |
| Interpretation | Judge against the hypothesis set in advance; do not stop early on a promising trend or go looking for a metric that improved |
A hypothesis that works: "Moving product questions to the small tier will cut cost per resolved contact by at least 30%, while resolution rate falls by no more than 2 points and 95th-percentile latency does not rise."
How much traffic is enough?
The smaller the difference you want to detect, the more data you need, roughly in proportion to the square of how small it is. A rough rule for a rate (such as resolution) at 95% confidence and 80% power:
samples per group ≈ 16 × p × (1 − p) ÷ d², where p is the baseline rate and d the difference to detect.
At a 70% baseline, detecting a 5-point change needs about 16 × 0.21 ÷ 0.0025 ≈ 1,350 per group; detecting a 2-point change needs about 8,400 per group. Model outputs vary more than traditional software, which pushes these numbers up. Higher stakes call for higher confidence and therefore more data.
When to shadow test instead
| Situation | Approach |
|---|---|
| A single bad output could cause real harm (clinical, legal, financial) | Shadow test first |
| Not enough traffic to split | Shadow test, then a small canary |
| Some exposure is acceptable and user behaviour is what matters | A/B test |
| You need to know how users react (satisfaction, conversion, escalation) | A/B test, since shadow outputs never reach users |
Shadow tests compare outputs on real inputs, usually with graders and human review. They cannot measure user reactions.
Iterating
Every experiment should leave the system better instrumented: new failure cases go into the golden dataset, and the winning version becomes the new control. Improvement is a loop, not a launch.
In the exam
Experiment questions give a proposed change, traffic volumes and a risk level. Choose offline evaluation first, shadow testing for high-risk changes or low traffic, and A/B testing when user behaviour must be measured and exposure is acceptable. Check that the hypothesis, primary metric and stop rule were set in advance, assignment is random and sticky, and the sample is large enough for the effect size.
Check your understanding
A new model for drafting medication instructions has passed offline evaluation. What should come next?
Show answer
Answer: D. A single bad instruction could harm a patient, so the new version should process real inputs without reaching patients until reviewed. Opt-in groups are also not random.
Three days into a planned four-week A/B test, the new version shows a 6-point lift. The product owner wants to ship now. What is the right response?
Show answer
Answer: C. Early results are noisy, and stopping on a good-looking trend inflates false positives. The stop rule was set in advance for a reason.
Which TWO elements belong in an A/B test plan before it starts?
Show answer
Answer: A, D. Deciding the primary metric, guardrails and stopping point in advance is what makes the result trustworthy. Picking metrics afterwards and steering inputs both bias the outcome.
Apply it: Design Corvane's tier experiment
In lesson 2.1 you designed an offline evaluation for moving product questions from the balanced to the small tier. It passed. Now design the live experiment. Product questions are 21% of 95,000 contacts a month, and the current resolution rate on them is 70%.
- Write the hypothesis, with the primary metric and the guardrails.
- Decide on shadow, A/B or both, and justify it from the risk.
- Estimate the sample size and duration to detect a 2-point drop in resolution, using the rule of thumb in this lesson.
Done when: you know how many days the test must run at current volume. - Write the stop and rollback rules.
Show model answer
1. Hypothesis. Moving product questions to the small tier cuts cost per resolved product contact by at least 30%, while resolution rate falls by no more than 2 points, 95th-percentile latency does not rise, and citation accuracy stays at or above 98%. Primary metric: resolution rate (the risk being tested); cost per resolved contact is the expected benefit and is reported alongside.
2. Approach. An A/B test. A wrong product answer is recoverable (the customer can ask again or escalate), user behaviour is exactly what needs measuring, and the offline evaluation has already passed. A shadow phase adds little here. Assignment is random and sticky per customer, 50/50.
3. Sample size. 16 × 0.70 × 0.30 ÷ 0.02² = 16 × 0.21 ÷ 0.0004 = 8,400 contacts per group. Volume: 0.21 × 95,000 ≈ 19,950 a month, about 665 a day, so about 333 per group per day. 8,400 ÷ 333 ≈ 25 days. Plan four weeks, and avoid starting just before peak season, when the traffic mix changes.
4. Rules. Fixed duration of four weeks; no decision before then. Stop early only for harm: citation accuracy below 95% or escalations up more than 5 points in any week, which also triggers rollback. At the end, adopt only if resolution is within 2 points, latency has not risen and cost per resolved contact has fallen by at least 30%; otherwise keep the balanced tier.