What you need to know
A system that passed its evaluation can still drift in production: the inputs change, documents are updated, users find new ways to ask, upstream systems change their formats. Monitoring is how you notice before customers or auditors do.
What to monitor
| Area | Signals |
|---|---|
| Quality | Sampled online evaluation scores, human-review outcomes, user corrections, escalation and override rates |
| Reliability | Error rates, timeouts, tool failures, stop reasons (especially output-limit and context-exceeded), retries |
| Latency | 50th and 95th percentile total time and time to first token, per route |
| Cost | Tokens and cost per request and per successful outcome, cache hit rate, per route and per tenant |
| Safety and security | Guardrail triggers, blocked actions, injection attempts detected, policy violations found in review |
Online evaluation
Offline test sets do not see this week's traffic. Online evaluation grades a sample of real production outputs with the same graders: code checks on everything that can be checked cheaply, a model judge on a sample, and human review on a smaller sample plus anything flagged. Trends in these scores are the earliest warning of drift.
Classify outputs, not only errors
A useful habit is to sort reviewed outputs into three bins:
- Acceptable: meets the criteria.
- Needs revision: fixable by regenerating or editing.
- Needs override: a person or a different path must take over.
The rates of each bin, by route and slice, show where the system is weakening. Every "needs override" case is a candidate for the golden dataset.
Alerts that mean something
Alert on changes from a baseline, sized to matter, sustained for long enough to rule out noise: escalation rate up 4 points for a day, cost per success up 30% for an hour, 95th-percentile latency over target for 15 minutes, any zero-tolerance violation at once. Each alert needs an owner and a first step.
Close the loop
Monitoring that only produces dashboards changes nothing. The loop is: detect, diagnose (lesson 4.4), fix, add the case to the evaluation set, and confirm the metric recovers.
In the exam
Monitoring questions describe a quality drop, a cost spike or a silent change, and ask what should have caught it or what to add. Strong answers include online evaluation on sampled traffic, outcome signals, baselines with meaningful thresholds, pinned and logged versions, and a loop back into the evaluation set.
Check your understanding
A summarization service has had no errors for months, yet reviewers say summaries have become less accurate. What monitoring was missing?
Show answer
Answer: D. Quality drift does not raise errors. Grading a sample of live outputs with the same criteria as the offline evaluation shows the trend.
An assistant's behaviour changed overnight with no deploy on the team's side. The application calls a model alias rather than a pinned version. What should change?
Show answer
Answer: C. A moving alias can change the model underneath the application. Pinning makes model changes deliberate and testable.
Which TWO alert conditions are well designed?
Show answer
Answer: A, C. Good alerts compare against a baseline with a meaningful, sustained threshold, or fire immediately on zero-tolerance events. Single-request and any-change alerts are noise.
Apply it: Monitor Larkfield's intake in production
Larkfield's intake system goes live next month. Design its monitoring.
- Choose two signals for each area: quality, reliability, latency, cost, safety.
- Design online evaluation: what is checked on every claim, what is sampled, and who reviews.
Done when: you know how many claims a week a person looks at. - Write four alerts, each with a threshold, an owner and a first step.
- Describe how findings reach the evaluation set.
Show model answer
1. Signals
| Area | Signals |
|---|---|
| Quality | Handler correction rate per field; sampled extraction accuracy against the source |
| Reliability | Claims stuck in the queue over 4 hours; Policy Admin call failures |
| Latency | Time from receipt to acknowledgement (against the one-day SLA); extraction 95th percentile |
| Cost | Cost per claim; share of claims over 20 pages |
| Safety | Any acknowledgement mentioning a decision on cover; any EU claim processed outside the EU route |
2. Online evaluation. Code checks on every claim: schema valid, key fields match Policy Admin, no decision language, correct regional route. A model judge on 5% of claims for completeness of the summary. The intake team lead reviews 50 claims a week (stratified by channel, with all scanned-post claims flagged as low confidence), sorting each into acceptable, needs revision or needs override.
3. Alerts
| Alert | Threshold | Owner | First step |
|---|---|---|---|
| Correction rate | Any field above 3% for a day | AI team | Check recent prompt or schema changes; pull examples |
| SLA risk | More than 50 claims queued over 4 hours | Claims IT | Check model availability and the fallback to handlers |
| Decision language | Any occurrence | Compliance and AI team | Block sending; review the template |
| Cost per claim | 40% above the 30-day baseline | AI team | Check long-document volume and cache hit rate |
4. Feedback. Every "needs override" claim and every corrected field is added weekly to the golden dataset under its slice, and the next prompt or model change must pass them.