What you need to know
Monitoring (lesson 4.5) tells you what is happening. A feedback loop decides what happens next, and who needs to know. Without one, dashboards fill up, alerts are ignored, and stakeholders hear about problems from their customers.
The loop
- Signals: what is the system showing? Metrics, review findings, user feedback, incidents.
- Triage: what needs attention now, what can wait, what is noise?
- Decide: does this need an engineering fix, an architect's review, a stakeholder decision, or nothing?
- Act: fix, adjust a guardrail, escalate, or change scope.
- Review: did it work? Should the trigger or the rule itself change?
Four triage destinations
| Destination | What goes there | Example |
|---|---|---|
| Internal monitoring | Routine engineering signals within agreed limits | 95th-percentile latency up 50 ms, still under target |
| Architect review | Signals that threaten a design assumption or business constraint | Evaluation score falling three weeks running; cost per case over budget |
| Stakeholder review | Agreed commitments and governance checkpoints | SLA breach; scheduled compliance confirmation; scope change |
| Noise | Self-resolving or insignificant events | One malformed request from a known bad client; a retry that succeeded |
Escalating everything to stakeholders is as harmful as escalating nothing: it trains them to ignore you. Agree the routing of each signal in advance.
SLAs are commitments
A service level agreement states:
- the metric (for example, 95th-percentile response time, measured at the gateway),
- the threshold and the measurement window (under 3 seconds, per rolling hour),
- what counts as a breach (two consecutive hours over threshold),
- the response (who is notified, what is done, by when, and what the customer is told).
Thresholds come from somewhere tangible: user expectations, business criticality, evaluation criteria, agreed budgets. A number the engineering team picked because it was easy to meet is not an SLA. For AI systems, SLAs often include quality as well as speed: an accuracy floor on the evaluation set, or a maximum escalation rate.
Managing expectations
Stakeholders' expectations drift: early demos set a high bar, a good first month sets another, a competitor's announcement sets a third. Keep a shared, written record of what was agreed (criteria, SLAs, known limitations) and review it at a regular cadence, with production evidence. Surprises should come from the world, not from misunderstandings.
In the exam
These questions describe production signals and ask where each should go, how to define an SLA, or what to do about a scheduled check. Use the loop and the four triage destinations, define SLAs with metric, threshold, breach and response, base thresholds on business needs, and keep calendar-based governance checks separate from metric alerts.
Check your understanding
All quality and latency metrics have been green for a quarter. A quarterly audit of outputs against the documentation standard is due. What should happen?
Show answer
Answer: B. Calendar-based checks are obligations, not reactions to metrics. Good metrics do not substitute for a required audit.
Which item belongs in an SLA definition but is missing from "the assistant must be fast"?
Show answer
Answer: A. An SLA is a commitment: what is measured, the limit, what counts as a breach and what happens then. "Fast" defines none of these.
Which TWO signals should go to architect review rather than internal monitoring?
Show answer
Answer: A, B. Signals that threaten agreed constraints or design assumptions need architect review. One-off noise and changes within limits stay with engineering monitoring.
Apply it: Write Corvane's governance table and SLA
Corvane's assistant is launching. Write the production-signal governance table and the chat SLA the support director will sign.
- List at least six signals with their trigger and triage destination (internal monitoring, architect review or stakeholder review).
Done when: at least one signal is calendar-based. - Write the chat SLA with metric, threshold and window, breach rule and breach response.
- Say where each threshold came from.
- Describe the review cadence with the support director.
Show model answer
1. Governance table
| Signal | Trigger | Destination |
|---|---|---|
| Chat 95th-percentile latency | Over 3 s for two consecutive hours | Stakeholder review (SLA breach); engineering acts at once |
| Chat 95th-percentile latency | Up 20% but under 3 s | Internal monitoring |
| Resolution rate on in-scope contacts | Down 3 points over two weeks | Architect review |
| Cost per resolved contact | Over $0.12 for a month, or peak-season projection over budget | Architect review, then Finance |
| Card data detected in model input | Any occurrence | Stakeholder review (security) the same day |
| CSAT on assistant-handled contacts | Below the human-handled baseline for a month | Stakeholder review |
| Quarterly review of 300 sampled conversations | Every quarter | Stakeholder review (calendar) |
| Help-centre restructuring | Any announced change | Architect review before it ships |
2. Chat SLA. Metric: 95th-percentile time from customer message to first word of the reply, measured at the chat gateway. Threshold: 3 seconds, per rolling hour, during service hours. Breach: two consecutive hours over threshold. Response: on-call engineer engaged within 15 minutes; support director notified within an hour; if not restored within four hours, chat routes to human agents with a status banner.
3. Sources. 3 seconds comes from the case's stated target; the resolution and cost thresholds from the success criteria in lesson 4.1 and the value case in lesson 1.6; the quarterly review from the support director's quality process.
4. Cadence. A monthly review with the support director: SLA performance, quality and cost trends, open issues, and any changes to scope or thresholds, recorded in writing; plus the quarterly sampled review.