What you need to know
When a Claude-powered system misbehaves, the architect is often the person who knows why. The fast response is to fix it yourself. The lasting response is to fix it with the team and leave behind what they need to fix the next one alone.
From symptom to cause
Most operational symptoms trace back to a small set of architectural causes. Writing the mapping down is the start of a runbook.
| Symptom | Likely causes | First action |
|---|---|---|
| Quality declined gradually with no code change | Model version change, prompt change, retrieval drift as content grew or changed | Run the evaluation set; check model, prompt and corpus change history |
| Latency spiked | Context grew, a tool slowed down, the cache stopped hitting | Read traces for the slowest step; check tokens per request and cache hit rate |
| Intermittent tool failures | Expired credentials, rate limits, unhandled error paths | Trace one failed call end to end; check auth and limits for that tool |
| Cost rose without more usage | Larger tier in use, cache regression, growing context, retry loops | Compare model mix, cache hits and tokens per request with the budget model |
| Requests rejected or outputs cut off | Context too large; output limit too low | Check the error type and stop reasons; count tokens |
| Agent loops or never finishes | Missing stopping rule, a tool that keeps failing quietly | Read the trace; check turn limits and tool error handling |
Runbooks
A runbook entry records: the symptom as the on-call engineer will see it, how to confirm it, the likely causes in order, the first action for each, how to verify the fix, and when to escalate. Good entries are written after real incidents, tested by someone other than the author, and updated when the system changes.
Escalation paths
People need to know where their responsibility ends. An escalation path names who handles what (first-line on-call, the team owning the AI service, the architect, the platform or vendor) and the conditions for moving up: severity, duration, customer impact, safety or compliance implications. Without it, issues either sit with someone who cannot fix them or go straight to the busiest person.
Support Claude Code users too
Engineering teams using Claude Code have their own recurring issues: configuration not loading where expected, permissions blocking a needed command, an MCP server failing to connect, context filling up in long sessions. The same approach applies: a short internal troubleshooting page, champions as first line, and a clear route to the platform team.
In the exam
Support questions describe a symptom and ask for the likely cause, the first action, or how to stop the same problem recurring. Use the symptom-to-cause mapping, start with evidence (traces, change history, evaluation runs), and prefer answers that leave behind a runbook entry and a clear escalation path.
Check your understanding
An assistant's answer quality has slowly declined over two months. There were no code changes. What is the best first action?
Show answer
Answer: D. Gradual decline without code changes points to model, prompt or retrieval drift. Measuring against the evaluation set and checking each history finds which.
Costs rose 50% this month with flat usage. Which TWO checks come first?
Show answer
Answer: A, D. Rising cost without rising usage most often comes from cache regressions or a shift to more expensive models. Both show in usage and cost data.
What makes a team more self-sufficient after an incident the architect helped resolve?
Show answer
Answer: C. A runbook entry and a clear escalation path turn one fix into a repeatable capability for the team.
Apply it: Write Corvane's assistant runbook
Corvane's assistant is live, and the e-commerce engineering on-call rota will support it. Write the start of their runbook.
- Write four runbook entries for symptoms likely in this system, each with confirmation steps, likely causes, first action, verification and escalation condition.
Done when: at least one entry covers a slow quality decline and one a cost rise. - Write the escalation path: levels, who, and when to move up.
- Name the routine checks that would catch problems before they become incidents.
Show model answer
1. Runbook entries
| Symptom | Confirm | Likely causes | First action | Verify | Escalate when |
|---|---|---|---|---|---|
| Product answers getting worse over weeks | Sampled grading trend; escalations on product contacts | Help-centre or manual changes not re-indexed well; prompt or model change | Check content publish log and re-index status; run retrieval recall set | Recall and graded accuracy back to baseline | Not restored in 2 days, or a change of approach is needed |
| Chat latency over 3 s | Gateway 95th percentile; traces | Cache misses, slow Order Management System calls, larger contexts | Read slowest spans; check cache hit rate and OMS latency | 95th percentile under 3 s for an hour | SLA breached for two hours |
| Cost per contact up 30% | Usage and cost data by route | Cache regression, model mix change, retry loops, longer conversations | Compare cache hits, model mix and tokens per contact with last month | Cost per contact back within budget model | Projected month over budget |
| Order-status tool errors | Tool error rate; one traced failure | Expired credentials, OMS rate limits, API change | Trace one failure; check credential expiry and limits | Error rate back to baseline | OMS-side fault, or over 15 minutes of customer impact |
2. Escalation path. Level 1: on-call engineer, using the runbook. Level 2: the AI service team lead, for anything not covered or not resolved within the runbook's limits. Level 3: the architect, for design-level causes or changes to agreed thresholds. Immediately to security and the support director: any card data in model inputs, any cross-customer data exposure.
3. Routine checks. Weekly retrieval-recall run; weekly cost per contact against the budget model; monthly sampled quality review; a check after every help-centre restructuring; a model-version and deprecation review each quarter.