What you need to know
When a Claude-powered feature gives a wrong answer at 2 a.m., someone has to find out which model, prompt, inputs, tools and user produced it. At scale, with many routes, tenants and agents, that is impossible without deliberate instrumentation.
Log four kinds of information per call
| Category | Fields |
|---|---|
| Request | Model and version, prompt template version, input tokens, which tools were offered |
| Response | Output tokens, latency (total and time to first token), stop reason, tool calls made |
| Context | Trace ID, user role (not raw identity where avoidable), tenant, session, whether the cache was hit |
| Outcome | Did the downstream system accept the result? Did the user escalate, correct or rate it? |
Outcome is the field most often missing, and the one that connects technical behaviour to business results.
Four layers of observability
- Request tracing. One trace ID follows a request from the gateway through retrieval, every model call, every tool call and every subagent. Standard tracing tools (such as OpenTelemetry) let agent traces join the application's existing traces.
- Metric aggregation. Latency percentiles, error rates, token use, cost, cache hit rate and outcome rates, per route and per tenant.
- Anomaly detection. Alerts on changes from the baseline: a jump in escalations, a fall in cache hits, a cost spike for one tenant, a rise in a particular stop reason.
- Change attribution. Every deploy, prompt version, model change and configuration change is recorded, so a shift in a metric can be tied to its cause.
Attribution and isolation
At scale, you need to know who is using capacity and money, and to stop one consumer from starving others.
- Separate keys or workspaces per team, product or tenant let usage, cost and rate limits be tracked and capped separately. One shared key hides who caused a spike, and one heavy tenant can exhaust a limit everyone shares.
- Usage and cost reporting by key or workspace feeds chargeback and budget alerts.
- Per-tenant dashboards and limits in your own application add a second layer for products that serve many customers.
Privacy in the logs
Logs and traces can become the biggest store of sensitive data in the system. Decide what is recorded: redact or hash personal data, avoid full prompt and response bodies where metadata is enough, restrict access, and set retention to match policy. Trace exporters usually let you control whether message content is included.
In the exam
Observability questions describe an incident that could not be diagnosed, a cost or rate-limit problem that could not be attributed, or a multi-agent run whose failure could not be traced. The answer adds the missing piece: trace IDs across components, versioned prompts and models in logs, outcome signals, per-tenant keys or workspaces, or anomaly alerts. Watch for answers that log everything verbatim without regard to privacy.
Check your understanding
A shared API key serves 40 business customers. One customer's traffic spike causes rate-limit errors for all of them. What design prevents this?
Show answer
Answer: B. Separate keys or workspaces isolate tenants' limits and make usage attributable. A higher shared limit only delays the same problem.
Quality dropped on Tuesday afternoon. Logs show model, tokens and latency, but not which prompt version served each request. What is missing?
Show answer
Answer: A. To link a quality change to its cause, logs must record which prompt and configuration version handled each request, alongside deploy history.
Which TWO signals catch a fluent but wrong answer that raised no error?
Show answer
Answer: B, C. Outcome signals reveal silent failures. A wrong answer usually returns a successful status and normal token counts.
Apply it: Instrument Quillstone's in-product assistant
Quillstone's in-product assistant will serve 60 banks, from 40 users to 4,000. Each bank must only see its own data, AI usage is billed back per bank, and the assistant calls Claude, two internal services and sometimes a subagent for long reports.
- List the fields every trace and log line must carry.
- Design attribution and isolation across the 60 tenants.
Why: chargeback and fairness both depend on it. - Pick four metrics with alert conditions.
- Set the privacy rules for what is logged and for how long.
Show model answer
1. Fields. Trace ID (shared with the subagent and the internal services), tenant ID, user role, route, model and version, prompt template version, input and output tokens, cache hit, latency and time to first token, stop reason, tools called, and outcome (accepted, edited by the user, rejected, escalated).
2. Attribution and isolation. One workspace and API key per bank, with a spending limit and rate limit sized to its contract; the provider's usage and cost reports by workspace feed monthly chargeback. Inside the product, per-bank request quotas protect the platform from a runaway client, and every data-access tool is scoped to the calling bank's tenant ID on the server.
3. Metrics and alerts
| Metric | Alert when |
|---|---|
| 95th-percentile latency per route | Above target for 15 minutes |
| Cost per bank per day | Above 150% of its 30-day average |
| Rate-limit errors per bank | Any sustained occurrence |
| Rejected or edited outputs | Up 5 points on the 7-day baseline, especially just after a prompt or model change |
4. Privacy. No full prompts or responses by default; log metadata and hashed identifiers. A sampled, redacted content store for quality review, access-controlled and kept for 30 days. Trace exports configured to exclude message content. Retention for operational logs set by Quillstone's SOC 2 policy, and nothing from one bank visible to another bank's support staff.