What you need to know
When an AI system misbehaves, the tempting fix is the biggest lever: a new model, a longer prompt. The exam rewards the opposite habit: classify the failure first, then apply the smallest fix that addresses its cause.
A failure taxonomy
| Failure | What you see | Typical cause | First action |
|---|---|---|---|
| Prompt failure | Inconsistent format or behaviour; the model does something reasonable but not what was wanted | Ambiguous or missing instructions | Make the instruction specific; add an output contract or examples |
| Hallucination | Fluent, confident, unsupported facts (prices, dates, policy terms, citations) | The model was not given the facts, or not told to rely only on them | Ground it: tool call or retrieval for the facts; allow "I don't know"; require citations |
| Retrieval failure | Answers based on the wrong passage, or "not found" when the answer exists | Chunking, index or filter problems | Inspect what was retrieved; fix chunking, filters or index type |
| Model mismatch | Consistently shallow or incomplete work on hard cases; fine on easy ones | Tier too small for the task (or too slow or costly for it) | Evaluate another tier on the failing slice |
| Context problem | Quality degrades late in long sessions; truncated outputs | Cluttered or overflowing context | Trim, compact, or split the work (lesson 2.4) |
| Orchestration failure | Incomplete reports, missing pieces, silent gaps in multi-agent output | A subagent failed and synthesis carried on | Trace across agents; validate results; reconcile coverage |
| Integration failure | Errors, timeouts, empty tool results treated as answers | Downstream outage, bad tool error handling | Retries, clear tool errors, fallbacks |
Diagnose from evidence
- Read the trace, not the summary: what was in the context, which tools ran, what they returned, what the stop reason was. A response cut off at the output limit is not a reasoning problem.
- Find the slice. Does it fail on all inputs or a category (long documents, one product line, one language)? A slice-specific failure points to data, retrieval or model fit; a universal one points to the prompt or configuration.
- Check what changed. Prompt version, model version, index refresh, tool change. Most sudden regressions follow a change.
- Reproduce, then fix, then add a test. Every diagnosed failure becomes a case in the golden dataset.
In the exam
Diagnosis questions describe a symptom, sometimes with log details, and offer fixes. Map the symptom to a failure type using the evidence (what was retrieved, the stop reason, which slice fails, what changed recently), then pick the first action that addresses that cause. The model change is rarely the first answer.
Check your understanding
An assistant states a product warranty of 5 years; the real warranty is 2. Logs show no warranty passage was retrieved for the question. What type of failure is this, and what is the first fix?
Show answer
Answer: B. The model filled a gap it should not have had. Getting the right passage into context, and permitting an honest "not found", addresses the cause.
Reports from a multi-agent research system sometimes omit whole topics, with no error raised. What should be checked first?
Show answer
Answer: A. Silent gaps in multi-agent output usually mean a subagent failed and synthesis carried on. Tracing and coverage reconciliation find and prevent that.
Summaries end mid-sentence on long documents. Which TWO checks come first?
Show answer
Answer: B, C. Truncation is usually an output limit or a full context window, both visible in the stop reason and token usage. It is not a prompting or retrieval issue.
Apply it: Triage Larkfield's first month
Larkfield's intake system has been live for a month. The operations team reports four problems:
- (a) Some acknowledgements tell customers their excess is $500 when their policy says $1,000.
- (b) About one claim in fifty is created with no date of loss, although the email clearly gives one.
- (c) After a prompt update last Tuesday, acknowledgements became noticeably longer and more formal.
- (d) Extraction quality is fine on emails and web forms but poor on scanned post.
- Classify each problem using the taxonomy, with the evidence you would look for to confirm it.
- Give the first action for each.
Done when: no first action is "switch to a bigger model" without evidence. - Say how each fix is protected against coming back.
Show model answer
| Problem | Classification | Evidence to check | First action |
|---|---|---|---|
| (a) Wrong excess | Hallucination: excess is not in the claim, so the model filled it in | Trace shows no excess value from Policy Admin in the context | Remove excess from the drafted text, or insert it from Policy Admin by code |
| (b) Missing date of loss | Prompt or schema failure, possibly an extraction edge case | Compare the failing emails: are dates written in words, or relative ("last Friday")? | Specify date handling and relative dates in the prompt with examples; make the field required in the schema and route failures to a handler |
| (c) Longer, formal acknowledgements | Prompt failure from a change | Prompt version in the logs changes on Tuesday | Roll back or fix the prompt; add tone and length checks to the evaluation |
| (d) Poor on scanned post | Input-quality or model-fit problem on one slice | Compare OCR output to the scans; check confidence by channel | Improve OCR first; if the text is good and extraction still fails, evaluate a stronger tier for the scanned slice only |
Protection. Each failing case goes into the golden dataset under its own slice (policy facts in drafts, relative dates, tone, scanned post). Code checks block any acknowledgement that contains a policy figure not supplied by Policy Admin, and the evaluation runs on every prompt change before release.