What you need to know
Optimization is the Domain 4 counterpart to design: the system works, and now it must cost less, respond faster or both, without falling below the quality bar. Every optimization is a change, so it goes through the same evaluation gate as any other.
Step 1: find where it goes
Before changing anything, break cost and latency down:
- by route (which features cost the most?),
- by component (system prompt, tool definitions, retrieved context, conversation history, tool results, output, thinking),
- by distribution (is the bill driven by typical requests or by a long tail of huge ones?).
Agents deserve special attention: context that grows turn by turn means the later turns cost far more than the early ones, and a few long sessions can dominate the bill.
Step 2: apply levers in order of value
| Lever | Saves | Quality risk |
|---|---|---|
| Remove what is not needed (unused tools, repeated instructions, whole documents where a section would do) | Tokens, latency | Low |
| Cache stable prefixes | Input cost and latency | None |
| Trim and summarize tool results; clear stale ones; compact long sessions | Tokens in long sessions | Low, if identifiers are preserved |
| Limit output length; ask for structure instead of prose | Output tokens, latency | Low |
| Right-size thinking (effort or budget) | Output tokens, latency | Evaluate |
| Right-size the model per route or step (route easy work to a smaller tier) | Cost, latency | Evaluate |
| Batch work that can wait | About half the cost | None, but slower |
| Parallelize independent calls | Latency | None |
The first rows are nearly free; the later ones need evaluation evidence.
Step 3: verify and keep it
Run the evaluation suite before and after, compare by slice, and watch the production metrics after rollout. Add a budget alert so a later change that undoes the saving is noticed.
In the exam
Optimization questions give a cost or latency problem and several changes. Prefer the change that targets the measured driver with the least quality risk (trimming, caching, managing agent context, output limits), then evidence-backed model and thinking changes, then batch for work that can wait. Reject options that skip evaluation, remove safety controls, or optimize a part of the bill that is not the problem.
Check your understanding
An agent's cost per task has tripled since it started using a web search tool that returns full pages. What is the first fix?
Show answer
Answer: B. The driver is the volume of tool output carried in context. Trimming and clearing it targets the cause without reducing capability.
A team cut cost per call by 40% by switching tiers, but escalations to staff rose by 25%. How should the change be judged?
Show answer
Answer: A. The business cost includes escalations. The change may cost more overall even though each call is cheaper.
A nightly job classifies 50,000 documents with no user waiting. Which TWO levers fit?
Show answer
Answer: A, B. Work that can wait suits batch pricing, and a shared instruction prefix suits caching. Streaming is irrelevant without a user, and thinking adds cost unless evaluation shows a need.
Apply it: Cut the cost of Quillstone's incident agent
Quillstone's incident-diagnosis agent runs about 300 investigations a month on the balanced tier (Tier M prices). An average investigation takes 30 turns. Every log query result stays in the history, so input grows from about 10,000 tokens on the first turn to about 120,000 on the last, averaging 65,000 per turn. Output averages 800 tokens per turn. The first 10,000 tokens (instructions and tool definitions) never change.
- Estimate the current monthly cost.
- Choose three optimizations in order, using the lever table.
- Estimate the new cost, assuming the average input per turn falls to 20,000 tokens of which 10,000 are a cached prefix.
Done when: you have a before and after figure per investigation and per month. - Say what must be evaluated before rollout.
Show model answer
1. Current cost. Input: 30 turns × 65,000 = 1,950,000 tokens × $3 per million = $5.85. Output: 30 × 800 = 24,000 × $15 per million = $0.36. About $6.21 per investigation, × 300 = about $1,860 a month.
2. Optimizations, in order
- Cap log query results: return matching lines and counts, not raw dumps.
- Clear tool results after the agent has written its findings into a short structured notes record, and compact the history if a session runs long.
- Cache the stable 10,000-token prefix.
3. New cost. Per turn: 10,000 cached tokens at $0.30 per million ($0.003) plus 10,000 uncached at $3 per million ($0.03) = $0.033. × 30 turns = $0.99 input. Output unchanged at $0.36. About $1.35 per investigation, × 300 = about $405 a month, a saving of about 78%.
4. Evaluate before rollout. On a set of past incidents with known root causes: diagnosis accuracy must not fall, the agent must still cite the log lines behind its conclusion (checking the notes keep identifiers such as request IDs and timestamps), and the number of turns must not rise enough to erase the saving.