What you need to know
Most configuration choices trade accuracy against latency (and usually cost). The architect's job is to know which dial moves which outcome, find the constraint that is actually tight, and defend each setting with numbers.
The dials
| Setting | Effect on accuracy | Effect on latency |
|---|---|---|
| Model tier | Larger tiers handle harder reasoning | Larger tiers are slower |
| Extended thinking and effort | Often better on multi-step problems | Adds thinking time and tokens |
| Prompt and context size | Relevant context helps; clutter hurts | More input takes longer to process |
| Prompt caching | No change | Faster on cache hits |
| Retrieval depth and reranking | More candidates and reranking find better passages | Each adds a step |
| Sequential versus parallel tool calls | No change | Independent calls in parallel finish sooner |
| Output length limits | Too short truncates answers | Shorter outputs finish sooner |
| Streaming | No change | Much faster perceived latency: the first words arrive early |
| Batch processing | No change | Much slower (hours), much cheaper; for work no one waits on |
Find the load-bearing setting
When a design has both a latency target and an accuracy target, one of them is usually binding. Ask: which setting, if relaxed, would break the tightest budget? That is the setting to protect, and the one to justify most carefully.
- If latency is binding: keep thinking low or off, use a faster tier where the evaluation allows, cache the stable prefix, run independent tools in parallel, stream the response, and skip reranking unless retrieval quality demands it.
- If accuracy is binding: spend latency where the evaluation shows a gain (a stronger tier, more thinking, reranking) and recover time elsewhere (caching, parallel calls, streaming, fewer retrieved chunks).
Perceived versus actual latency
Streaming does not make the full answer arrive sooner, but users see progress within a fraction of a second, which is what most chat experiences need. For a backend step whose output is parsed before anything is shown, streaming does not help; the full time is what counts.
Different routes, different settings
One application rarely needs one configuration. A chat route may use a small tier, no thinking and streaming; a nightly analysis job on the same data may use a larger tier, more thinking and batch pricing. Configure per route, not per application.
In the exam
These questions give targets for latency, accuracy and sometimes cost, and a configuration that misses one. Identify the binding constraint, then pick the change that moves it without breaking the others: thinking, tier, caching, parallel tools, streaming, reranking or batch. Answers that justify the change with a measurement on the right percentile are usually the strongest.
Check your understanding
A chat assistant meets its accuracy target, but users complain it feels slow. Full answers take 4 seconds. Which change improves the experience most cheaply?
Show answer
Answer: D. Users experience the time to the first words. Streaming cuts perceived latency without touching accuracy; the other options leave latency the same or make it worse.
An agent calls three independent APIs one after another, each taking about 800 ms. What reduces latency without affecting accuracy?
Show answer
Answer: C. Independent tool calls can run at the same time, saving over a second per turn with no change to what the model sees or decides.
A nightly job analyses 20,000 documents with no user waiting. Which TWO choices fit?
Show answer
Answer: A, D. With no one waiting, latency is not binding. Batch pricing cuts cost, and the time budget can go to whatever improves accuracy.
Apply it: Justify Corvane's chat configuration
Corvane's chat route must reply within 3 seconds at the 95th percentile and stay within its accuracy threshold. A test at peak load shows: balanced tier, extended thinking on, no caching, order and returns tools called one after the other, hybrid retrieval with reranking on every product question, no streaming. Results: 95th-percentile latency 6.1 seconds; accuracy 2 points above the threshold.
- Name the binding constraint and explain how you know.
- List the changes you would make, in order, with the effect you expect from each.
Done when: each change says whether it affects accuracy, latency or both. - Say what you would measure before accepting the new configuration.
Show model answer
1. Binding constraint. Latency: it misses by more than 3 seconds, while accuracy has a 2-point margin. Any change may spend at most that margin.
2. Changes, in order
| Change | Expected effect |
|---|---|
| Cache the 6,000-token stable prefix | Faster on every request; no accuracy change |
| Call the order and returns tools in parallel | Removes one round trip on returns contacts; no accuracy change |
| Stream the reply | First words under a second; full time unchanged |
| Evaluate with thinking off | Likely the largest saving; accept only if accuracy stays above threshold |
| Rerank only when the top hybrid results are close in score, or rerank fewer candidates | Saves a step on most product questions; check retrieval recall |
| Small tier for order-status phrasing only | Faster on the highest-volume route; facts come from the tool, so risk is low |
3. Before accepting. A peak-load test reporting 95th-percentile total latency and time to first words, the full accuracy evaluation with the product-question slice reported separately, and retrieval recall with the new reranking rule. Accept only if latency is under 3 seconds and accuracy is still above threshold.