What you need to know
The trade
| Synchronous Messages API | Message Batches API | |
|---|---|---|
| Cost | Standard | About 50% less |
| When you get results | Seconds | Within 24 hours; often much sooner, but no latency SLA |
| Tool calls mid-request | Yes, your loop runs tools between calls | No: a batch request cannot pause to run a tool and continue |
Match the API to the workflow
- Batch: overnight reports, weekly audits, nightly test generation, a backlog of historical documents due tomorrow. Nobody is waiting.
- Synchronous: anything that blocks a person or a process, such as a pre-merge check developers wait on, or a live chat.
custom_id
Each request in a batch carries a custom_id you choose (for example the document ID). Results are matched back to inputs by that ID, not by position.
requests = [{"custom_id": doc.id,
"params": {"model": MODEL, "max_tokens": 1024,
"messages": [{"role": "user", "content": doc.text}]}}
for doc in docs]
batch = client.messages.batches.create(requests=requests)Handle failures surgically
When 200 of 10,000 requests fail, identify them by custom_id, fix the cause, and resubmit only those. Fixes depend on the failure: chunk documents that exceeded the context limit; simply resubmit transient errors. Reprocessing all 10,000 doubles the cost for no gain.
Get the prompt right before the big run
Refine the prompt on a sample set first. A prompt flaw found after 30,000 documents means 30,000 resubmissions.
Plan submission frequency against an SLA
If results are promised within a deadline and a batch can take up to 24 hours, the worst case is wait before submission + 24 hours. To guarantee a 30-hour SLA, a document must never wait more than 6 hours to be submitted; submitting every 4 hours keeps the worst case at 28 hours, with margin.
In the exam
Expect a pair of workflows to place (one blocking, one overnight), a failure-handling question with oversized documents and timeouts, a submission-frequency calculation, or a cost question about a large backlog.
Check your understanding
A nightly debt report and a blocking pre-merge check both use the API. Which should move to batches?
Show answer
Answer: B. Batches have no latency guarantee, so they suit latency-tolerant work. A check that blocks merging must stay synchronous.
200 of 10,000 batch requests failed: some exceeded context limits, some timed out. What do you do?
Show answer
Answer: C. Resubmit only the failures, fixed for their cause: chunk documents that were too large and resubmit transient failures as they are.
Results must be delivered within 30 hours and batches can take up to 24. How often should documents be submitted?
Show answer
Answer: D. Worst case is the wait before submission plus 24 hours. Keeping the wait at or under 6 hours meets 30 hours; 4 hours gives a buffer.
Apply it: Bayview's monthly backlog
Bayview must process a backlog of 30,000 historical documents each month, needed by the next business day at the lowest cost. Plan the run.
- Choose the API and justify it against the requirements.
- Describe the pre-run step that protects the cost of the run.
- Define custom_id and the failure-handling procedure.
- Say which part of Bayview's work must stay synchronous.
Show model answer
1. API. Message Batches: about half the cost, and next-business-day delivery fits the up-to-24-hour window. No step needs tools mid-request.
2. Pre-run. Run the extraction prompt synchronously on a stratified sample of 300 documents (invoices, receipts, contracts, other), fix problems, and freeze the prompt before submitting 30,000.
3. custom_id and failures. custom_id = Bayview's document ID plus the client code (for example C114-DOC-883201). After the batch ends, collect failed IDs; split documents that exceeded the context limit into sections and resubmit; resubmit transient failures unchanged; send anything failing twice to review.
4. Synchronous. Same-day urgent invoices that clients flag for immediate payment, where someone is waiting on the result.