| Property | Value |
|---|---|
| Cost | ~50% savings |
| Window | Up to 24 hours |
| Latency SLA | None guaranteed |
| Multi-turn tool calling in one request | Not supported |
| Correlation | custom_id per request/response pair |
Batch for non-blocking, latency-tolerant work — overnight reports, weekly audits, nightly test generation. Synchronous for blocking workflows — anything a person is actively waiting on, like a pre-merge check.
The arithmetic most candidates get wrong by only computing half of it: worst case = time until the next submission window plus up to 24 hours of processing. Submitting once a day means a 48-hour worst case. Submitting every 4 hours caps the worst case at 28 hours. Most candidates compute only the 24-hour processing term and miss the wait-for-window term entirely.
On failures: resubmit only the failed documents by custom_id, with any needed modification (e.g., chunking a document that exceeded a context limit) — not the whole batch. And refine prompts against a small sample set before batching a large volume; first-pass success rate is the dominant lever on total cost, since failed items still cost something and still need a resubmission round-trip.
The discriminator: is a human waiting? Blocking → synchronous, regardless of the cost savings on the table. Not blocking → batch is close to free money.
