Comparison
Streaming vs batch processing
Streaming improves perceived speed; batch halves the bill. They are not alternatives — they serve different traffic, and most teams under-use one of them.
The short answer
These are not competing options, they are different traffic. Stream anything a user is watching — it costs nothing extra and improves perceived latency five to ten times. Batch anything nobody is waiting on, for a discount that is commonly around fifty percent. Most teams stream correctly and batch far less than they could.
At a glance
| Streaming | Batch | |
|---|---|---|
| Latency | First token in under a second | Hours |
| Cost | Standard rate | Around 50% off |
| Rate limit quota | Interactive quota | Usually separate |
| Implementation | Handle a token stream | Submit and poll |
| Right for | Anything user-facing | Anything not user-facing |
| Quality difference | None | None |
When to choose which
Choose Streaming when
- A person is waitingChat, assistants, anything where text appears on a screen. There is no reason not to stream, and the perceived latency improvement is large for no cost.
- Responses are longA two-thousand token answer takes tens of seconds to generate. Without streaming that is a blank screen; with it the user is reading before generation finishes.
- You are hitting timeoutsStreaming keeps the connection active and removes most timeout exposure on long generations.
Choose Batch when
- Nobody is waiting for the responseNightly classification, backfills, evaluation runs, embedding generation, enrichment pipelines, summarising yesterday’s tickets. The list is longer than most teams assume.
- You are near your rate limitBatch usually sits outside the interactive quota, so moving work there frees throughput as well as money.
- Volume is high and margin mattersFifty percent off a large recurring bill, with no quality trade-off, is the cleanest saving available anywhere in this domain.
What it costs either way
Streaming costs nothing extra. It is the same request with the same billing, delivered differently.
Batch is commonly discounted around fifty percent on both input and output, and stacks with prompt caching where the provider supports both.
The barrier to batching is engineering, not economics: it needs a different code path with submission and polling rather than a synchronous call. That is why it is skipped, not because the trade-off is unclear.
For most products the batchable share is well above half of all requests, because interactive traffic is a smaller fraction of total volume than it feels like.
The mistake people make
Assuming everything is interactive
Teams look at their traffic, see that the product is a chat interface, and conclude that nothing can be batched. Then they count and find that the majority of requests are classification, enrichment and evaluation running in the background with nobody watching. Auditing which requests actually block a user is usually an afternoon and frequently halves a bill.
How to decide
- 1Stream everything user-facing. There is no argument against it.
- 2List every request type and mark which ones block a user. Be strict — "we might want it fast one day" is not blocking.
- 3Move the unblocked ones to batch, starting with the highest volume.
- 4Check whether batch sits outside your rate limit quota; if it does, that is a second benefit worth counting.
Price it yourself
Frequently asked questions
- Does batch produce different output?
- No. Same model, same parameters, same output distribution. You are buying a discount by giving up immediacy, not quality.
- How long does batch actually take?
- Providers quote a completion window measured in hours, often up to twenty-four. Design for the worst case rather than the observed average — the guarantee is what you are buying.