Cost
Throughput
Throughput is how much work you can push through per unit of time, bounded by your rate limits and your concurrency.
Sustaining a given requests-per-minute figure requires that rate multiplied by the per-request latency in concurrent workers. Six hundred requests per minute at four seconds each needs forty in flight.
That makes latency an infrastructure cost, not just a user-experience one: a slower model needs more concurrency for the same throughput.
In practice
Output tokens per second decides whether streaming feels instant. Above about 55 tokens per second the text arrives faster than most people read, so the wait collapses to time to first token. Below roughly 20 the user is watching the text form, and no amount of interface polish disguises it.
Common questions
How fast is fast enough for streaming?
Above roughly 55 tokens per second the text arrives faster than most people read, so the perceived wait collapses to time to first token. Below about 20 the user watches the text form.
Does throughput vary during the day?
Yes — shared infrastructure means peak hours are slower. If a latency target matters, measure at p95 across a full day rather than from a handful of requests made in the afternoon.