Cost
Rate limit
A rate limit caps how much you may send across all requests in a period, usually expressed as tokens per minute and requests per minute.
Also written: TPM, RPM, tokens per minute
It is unrelated to the context window, which limits the size of one request. You are bound by whichever quota you hit first, and for anything with substantial prompts that is normally tokens per minute.
Exceeding it produces a 429. The correct handling is exponential backoff with jitter — an immediate retry compounds the problem across every concurrent worker and turns a brief overage into a sustained outage.
In practice
The failure mode is rarely the limit itself, it is the retry. Twenty workers that all hit a 429 and all retry after exactly one second reproduce the same burst one second later, indefinitely. Exponential backoff with jitter is what breaks the synchronisation; without the jitter, backoff alone just moves the collision.
Common questions
How should I handle a 429 error?
Exponential backoff with jitter and a maximum attempt count. The jitter is the part people omit and the part that matters — without it, every worker retries at the same moment and reproduces the burst that caused the limit.
Do rate limits count input or output tokens?
Both, usually as separate limits alongside a requests-per-minute cap. A workload can be well under the request limit and still be throttled on tokens, which is why long prompts hit limits sooner than their request count suggests.