Rate limit
A rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets.
Limits are usually expressed two ways at once: requests per minute, and tokens per minute. A workload can be well under the request limit and still be throttled on tokens, which is what happens to applications sending long prompts, and it is the version people diagnose last.
The correct client response is exponential backoff with jitter. Retrying immediately makes the condition worse, and retrying on a fixed interval synchronises every client in a fleet onto the same moment, which produces a self-inflicted thundering herd at each window boundary.
The trap specific to this class of API is that 429 is overloaded. On this service both an exhausted balance and a genuine rate limit return 429, distinguished only by error.code: insufficient_credits means top up, rate_limit_exceeded means back off. A client that retries every 429 with backoff will retry forever on the first, because no amount of waiting adds credit.
Concurrency is a related but separate constraint. A limit on simultaneous in-flight requests caps throughput independently of any per-minute figure, and it is the one that binds first on batch workloads.
What it costs you
A rate limit never appears on an invoice, but it decides how much work you can actually buy in an hour, which makes it a capacity constraint disguised as an error code. The real cost is engineering: retry logic that mishandles 429 turns a recoverable condition into dropped requests, and a batch job tuned to the request limit while the token limit is the binding one runs at a fraction of the throughput it was sized for.
Related terms
- OpenAI-compatible APIAn OpenAI-compatible API is an endpoint that accepts the same request and response shapes as OpenAI's own API, so an existing client can be pointed at it by changing only the base URL, the API key and the model name.
- Throughput (tokens per second)Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
- Time to first token (TTFT)Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.