Rate limits, concurrency and backoff

The numbers

Five endpoints are rate limited, each on its own counter, each on a 60-second window. The daily column is arithmetic rather than a quota -- there is no daily cap, only the per-minute one sustained.

Five buckets, not one pool

The counter key is the endpoint name joined to your API key, which means the five limits above do not share a budget. Exhausting your chat allowance has no effect on embeddings, and a video batch cannot starve an agent loop.

The aggregate that follows is 490 requests per minute on a single key, and that is a real number rather than a theoretical one. A pipeline that embeds a document, calls a model about it and generates a clip is spending from three separate allowances at once.

It also means the thing to measure when you are being limited is which endpoint, not how much traffic. A retry storm concentrated on chat while embeddings sits idle is a queueing problem in your code, not a capacity problem in ours.

The window is fixed, which means 120 requests in two seconds is legal

The limiter is a fixed window, not a sliding one and not a token bucket. The first request of a window stamps a reset time 60 seconds out; the window does not slide as requests arrive. Everything resets at once when it expires.

The consequence is the classic fixed-window boundary effect, and it is worth publishing rather than leaving to be discovered. Sixty requests in the last second of one window and sixty in the first second of the next are both within the limit, so 120 requests can legitimately land inside a two-second span. If you are sizing a worker pool against these numbers, size it against the burst, not the average.

It cuts the other way too. A client that fires 60 requests instantly at the start of a window is then blocked for the remaining 59 seconds, even though its average rate over the minute is exactly at the limit. Pacing requests evenly is not politeness here; it is how you avoid a self-inflicted 429.

Two limiters sit in the path, and they count different things

The per-endpoint counters above are enforced at this API's edge, keyed on your API key. Behind it, the gateway that meters inference applies its own limit, keyed on the caller's IP address. They are not the same limiter and they do not share state.

One detail of that second limiter is worth knowing if you run behind a proxy. The relay strips whatever x-forwarded-for arrived on the inbound request and re-sets it from the connection, rather than passing the header through. That is not header hygiene: the gateway rate-limits on that address, so forwarding a caller-supplied value verbatim would let anyone forge the origin the limiter counts against.

The practical reading is that the published numbers are the contract you should build against, and the address the gateway sees is the connection's, not one you can influence.

There are two kinds of 429, and the code is the only difference

Running out of credit returns 429 here, not 402. That is a deliberate compatibility decision -- the platform this API is interchangeable with returns 429 for an exhausted balance, and a client that special-cases 402 would stop noticing it had run out. The cost of honouring that contract is that the status no longer identifies its own cause.

So the cause moves into error.code, and every 429 this API emits carries one of exactly two values. A 429 with a null code is a bug on our side, not a case for you to handle. Branching on the status alone is the failure this contract exists to prevent: it produces a client that reads an exhausted balance as a rate limit and backs off forever, waiting for a condition only a payment can clear.

try:
    response = client.chat.completions.create(...)
except openai.RateLimitError as e:
    code = e.body["error"]["code"]
    if code == "insufficient_credits":
        alert_billing()          # no amount of waiting clears this
    else:
        reset = int(e.response.headers["x-ratelimit-reset"])   # unix seconds
        sleep(max(0, reset - time.time()) + random.uniform(0, 1))
        retry()

The headers, on every response rather than only on failures

Three headers accompany every rate-limited endpoint's response, success and failure alike. That matters: a client that only reads them on a 429 finds out it is near the limit exactly one request too late.

x-ratelimit-reset is a Unix timestamp in seconds, not a duration. Backing off by a fixed number of seconds instead of sleeping until that timestamp is the most common backoff bug we would expect to see, because it works in testing and drifts under load.

The 429 body carries the same information in prose, and deliberately so -- it names the limit, the window and an ISO timestamp to retry after, so a developer reading a single log line does not have to reconstruct which of the five endpoints was throttled.

Concurrency is not limited. Request rate is

There is no cap on in-flight requests, no per-account connection ceiling, and no queueing tier you can buy into. What is bounded is how many requests you start per minute, which is a different constraint and shapes a client differently.

For a long-running workload that distinction is the whole design. Sixty requests a minute on chat means 60 concurrent completions is fine and 61 starts a minute is not, so a worker pool should be sized by start rate rather than by pool depth. A semaphore on concurrency alone will still trip the limiter.

One request-level ceiling does exist and is separate from all of this: embeddings accepts at most 2,048 inputs per call, which is the upstream contract's own ceiling. Batching past it is a client bug rather than a large request, and the error says so with the count you sent.

What the limits mean for three real workloads

The limits only become concrete when you divide a job by them, and in every case below the answer is that the job is a scheduling problem before it is a budget problem.

What this limiter is not, stated plainly

The edge limiter is in-process. Each serving instance keeps its own counter, and the platform runs as many instances as traffic demands, so the numbers above bound one client against one instance rather than acting as a single global ceiling. In practice that means you may observe more than 60 requests a minute succeeding.

Do not build on that. It is a known gap with a known fix -- shared state -- and the published numbers are the contract; the surplus is an artefact. A client designed around the surplus will break on the day the limiter becomes global, and it will break under exactly the load that made it worth building.

Two smaller behaviours round out the surface. An unsupported method returns this API's own 405 with an allow header rather than the framework's generic answer, so an SDK that only parses the error envelope never receives something else because you used the wrong verb. And a CORS preflight succeeds even on an unknown path, deliberately: a 404 answered behind a failed preflight reaches the browser as an opaque CORS error, which hides the actual fixable problem, which is that the URL is wrong.