Batch inference
Batch inference is the processing of many independent inputs as one job rather than one interactive request at a time — either by packing several inputs into a single API call where the endpoint accepts them, or by running a large queue of calls concurrently with nobody waiting on any individual answer.
Two different things share the name. Server-side batching is the provider packing concurrent requests onto the same GPU to raise utilisation, and it is invisible to you except in the latency it costs. Client-side batching is the part you control, and it means either more inputs per request or more requests in flight.
Some endpoints genuinely take a list. Embeddings do: this API accepts up to 2,048 inputs in a single call, which is the same ceiling the contract it implements sets, and that turns two thousand round trips into one. Chat completions do not — one conversation per call — so batching there means concurrency, bounded by the rate limit and by how many workers you are prepared to run.
Concurrency runs into the token-per-minute limit before the request-per-minute one on almost any bulk job, because bulk jobs send long prompts. That is the limit to size against, and the one people diagnose last.
What batching does not buy here is a discount. There is one published rate per model and it is the same whether a request is interactive or the four-millionth item in an overnight queue. Some providers offer a cheaper asynchronous tier; we do not, and pretending otherwise would be the easiest thing on this page to get wrong.
What it costs you
Batching moves cost off your side of the ledger rather than off the rate. The tokens are identical, so what you recover is round trips, worker time and wall-clock — real money, but yours. The corollary is the useful part: with no batch tier to wait for, a job that could run at three in the morning has no cheaper hour to run in, and the only levers left on a bulk workload are the ones that always work — a smaller model, a shorter prompt, and a cap on the output.
Worked from today’s rates
Embedding 500,000 documents of 400 tokens each on Qwen3 Embedding 8B is 200M input tokens — $6.00 at $0.030/M. Sent 2,048 to a call, that is 245 requests instead of 500,000, and the charge is identical either way: batching buys wall-clock, not a discount.
Related terms
- Rate limitA rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets.
- EmbeddingAn embedding is a fixed-length list of numbers representing a piece of text, produced by a model so that texts with similar meaning land close together in that vector space, which is what makes semantic search and retrieval possible.
- Throughput (tokens per second)Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.