Throughput (tokens per second)
Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Two different figures travel under the same name. Per-stream throughput is what one request experiences and is what a reader watching text appear perceives. Aggregate throughput is what a provider's hardware produces across all concurrent requests, and it is the number that decides serving economics.
The two trade against each other. Batching more requests onto the same GPU raises aggregate throughput and lowers per-stream throughput, because each request now shares the hardware. A provider tuning for cost is tuning for the first; a provider tuning for perceived speed is tuning for the second. Both are defensible and they produce noticeably different endpoints.
Model properties matter too. Decode speed is bounded largely by memory bandwidth, so a model with fewer active parameters per token generates faster, and quantization raises throughput for the same reason it lowers cost.
Throughput figures are only comparable when measured under the same conditions — same prompt length, same output length, same concurrency. Quoted numbers with none of that stated are not measurements.
What it costs you
Throughput does not change what a given response costs, because billing is per token and not per second. Where it shows up is in your own infrastructure: a worker waiting on a slow stream is a worker you are paying for, so halved throughput doubles the concurrency needed to sustain the same request rate. On a batch job the entire cost of slow generation is on your side of the invoice, not the provider's.
Worked from today’s rates
A 2,000-token reply from GLM-5.2 costs $0.00510 in output whether it arrives in two seconds or twenty. Throughput changes how many of those you can complete per worker per hour, not what each one is billed.
Related terms
- Time to first token (TTFT)Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
- Rate limitA rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets.
- QuantizationQuantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.