Quantization
Quantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality.
Weights are stored as floating-point numbers, usually 16-bit for a released open-weight model. Halving that to 8 bits halves the memory the model occupies and roughly halves the memory bandwidth each forward pass needs, which is the binding constraint during generation. The model gets cheaper and faster to serve without any change to its architecture.
The cost is accuracy. Rounding every weight to a coarser grid introduces error, and that error compounds across layers. Well-executed 8-bit quantization is usually close to indistinguishable from the original on general tasks. Aggressive 4-bit quantization is measurably worse, and the degradation shows up first on exactly the workloads people care most about: long-context reasoning, code generation and structured output.
The important point for a buyer is that precision is a serving decision, not a property of the model. The same model name can be served at several precisions by different providers, and the resulting endpoints behave differently while carrying identical model identifiers.
Nothing in a standard API response states the precision you were served at. Detecting it requires evaluating the endpoint against a task you care about, which is the only reliable method available from outside.
What it costs you
Quantization is where an unusually low per-token price usually comes from, and it is the variable a rate comparison silently omits. Two providers quoting the same model at different prices may not be selling the same thing, so the meaningful comparison is cost per acceptable answer rather than cost per token — a 4-bit endpoint that needs two attempts to produce valid structured output is more expensive than an 8-bit one that needs one, at any headline rate.
Related terms
- Inference providerAn inference provider is a company that runs open-weight model weights on its own or rented GPUs and sells access to them over an API, as distinct from the lab that trained and published the model.
- Open-weight modelAn open-weight model is one whose trained parameters are published under a licence permitting others to download, run and serve them, which is what makes it possible for anyone other than the original lab to offer that model as an API.
- Throughput (tokens per second)Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.