Mixture of experts (MoE)
A mixture-of-experts model divides its parameters into many specialised sub-networks and routes each token through only a few of them, so the parameters actually used per token — the active parameters — are a small fraction of the model's total size.
A dense model puts every token through every parameter. An MoE model puts each token through a router that selects a handful of experts, so compute per token scales with the active parameter count rather than the total. That is how a very large model can generate at the speed of a much smaller one.
The naming convention encodes both numbers. A model described as 235B A22B has around 235 billion total parameters and around 22 billion active per token. Total size sets the memory the weights occupy; active size sets the compute and memory bandwidth each token costs.
Those two numbers pull serving economics in opposite directions. All the weights must be resident even though only a fraction is used per token, so an MoE model is memory-hungry to host and cheap to run once hosted. That favours providers operating at enough scale to keep the weights loaded and busy.
Most of the strongest open-weight models released recently use this architecture, so it is now the common case rather than an exotic variant.
What it costs you
Serving cost tracks active parameters and memory footprint, not the headline parameter count, which is why a model with a very large total size is not automatically the expensive one on a rate card. Reading a price purely off a parameter count will mislead in both directions — and it also explains why per-token prices for large open-weight models have fallen faster than model sizes have grown.
Related terms
- QuantizationQuantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality.
- Open-weight modelAn open-weight model is one whose trained parameters are published under a licence permitting others to download, run and serve them, which is what makes it possible for anyone other than the original lab to offer that model as an API.
- Throughput (tokens per second)Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.