Reference
Glossary of LLM and inference terms
14 terms that decide what a language model API costs, defined in one sentence each. Every entry also says what the concept does to a bill, which is the part most glossaries leave out and the part that changes a decision.
Context window
Also called context length, context size, max context
A context window is the maximum number of tokens a model can hold in one request, counting the system prompt, the conversation history, any retrieved documents and the reply being generated together.
Embedding
Also called vector embedding, text embedding, embedding vector
An embedding is a fixed-length list of numbers representing a piece of text, produced by a model so that texts with similar meaning land close together in that vector space, which is what makes semantic search and retrieval possible.
Inference provider
Also called model host, inference API, model gateway, LLM provider
An inference provider is a company that runs open-weight model weights on its own or rented GPUs and sells access to them over an API, as distinct from the lab that trained and published the model.
Input vs output tokens
Also called prompt tokens, completion tokens, input tokens, output tokens
Input tokens are the text you send a model and output tokens are the text it writes back; they are billed at separate rates, and the output rate is almost always several times the input rate.
Mixture of experts (MoE)
Also called MoE, sparse model, active parameters, sparse mixture of experts
A mixture-of-experts model divides its parameters into many specialised sub-networks and routes each token through only a few of them, so the parameters actually used per token — the active parameters — are a small fraction of the model's total size.
Open-weight model
Also called open weights, open-source model, downloadable model
An open-weight model is one whose trained parameters are published under a licence permitting others to download, run and serve them, which is what makes it possible for anyone other than the original lab to offer that model as an API.
OpenAI-compatible API
Also called OpenAI-compatible endpoint, drop-in replacement, OpenAI SDK compatible
An OpenAI-compatible API is an endpoint that accepts the same request and response shapes as OpenAI's own API, so an existing client can be pointed at it by changing only the base URL, the API key and the model name.
Prompt caching
Also called cached input, prefix caching, cached tokens, KV cache reuse
Prompt caching is the reuse of an already-processed prompt prefix, so that repeated leading tokens — a system prompt, a tool schema, a fixed document — are billed at a reduced cached-input rate instead of the full input rate.
Quantization
Also called quantisation, INT8, FP8, 4-bit, GGUF
Quantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality.
Rate limit
Also called RPM, TPM, throttling, HTTP 429
A rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets.
Temperature
Also called sampling temperature, top_p, sampling parameters
Temperature is a sampling parameter that controls how much randomness a model uses when choosing each next token: at 0 it takes the most likely token every time, and higher values raise the chance of less likely tokens being picked.
Throughput (tokens per second)
Also called tokens per second, tok/s, decode speed, generation speed
Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Time to first token (TTFT)
Also called TTFT, first-token latency, prefill latency
Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
Token
Also called tokens, LLM token, text token
A token is the unit of text a language model reads and writes — roughly four characters of English, or about three quarters of a word — and it is also the unit that every language model API bills by.
Worked figures on these pages come from the published rate table and change with it. Nothing here is written down twice.