Reference
Glossary of LLM and inference terms
26 terms that decide what a language model API costs, defined in one sentence each. Every entry also says what the concept does to a bill, which is the part most glossaries leave out and the part that changes a decision.
Batch inference
Also called batching, bulk inference, offline inference, batch API
Batch inference is the processing of many independent inputs as one job rather than one interactive request at a time — either by packing several inputs into a single API call where the endpoint accepts them, or by running a large queue of calls concurrently with nobody waiting on any individual answer.
Context window
Also called context length, context size, max context
A context window is the maximum number of tokens a model can hold in one request, counting the system prompt, the conversation history, any retrieved documents and the reply being generated together.
Distillation
Also called knowledge distillation, teacher-student training, distilled model
Distillation is the training of a small model to reproduce the outputs of a larger one, so that the smaller student performs close to the teacher across a particular range of work while costing a fraction as much to serve.
Embedding
Also called vector embedding, text embedding, embedding vector
An embedding is a fixed-length list of numbers representing a piece of text, produced by a model so that texts with similar meaning land close together in that vector space, which is what makes semantic search and retrieval possible.
Fine-tuning
Also called finetuning, LoRA, supervised fine-tuning, custom model
Fine-tuning is further training of an already-trained model on your own examples so that it adapts to a task, a format or a voice, producing a new set of weights that then has to be served somewhere.
Idempotency key
Also called idempotent request, Idempotency-Key header, safe retry, duplicate charge
An idempotency key is a client-generated identifier attached to a request so a server can recognise a retry of that exact request and return the original result instead of performing — and charging for — the work a second time.
Inference provider
Also called model host, inference API, model gateway, LLM provider
An inference provider is a company that runs open-weight model weights on its own or rented GPUs and sells access to them over an API, as distinct from the lab that trained and published the model.
Input vs output tokens
Also called prompt tokens, completion tokens, input tokens, output tokens
Input tokens are the text you send a model and output tokens are the text it writes back; they are billed at separate rates, and the output rate is almost always several times the input rate.
KV cache
Also called key-value cache, attention cache, prefill cache, KV store
A KV cache is the set of attention key and value tensors a model holds for every token already in a sequence, so that producing each new token means attending to those stored tensors rather than recomputing the entire sequence from the beginning.
Mixture of experts (MoE)
Also called MoE, sparse model, active parameters, sparse mixture of experts
A mixture-of-experts model divides its parameters into many specialised sub-networks and routes each token through only a few of them, so the parameters actually used per token — the active parameters — are a small fraction of the model's total size.
Model weights
Also called weights, parameters, checkpoint, safetensors
Model weights are the numeric parameters a training run produces — the entire learned content of a model — and a published checkpoint of them, together with the code and configuration needed to run it, is enough to reproduce that model's behaviour anywhere the hardware exists.
Open-weight model
Also called open weights, open-source model, downloadable model
An open-weight model is one whose trained parameters are published under a licence permitting others to download, run and serve them, which is what makes it possible for anyone other than the original lab to offer that model as an API.
OpenAI-compatible API
Also called OpenAI-compatible endpoint, drop-in replacement, OpenAI SDK compatible
An OpenAI-compatible API is an endpoint that accepts the same request and response shapes as OpenAI's own API, so an existing client can be pointed at it by changing only the base URL, the API key and the model name.
Prepaid balance
Also called prepaid credits, credit balance, pay-as-you-go credits, insufficient_credits
A prepaid balance is credit bought before it is spent and drawn down by each request, so the amount already loaded is a hard ceiling on what an API can cost you — the opposite of postpaid billing, where usage accrues freely and is invoiced after the fact.
Prompt caching
Also called cached input, prefix caching, cached tokens, KV cache reuse
Prompt caching is the reuse of an already-processed prompt prefix, so that repeated leading tokens — a system prompt, a tool schema, a fixed document — are billed at a reduced cached-input rate instead of the full input rate.
Quantization
Also called quantisation, INT8, FP8, 4-bit, GGUF
Quantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality.
Rate limit
Also called RPM, TPM, throttling, HTTP 429
A rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets.
Streaming (SSE)
Also called server-sent events, SSE, stream: true, token streaming, incremental response
Streaming is the delivery of a model's answer as a sequence of server-sent events, each carrying the next fragment of text as it is generated, instead of as one JSON response returned after the whole answer is finished.
Structured output (JSON mode)
Also called JSON mode, json_object, json_schema, response_format, constrained decoding
Structured output is a request setting — response_format on an OpenAI-compatible API — that constrains a model to answer with valid JSON, either free-form or conforming to a schema you supply, rather than with prose your code then has to parse out.
System prompt
Also called system message, system role, developer message, instructions
A system prompt is the instruction block placed first in a request, under the system or developer role, that sets a model's persona, rules and output format for that request — and because the API keeps no state between calls, it is resent and re-billed on every single one.
Temperature
Also called sampling temperature, top_p, sampling parameters
Temperature is a sampling parameter that controls how much randomness a model uses when choosing each next token: at 0 it takes the most likely token every time, and higher values raise the chance of less likely tokens being picked.
Throughput (tokens per second)
Also called tokens per second, tok/s, decode speed, generation speed
Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
Time to first token (TTFT)
Also called TTFT, first-token latency, prefill latency
Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
Token
Also called tokens, LLM token, text token
A token is the unit of text a language model reads and writes — roughly four characters of English, or about three quarters of a word — and it is also the unit that every language model API bills by.
Token budget
Also called max_tokens, max_completion_tokens, output cap, context budget
A token budget is the ceiling an application deliberately puts on how many tokens one request may consume: the output cap sent as max_completion_tokens, and the input cap set by how much history and retrieved context the client is willing to include.
Tool calling (function calling)
Also called function calling, tool use, tools, tool_choice, function call
Tool calling is the mechanism by which a model, given a set of function schemas alongside the prompt, replies with a structured request to invoke one of them instead of with prose — your own code runs the function and sends the result back as a further message for the model to continue from.
Worked figures on these pages come from the published rate table and change with it. Nothing here is written down twice.