---
title: "Glossary"
canonical: "https://router.xark.io/glossary"
description: "26 definitions, each tied to what the concept does to a bill rather than left as a dictionary entry."
section: "core"
updated: "2026-09-02"
source: "https://router.xark.io/glossary.md"
---

# Glossary

26 definitions, each tied to what the concept does to a bill rather than left as a dictionary entry.

| Term | Definition | Document |
| --- | --- | --- |
| Batch inference | Batch inference is the processing of many independent inputs as one job rather than one interactive request at a time — either by packing several inputs into a single API call where the endpoint accepts them, or by running a large queue of calls concurrently with nobody waiting on any individual answer. | [read](https://router.xark.io/glossary/batch-inference.md) |
| Context window | A context window is the maximum number of tokens a model can hold in one request, counting the system prompt, the conversation history, any retrieved documents and the reply being generated together. | [read](https://router.xark.io/glossary/context-window.md) |
| Distillation | Distillation is the training of a small model to reproduce the outputs of a larger one, so that the smaller student performs close to the teacher across a particular range of work while costing a fraction as much to serve. | [read](https://router.xark.io/glossary/distillation.md) |
| Embedding | An embedding is a fixed-length list of numbers representing a piece of text, produced by a model so that texts with similar meaning land close together in that vector space, which is what makes semantic search and retrieval possible. | [read](https://router.xark.io/glossary/embedding.md) |
| Fine-tuning | Fine-tuning is further training of an already-trained model on your own examples so that it adapts to a task, a format or a voice, producing a new set of weights that then has to be served somewhere. | [read](https://router.xark.io/glossary/fine-tuning.md) |
| Idempotency key | An idempotency key is a client-generated identifier attached to a request so a server can recognise a retry of that exact request and return the original result instead of performing — and charging for — the work a second time. | [read](https://router.xark.io/glossary/idempotency-key.md) |
| Inference provider | An inference provider is a company that runs open-weight model weights on its own or rented GPUs and sells access to them over an API, as distinct from the lab that trained and published the model. | [read](https://router.xark.io/glossary/inference-provider.md) |
| Input vs output tokens | Input tokens are the text you send a model and output tokens are the text it writes back; they are billed at separate rates, and the output rate is almost always several times the input rate. | [read](https://router.xark.io/glossary/input-vs-output-tokens.md) |
| KV cache | A KV cache is the set of attention key and value tensors a model holds for every token already in a sequence, so that producing each new token means attending to those stored tensors rather than recomputing the entire sequence from the beginning. | [read](https://router.xark.io/glossary/kv-cache.md) |
| Mixture of experts (MoE) | A mixture-of-experts model divides its parameters into many specialised sub-networks and routes each token through only a few of them, so the parameters actually used per token — the active parameters — are a small fraction of the model's total size. | [read](https://router.xark.io/glossary/mixture-of-experts.md) |
| Model weights | Model weights are the numeric parameters a training run produces — the entire learned content of a model — and a published checkpoint of them, together with the code and configuration needed to run it, is enough to reproduce that model's behaviour anywhere the hardware exists. | [read](https://router.xark.io/glossary/model-weights.md) |
| Open-weight model | An open-weight model is one whose trained parameters are published under a licence permitting others to download, run and serve them, which is what makes it possible for anyone other than the original lab to offer that model as an API. | [read](https://router.xark.io/glossary/open-weight-model.md) |
| OpenAI-compatible API | An OpenAI-compatible API is an endpoint that accepts the same request and response shapes as OpenAI's own API, so an existing client can be pointed at it by changing only the base URL, the API key and the model name. | [read](https://router.xark.io/glossary/openai-compatible-api.md) |
| Prepaid balance | A prepaid balance is credit bought before it is spent and drawn down by each request, so the amount already loaded is a hard ceiling on what an API can cost you — the opposite of postpaid billing, where usage accrues freely and is invoiced after the fact. | [read](https://router.xark.io/glossary/prepaid-balance.md) |
| Prompt caching | Prompt caching is the reuse of an already-processed prompt prefix, so that repeated leading tokens — a system prompt, a tool schema, a fixed document — are billed at a reduced cached-input rate instead of the full input rate. | [read](https://router.xark.io/glossary/prompt-caching.md) |
| Quantization | Quantization is the compression of a model's weights from higher-precision numbers to lower-precision ones — typically 16-bit down to 8-bit or 4-bit — which reduces the memory and compute needed to serve it, at some cost to output quality. | [read](https://router.xark.io/glossary/quantization.md) |
| Rate limit | A rate limit is a cap on how many requests or how many tokens an API will accept from one key in a given window, enforced by rejecting further calls with HTTP 429 until the window resets. | [read](https://router.xark.io/glossary/rate-limit.md) |
| Streaming (SSE) | Streaming is the delivery of a model's answer as a sequence of server-sent events, each carrying the next fragment of text as it is generated, instead of as one JSON response returned after the whole answer is finished. | [read](https://router.xark.io/glossary/streaming.md) |
| Structured output (JSON mode) | Structured output is a request setting — response_format on an OpenAI-compatible API — that constrains a model to answer with valid JSON, either free-form or conforming to a schema you supply, rather than with prose your code then has to parse out. | [read](https://router.xark.io/glossary/structured-output.md) |
| System prompt | A system prompt is the instruction block placed first in a request, under the system or developer role, that sets a model's persona, rules and output format for that request — and because the API keeps no state between calls, it is resent and re-billed on every single one. | [read](https://router.xark.io/glossary/system-prompt.md) |
| Temperature | Temperature is a sampling parameter that controls how much randomness a model uses when choosing each next token: at 0 it takes the most likely token every time, and higher values raise the chance of less likely tokens being picked. | [read](https://router.xark.io/glossary/temperature.md) |
| Throughput (tokens per second) | Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes. | [read](https://router.xark.io/glossary/throughput.md) |
| Time to first token (TTFT) | Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives. | [read](https://router.xark.io/glossary/time-to-first-token.md) |
| Token | A token is the unit of text a language model reads and writes — roughly four characters of English, or about three quarters of a word — and it is also the unit that every language model API bills by. | [read](https://router.xark.io/glossary/token.md) |
| Token budget | A token budget is the ceiling an application deliberately puts on how many tokens one request may consume: the output cap sent as max_completion_tokens, and the input cap set by how much history and retrieved context the client is willing to include. | [read](https://router.xark.io/glossary/token-budget.md) |
| Tool calling (function calling) | Tool calling is the mechanism by which a model, given a set of function schemas alongside the prompt, replies with a structured request to invoke one of them instead of with prose — your own code runs the function and sends the result back as a further message for the model to continue from. | [read](https://router.xark.io/glossary/tool-calling.md) |