Time to first token (TTFT)
Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
Generation has two phases. Prefill processes the entire prompt to build the model's internal state; decode then emits output tokens one at a time. TTFT is essentially the duration of prefill plus whatever queueing happened before it, and it grows with prompt length because the whole prompt must be read before the first output token can exist.
This is why a long-context request feels slow before it produces anything. A request carrying two hundred thousand tokens of context has two hundred thousand tokens to process before the first character appears, regardless of how short the answer will be.
Prompt caching cuts directly into this. A cached prefix has already been prefilled, so the work skipped is exactly the work TTFT is made of — which is why caching improves perceived latency and cost at the same time, from the same mechanism.
Queueing is the other component and the one outside your control. A provider batching requests for GPU efficiency is trading your TTFT for its utilisation, and that trade is invisible from the client side.
What it costs you
TTFT is not billed, which is exactly why it is easy to ignore until it is a product problem: you pay the same per token whether the first one arrives in 200 milliseconds or four seconds. It becomes a cost indirectly, through the same lever that reduces it — shortening or caching a prompt lowers the bill and the wait together, so on any interactive workload latency work and cost work are the same work.
Related terms
- Throughput (tokens per second)Throughput is the rate at which a model emits output tokens once generation has started, usually quoted in tokens per second, and together with time to first token it determines how long a complete response takes.
- Prompt cachingPrompt caching is the reuse of an already-processed prompt prefix, so that repeated leading tokens — a system prompt, a tool schema, a fixed document — are billed at a reduced cached-input rate instead of the full input rate.
- Context windowA context window is the maximum number of tokens a model can hold in one request, counting the system prompt, the conversation history, any retrieved documents and the reply being generated together.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.