KV cache
A KV cache is the set of attention key and value tensors a model holds for every token already in a sequence, so that producing each new token means attending to those stored tensors rather than recomputing the entire sequence from the beginning.
Without it, generating the nth token would mean reprocessing the n-1 before it, and a long answer would cost quadratic work. With it, each new token appends one entry and reads the rest. Every practical implementation of generation depends on this, which is why it is a hardware constraint rather than an optimisation.
It lives in GPU memory next to the weights, and it grows linearly with sequence length and with the number of concurrent requests. That product — length times concurrency — is the real ceiling on how many long-context requests a given machine can hold at once, and it is the number a serving operator is actually managing.
Prompt caching is this cache made durable. When a provider bills a repeated prefix at a reduced rate, what it is selling is the KV entries for that prefix already existing, so the prefill work is skipped. The same skipped work is what lowers time to first token, which is why caching improves latency and cost through one mechanism rather than two.
It also explains why long context is expensive to serve in a way a token count understates: a long request occupies memory for its entire lifetime, not just for the instant it is computed.
What it costs you
The KV cache is the reason a provider's cost per request is not proportional to token count alone — a long request holds memory that cannot be sold to anyone else while it is in flight, and that occupancy is priced into every rate card whether or not it is described. From the buyer's side only one half is actionable, and it is the valuable half: a stable prompt prefix means those entries are reused rather than rebuilt, and every provider that publishes a cached-input rate is quoting you the difference.
Worked from today’s rates
Kimi K3 carries a 512K context window, and building the cache for a prefix from scratch is what the standard $1.85/M input rate pays for. Reusing entries that already exist is billed at $0.37/M — about 20% of the price, for the same tokens in the same request.
Related terms
- Prompt cachingPrompt caching is the reuse of an already-processed prompt prefix, so that repeated leading tokens — a system prompt, a tool schema, a fixed document — are billed at a reduced cached-input rate instead of the full input rate.
- Context windowA context window is the maximum number of tokens a model can hold in one request, counting the system prompt, the conversation history, any retrieved documents and the reply being generated together.
- Time to first token (TTFT)Time to first token, or TTFT, is the delay between sending a request and receiving the first token of the response, and it is the part of latency a person waiting on a streamed answer actually perceives.
Go deeper
Every rate quoted above is published in full on the pricing page, alongside the model publisher’s own official rate.