If you've ever wondered why running an AI model costs anything at all — why a chatbot response costs fractions of a cent but a billion of them costs millions — the answer lives in a piece of engineering called the KV cache. It's one of the most important, least understood concepts in AI infrastructure, and understanding it explains everything from API pricing tiers to why long conversations get slower.
The one-sentence version
A KV cache is the model's memory of what it has already read: instead of recomputing how every past token relates to every other token on each new step, the model stores those calculations and reuses them.
The problem it solves
Transformer models (the "T" in GPT) generate text one token at a time. To write the next token, the model runs an operation called attention: every token "looks at" every previous token to decide what's relevant.
Here's the trap. If generating token #1,000 required re-examining all 999 previous tokens from scratch, then token #1,001 would re-examine 1,000, and so on. Every single token would require redoing the work of all previous tokens. The cost per token would grow as your conversation gets longer — generating a 10,000-token document would be enormously expensive.
The fix: the two things attention needs from each past token — its K (key) and V (value) vectors — don't change once that token is generated. Past tokens can't retroactively change what they mean. So the model computes them once, stores them in GPU memory (the cache), and every future token just reads them.
Result: generating the next token gets one cheap new computation plus lookups, instead of recomputing everything. Autoregressive generation goes from "quadratic pain" to "linear and fast."
A simple analogy
Imagine re-reading an entire book from page one every time you wanted to recall a single plot point. That's a transformer without caching. The KV cache is a set of detailed margin notes: write down each page's key facts once, then glance at your notes instead of re-reading.
Where the cost actually goes
The catch: the cache lives in GPU memory (VRAM), and it's big.
A cache entry exists for every token, every layer, and every attention head. For a large model with a long context, cached tokens can occupy gigabytes. This is why:
- Longer contexts cost more. When you send a 100,000-token document, the provider computes and stores a 100,000-token cache. That memory is real, physical hardware doing real work — hence context-based pricing.
- The second request is cheaper. That's exactly what prompt caching discounts are: if your request shares a long prefix with a recent request (same system prompt, same document), the provider reuses the stored KV cache and charges you a fraction — often 50–90% less — for those tokens.
- Models are getting cache-efficient. Techniques like grouped-query attention (sharing keys/values across heads) and architectures like DeepSeek's recent KV designs — one reported to need just 1/4 the memory of the previous generation — exist specifically to shrink this cache.
Cache-related terms you'll see in provider pricing
| Term | What it means |
|---|---|
| Cached input / cache read | Tokens whose KV cache already exists — billed at a big discount |
| Cache write | First time computing a long prefix — sometimes billed at a small premium |
| Input tokens | Prompt tokens processed fresh (new KV cache computed) |
| Output tokens | Newly generated tokens — most expensive, since each requires a full forward pass |
This is also why "prompt caching" is the single biggest cost optimization available to API developers. Put your stable content (system prompt, retrieved documents, code context) first and your changing content (the user's actual question) last, and you pay the cached rate for the stable 95%.
Why this matters for your bill
A concrete example. Say you're building a support bot with a 10,000-token knowledge base and ~200-token user questions.
- Without prompt caching: every question pays full input price on 10,200 tokens.
- With prompt caching: the 10,000-token knowledge base hits the cached rate (often ~10% of full price), and only your 200 tokens bill at full input rate.
Same model, same quality — often 5–10x cheaper just from respecting how the KV cache works.
The takeaway
The KV cache is why AI APIs have a pricing structure at all: input vs output, cached vs uncached, short vs long context. It's not a billing gimmick — it maps directly to the physical resource (GPU memory) that inference providers actually spend. And it's why the cheapest providers win: the open market where compute providers compete on price rewards whoever manages cache memory most efficiently, and those savings pass to you.
FAQ
Does the KV cache persist between my API requests?
Only if the provider supports prompt caching AND your request shares an exact prefix with a recent one. Otherwise each request rebuilds its cache from scratch.
Why are output tokens more expensive than input tokens?
Each output token requires a full forward pass plus growing cache reads, and generation is sequential — it can't be batched as efficiently as processing a prompt in parallel.
What is "context caching" or "prompt caching"?
A provider feature that stores the KV cache for a shared prompt prefix so repeat requests skip recomputation. Discounts typically range from 50–90% on cached tokens.
Does a bigger context window mean a bigger KV cache?
Yes — cache size grows roughly linearly with context length (and depends on the architecture; modern designs shrink it substantially per token).
Run cache-efficient workloads on Qubax → [qubax.ai/models](https://qubax.ai/models)
Related: [What is inference-time compute?](https://qubax.ai/blog/2026-09-21-what-is-inference-time-compute-simple-explanation) · [What is Mixture of Experts?](https://qubax.ai/blog/2026-09-16-what-is-mixture-of-experts-simple-explanation) · [API docs](https://qubax.ai/docs)
