The One-Sentence Version
The KV cache is the reason your second question to an AI model is answered faster than your first one — and the reason very long conversations start costing more.
That's the whole article in one line. The rest is the story of why, and it starts with a single inconvenient fact about how language models read text.
The Problem: Reading the Same Book Twice
A large language model generates text one token at a time. To decide on the next token, it needs to pay attention to every token that came before — your system prompt, the conversation history, the code file you pasted in.
Here's the catch. In the standard attention mechanism, every time the model produces one new token, it re-computes its internal representation of every previous token from scratch. All of them. Every single time.
Imagine reading a 500-page book by re-reading the entire book from page 1 every time you want to write the next word of your summary. That's what generation without a cache looks like. For a 100,000-token conversation, generating the 100,001st token would require recomputing attention over all 100,000 prior tokens — and then you'd throw almost all of that work away and do it again for token 100,002.
Computers are fast, but they're not that fast. Something had to change.
The Fix: Stop Throwing Away Your Homework
In 2019, a paper modestly titled "Efficient Memory Management for Large Language Model Serving with PagedAttention" — building on an insight from the original Transformer paper — formalized the fix. The key observation:
The internal representation of a past token doesn't change when you generate a new token.
Think about what that means. When the model processes your prompt, it computes two things for every token: a Key and a Value (that's the "KV" — together with a "Query," these are the three vectors attention uses, analogous to searching a database). Once computed for token #47, the Key and Value for token #47 are frozen. Token #1000 arriving later doesn't alter them.
So instead of recomputing everything: compute each token's K and V once, store them in memory, and reuse them for every subsequent generation step.
- Without KV cache: generating token N costs work proportional to N. Total cost of a response grows quadratically with length.
- With KV cache: generating token N costs a small constant amount plus attention lookup. Total cost grows linearly.
That single change is what makes interactive chat economically and technically feasible. It turned a quadratic problem into a linear one.
A Simple Analogy: The Librarian's Index Cards
Picture a librarian building a card catalog as you dictate a book to them.
- No cache: every time you add a sentence, the librarian re-reads the entire manuscript from the beginning and rewrites all their index cards from scratch.
- With KV cache: the librarian writes one new index card per new sentence and files it in the catalog. When you ask a question about page 300, they consult the existing cards — already filed, already accurate — and only make a card for the new sentence.
The index cards (Keys and Values) are the KV cache. The questions you ask (Queries) are computed fresh for each new token, but the cards they look up are reused.
Why It Matters for Your API Bill
Here's where the KV cache stops being an infrastructure trivia question and starts being money.
1. Prompt caching discounts
Because cached KV entries can be reused across requests, providers can charge less for repeated prompt prefixes. OpenAI's prompt caching gives up to 90% off cached input tokens; Anthropic's equivalent offers up to 90% off after the first time a prefix is seen (with a 5-minute sliding window); and on Qubax AI many models pass prompt-caching discounts straight through. If your app sends the same system prompt every request — and virtually every app does — you're leaving money on the table if caching isn't on.
2. Context length pricing
The cache grows with every token in context. On typical architectures, KV cache memory scales with (number of layers) × (context length) × (a model-specific constant). A 1M-token context window can require many gigabytes of KV memory per request — which is exactly why million-token models are priced per-token the way they are, and why some providers meter long-context requests differently. Long context isn't just a capability; it's a memory product.
3. Throughput and latency
KV-cache hits make responses start faster (less prefill compute) and let providers serve more users per GPU. If you've ever wondered why two models with identical pricing feel different in practice, cache hit rates are often the invisible variable.
Where the KV Cache Lives (and Why It's a Bottleneck)
The cache is stored in GPU memory — HBM, the expensive, fast memory sitting on the accelerator itself. And for long contexts it gets huge.
How huge? For a typical large model, KV cache memory per token is roughly:
2 (K and V) × layers × kv_heads × head_dim × bytes_per_valueMultiply that by a million tokens of context and you can quickly exceed the memory of the accelerator you're running on — before you've stored a single model weight. This is why the field has invented an entire discipline of making the cache smaller:
- MQA / GQA (grouped-query attention): share Keys and Values across query heads so the cache shrinks by 4–8× with minimal quality loss. Nearly every modern model — Llama 3, Qwen, GLM, DeepSeek — uses GQA.
- Sliding-window attention: only cache the last N tokens; older ones fall out of the window.
- Quantized caches (FP8 KV): store each cached value in half the bytes.
- PagedAttention: manage cache blocks like an OS manages memory pages, eliminating fragmentation. (This is the technique behind vLLM, the serving stack that popularized it.)
You don't need to memorize these. The takeaway is simpler: the KV cache is the reason long-context models are expensive to serve, and every "magic" efficiency feature you read about in model release notes is, half the time, a scheme to make this cache smaller.
KV Cache vs. Prompt Caching vs. RAG — Don't Mix These Up
Three concepts that sound similar and are constantly confused:
| Concept | What it is | Reuses KV? |
|---|---|---|
| KV cache | The on-GPU store of computed Keys/Values for tokens already processed | — (it is the thing) |
| Prompt caching | Provider feature: reuse cached prefix KVs across requests for a discount | Yes |
| RAG | Retrieval: fetch relevant documents at query time and stuff them into the prompt | No — it increases prompt length, though cached prefixes can offset it |
RAG solves a different problem (knowledge the model doesn't contain), and it actually increases context — but a well-designed RAG system keeps retrieved chunks in a stable prefix order so the KV cache can still be reused across queries.
Common Misconceptions
"The cache makes the model smarter."
No. It makes it cheaper and faster. Same model, same outputs, same weights — different economics. (Prompt caching can subtly affect provider-side routing, but capability is unchanged.)
"Caching means my data is stored."
The KV cache is numerical activation values, not stored text. It exists so the model doesn't recompute math. Privacy policies about training and retention are a separate concern entirely.
"Longer context is always better."
The cache cost of 1M-token contexts is real, and attention quality can degrade across very long contexts (the "lost in the middle" effect). Fit the context to the task, and keep the stable parts of your prompt first so caching works.
How to Actually Benefit, Starting Today
- Put stable content at the front of your prompt. System prompt, tool definitions, few-shot examples — anything identical across requests goes first. Mutable content (user question, timestamps) goes last. Cache hits require matching prefixes.
- Turn on provider prompt caching. On Qubax, many models pass through OpenAI- or Anthropic-style cache discounts automatically — check the model page for support.
- Right-size your context. Don't paste a 200k-token repo "just in case." Trim retrieval results, and prefer models with efficient long-context implementations when you genuinely need length.
- Batch stable work. If you're generating evaluations or summaries at scale, keeping prompt structure identical across requests maximizes cache hits.
- Compare true cost, not sticker price. Two models at the same per-token price can differ 2× in effective cost once cache discounts are counted. The Qubax model catalog shows live pricing across 300+ models with prompt-caching support noted.
The Bottom Line
The KV cache is the most economically important data structure in modern AI serving. It's the difference between quadratic and linear generation cost, the mechanism behind prompt-caching discounts of up to 90%, and the reason million-token contexts are a premium feature. You may never manage one directly — but understanding it means understanding your AI bill.
To explore models with transparent per-token pricing and prompt caching support, browse qubax.ai/models or read the docs.
FAQ
What is the KV cache in simple terms?
It's a table of pre-computed values (Keys and Values) for every token the model has already processed. Instead of redoing that math for every new token generated, the model looks up the cached values — turning quadratic generation cost into linear.
Does the KV cache change what the model outputs?
No. It's a pure performance optimization. With or without the cache, the outputs are mathematically identical (up to floating-point rounding). It changes speed and cost, not intelligence.
What does KV stand for?
Key and Value — two of the three vectors in the attention mechanism (the third is the Query). If attention is like searching a database, Queries are your search terms and the cached Keys/Values are the indexed records you search over.
Why does long context cost more if caching makes things cheaper?
Two different things. The KV cache grows with context length, so serving long contexts consumes GPU memory proportional to the tokens processed — that cost is passed into long-context pricing. Prompt caching (reusing the cache across requests) is what earns you discounts on repeated prefixes. Long context raises the floor; prompt caching lowers your bill below it.
How much can prompt caching save me?
Up to 90% on cached input tokens, depending on provider and model. If your app sends a 10,000-token system prompt on every request, caching typically pays for itself immediately.
Is the KV cache why my conversation gets slower as it gets longer?
Partly, yes. Each new token's Query must attend over all cached Keys/Values, so per-token work still grows with context length. The cache removed the quadratic part; the linear part remains.
Do all AI models use a KV cache?
Effectively all Transformer-based models served in production do, because the alternative is computationally absurd. Research variants (linear attention, state-space models like Mamba) replace standard attention entirely and don't have a KV cache in this form.
Where can I compare models with prompt caching and transparent pricing?
The Qubax AI model catalog lists 300+ models with live per-token pricing, context windows, and prompt-caching support — a good starting point for estimating real-world cost before you commit.
