What Is Prompt Caching? The Simple Explanation That Can Cut Your AI Costs by 90%

If you've ever looked at your AI API bill and wondered why you're paying to send the same 10,000-token system prompt on every single request, there's a feature you should know about: prompt caching. It's one of the most impactful — and most overlooked — cost and latency optimizations in modern AI development. In this article, we'll explain what prompt caching is, how it works, why it can cut your input costs by 50–90%, and when it doesn't help.

The Problem: You Pay for Every Token, Every Time

When you call an AI API, you send two kinds of text:

The prompt — your system instructions, conversation history, retrieved documents, code context
The user message — the actual new question or request

The model's pricing is per token. And here's the pain: in most real applications, the prompt is mostly identical between requests. A coding assistant re-sends the same 20,000 tokens of codebase context with every keystroke. A support chatbot re-sends the same knowledge base. A document Q&A tool re-sends the same PDF for every follow-up question.

Without caching, the API provider processes all of those repeated tokens from scratch each time — and you pay full price for them, every single request.

What Is Prompt Caching?

Prompt caching (sometimes called "context caching" or "implicit caching," depending on the provider) lets the API provider reuse the internal computation already done for the repeated prefix of your prompt.

Here's the key insight: when a large language model reads your prompt, it does an expensive computation called a forward pass — processing every token through the model's layers and producing intermediate results called the KV cache (key-value cache of attention states). For a fixed prompt prefix, this computation is deterministic and identical every time.

So providers got clever: instead of recomputing the KV cache for the same prompt prefix over and over, they store it and reuse it. On the next request, if the beginning of your prompt matches a cached prefix, the model skips straight to the first new token.

A simple analogy: imagine you're a lawyer reviewing a 500-page contract. The first time, you read all 500 pages and make margin notes. The client then asks a question and adds one new paragraph. You don't re-read all 500 pages — you keep your notes and just review the new paragraph. Prompt caching keeps the model's "margin notes."

How It Works in Practice

1. Prefix matching

Caching works on exact prefixes. If your prompt starts with the same tokens as a previous request, the shared prefix can be served from cache. The moment one character differs at the start, the cache is invalidated from that point on.

This is why prompt structure matters enormously. Put stable content first, variable content last:

code

[Cached zone — identical every request]
  - System instructions
  - Tool definitions
  - Knowledge base / documents
  - Conversation history (oldest first)

[Fresh zone — changes every request]
  - The user's new message

If you put the user's message before your documents, you break the cache on every request. Ordering isn't a style preference — it's money.

2. Cache writes and reads are priced differently

Most providers bill cached tokens differently:

Cache write (first time the prefix is seen): a small premium over normal input price, often +25%
Cache read (subsequent hits): a large discount, typically 50–90% off normal input price
Standard input (cache miss): normal price

The math works out overwhelmingly in your favor when your prefix is reused more than once or twice. A 100K-token system prompt read from cache at 90% off costs a tenth of what a full recompute does.

3. Cache lifetime

Caches are ephemeral. Providers typically keep a prefix cached for 5 minutes to 1 hour after its last use (some offer extended TTLs for a fee). Each new request that hits the cache refreshes the timer. This is designed for active sessions: a chat conversation, an agent loop, a coding session. If your traffic is one request every three hours, don't expect cache hits.

4. Implicit vs. explicit caching

Implicit/automatic caching: the provider watches for repeated prefixes ≥ some minimum length (often 1,024 tokens) and caches automatically. You get discounts with zero code changes.
Explicit caching: you mark a specific portion of the prompt as cacheable via an API parameter (e.g., setting a cache_control breakpoint or a cache_ttl field). Explicit caching gives you control over TTL and guarantees, at the cost of a little plumbing.

Check your provider's docs — behavior and discounts vary. On Qubax AI, you can compare providers that support prompt caching and their effective per-token rates in one place.

Why Prompt Caching Is a Big Deal

Cost: 50–90% off repeated input

For chat applications, agents, and RAG systems, the prompt is typically 10–100x larger than the user's message. Cutting input costs by 90% on the cached portion can transform your unit economics. An agent doing 50 steps per task with a 30K-token working context could see its bill drop by more than half.

Latency: dramatically faster time-to-first-token

Recomputing a 100K-token prefix takes real time — often several seconds of prefill. A cache hit skips most of that work, so the model starts generating its answer much sooner. For interactive apps, this is often a bigger UX win than the discount itself.

Scale: it makes long-context patterns viable

Features that were once cost-prohibitive — pasting entire codebases into context, giving an agent a full 200K-token document set, long multi-turn conversations — became practical precisely because caching reprices them from "insane" to "reasonable."

When Prompt Caching Doesn't Help

Caching is powerful but conditional. You won't benefit if:

Your prompts are mostly unique. One-shot, unrelated requests with different content every time (e.g., a public-facing summarizer) see few hits.
Traffic is too sparse. Cache TTLs of minutes mean low-volume endpoints rarely hit.
You interleave variable content early. Timestamps, random IDs, user names, or A/B test variants at the top of your prompt poison the prefix. Move them to the end or remove them.
Your prefix is below the minimum length. Most providers require ~1,024+ tokens before a prefix is even eligible.
You rewrite semantically-identical prompts differently. Caching is exact token matching, not semantic. "Summarize this doc" and "Summarise this doc" are different prefixes.

Best Practices Checklist

Structure prompts: static first, dynamic last. System prompt → tools → documents → history → user message.
Strip request-specific noise (timestamps, request IDs) from the stable prefix.
Batch related work while the cache is warm — back-to-back calls within the TTL window all benefit.
Measure hit rates. Most providers expose cached-token counts in usage responses. If your hit rate is near zero, fix prompt structure before blaming pricing.
Design agents around it. Keep the agent's working memory as a stable, append-only prefix rather than reshuffling messages between calls.
Compare providers on cached-input pricing, not just headline input price — the effective gap between providers widens enormously once caching is factored in.

FAQ

What is prompt caching in simple terms?

Prompt caching lets an AI API provider reuse the computation already performed for the identical beginning of your prompt on repeat requests, instead of redoing it from scratch. You get a large discount (often 50–90%) on cached input tokens and faster response times.

How much does prompt caching save?

Typically 50–90% on the cached portion of input tokens, depending on the provider. There is usually a small one-time premium (often +25%) for writing to the cache the first time. For prompt-heavy applications like chat, agents, and RAG, this can cut overall serving costs dramatically.

Does prompt caching change model output quality?

No. The computation reused from cache is mathematically identical to recomputing it. Output quality and behavior are unaffected — only cost and speed change.

How long does a prompt cache last?

Commonly 5 minutes to 1 hour after the last request that used the prefix, with the timer refreshed on each cache hit. Some providers offer extended TTLs for an extra fee.

Do I need to change my code to use prompt caching?

Often no — many providers cache implicitly once your repeated prefix exceeds a minimum length (commonly 1,024 tokens). Explicit caching options give you more control. The main "code change" is usually prompt restructuring: stable content first, variable content last.

Which models support prompt caching?

Most frontier models from major providers support it, including long-context models from OpenAI, Anthropic, and Google. You can browse providers, cached-input pricing, and per-token rates on the Qubax model catalog and find integration details in the Qubax docs.

A Concrete Cost Example

Let's make this tangible with a realistic coding-assistant workload:

System prompt + codebase context: 30,000 tokens
User message per request: 500 tokens
Model output per request: 800 tokens
1,000 requests per day in active sessions

Without caching: you pay full input price on 30,000 tokens × 1,000 requests = 30 million input tokens per day, just for repeated context.

With caching (90% discount on reads, one write per session): assuming each user session makes ~10 requests sharing the same prefix, you pay the cache-write premium once per session (100 writes) and the discounted rate for the rest (9,000 reads). The repeated context effectively costs you roughly a tenth of the uncached price — savings that compound daily.

Multiply that across a product with thousands of users and the difference between "caching-aware" and "caching-oblivious" prompt design is often the single largest line-item swing in your AI budget — bigger than any model switch you could make. Before you migrate providers to save 10% on headline pricing, make sure your prompts are actually cache-friendly. It's the cheapest optimization in AI engineering: it costs a prompt restructure and pays out on every request, forever.