If you are building an AI-powered application and your API bill is growing faster than your user base, prompt caching is the single most effective optimization you can implement. It is not a hack, not a workaround — it is a built-in feature of modern AI APIs that most developers simply do not use.
In this tutorial, you will learn what prompt caching is, how it works, and how to implement it in a production application to cut your token costs by up to 80%.
What Is Prompt Caching?
Prompt caching is a mechanism where the AI provider stores the intermediate computation (called the KV cache) of your prompt's prefix, so that if you send the same prefix again, the provider can skip recomputing it.
Here is the key insight: in most real-world applications, a large portion of your prompt is static — system instructions, tool definitions, few-shot examples, context documents. Only the user's actual query changes between requests.
Without caching, the provider processes the entire prompt from scratch every time. With caching, the static portion is computed once and reused, and you are only charged a fraction of the normal input price for those cached tokens.
The savings are real
Most major API providers offer cached input tokens at a 50–90% discount compared to standard input pricing. If your prompts are 80% static (which is common for agentic applications), the savings compound quickly.
How Prompt Caching Works Under the Hood
When you send a prompt to an LLM, the model processes it in two phases:
- Prefill — The model reads your entire prompt and computes a key-value (KV) cache for every token. This is the compute-heavy phase.
- Decode — The model generates output tokens one at a time, using the KV cache to attend to previous tokens.
Prompt caching stores the KV cache from the prefill phase. When you send a new request with the same prefix, the provider loads the cached KV cache instead of recomputing it. The cached portion is billed at a reduced rate.
Important constraints
- The prefix must match exactly. Any change to a single token in the cached portion invalidates the cache. The cache is prefix-based — it matches from the beginning of the prompt.
- There is a minimum cacheable length. Most providers require the prefix to be at least 1024 tokens (and sometimes more) before caching kicks in.
- Caches expire. The KV cache is typically stored for 5–60 minutes of inactivity, depending on the provider. If no request uses the cache within that window, it is evicted.
- Cache writes cost more. The first request that creates a cache is billed at a premium (often 1.25x the standard input price). Subsequent cache hits are billed at the discounted rate.
Step-by-Step Implementation
Let's build a practical example. We will use the Qubax AI API (OpenAI-compatible) to implement prompt caching for a customer support agent.
Step 1: Structure Your Prompt for Caching
The golden rule of prompt caching: put static content first, dynamic content last.
import openai
client = openai.OpenAI(
base_url="https://api.qubax.ai/v1",
api_key="your-qubax-api-key"
)
# Static prefix — this will be cached
SYSTEM_PROMPT = """You are a customer support agent for TechFlow, a SaaS company.
Follow these rules:
1. Always greet the customer by name
2. Reference their account tier (Free, Pro, Enterprise)
3. Never promise refunds — escalate to billing team
4. Keep responses under 200 words
5. If you don't know the answer, say so honestly
Company policies:
- Free tier: email support only, 48h response time
- Pro tier: chat + email, 4h response time
- Enterprise tier: 24/7 priority support, 1h response time
Product documentation:
[... 2000 tokens of product docs ...]
"""
# Dynamic suffix — changes every request
def build_prompt(user_message, customer_name, account_tier):
return [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Customer: {customer_name} ({account_tier} tier)\nQuestion: {user_message}"}
]The system prompt (including the 2000 tokens of product docs) is identical across all requests. Only the user message changes. This is the ideal structure for caching.
Step 2: Choose a Cache-Supported Model
Not all models support prompt caching. Check the Qubax AI model catalog for models with caching support. Most frontier models from Anthropic, OpenAI, and DeepSeek support it.
response = client.chat.completions.create(
model="claude-sonnet-5", # Supports prompt caching
messages=build_prompt(
user_message="How do I upgrade my plan?",
customer_name="Jane Doe",
account_tier="Pro"
),
max_tokens=300
)Step 3: Explicitly Mark Cacheable Content (Anthropic-style)
Some providers (like Anthropic) let you explicitly mark which parts of the prompt should be cached using a cache_control marker:
response = client.messages.create(
model="claude-sonnet-5",
max_tokens=300,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"} # Cache this block
}
],
messages=[
{"role": "user", "content": f"Customer: Jane Doe (Pro tier)\nQuestion: How do I upgrade?"}
]
)For OpenAI-compatible APIs (including Qubax), caching is typically automatic — the provider detects repeated prefixes and caches them without explicit markers.
Step 4: Measure Your Savings
Track your token usage to see the cache hit rate and cost savings:
import time
import json
from collections import defaultdict
class CacheMetrics:
def __init__(self):
self.requests = 0
self.cached_tokens = 0
self.uncached_tokens = 0
self.total_cost = 0.0
def record(self, usage, input_price, cached_price):
self.requests += 1
cached = usage.get("prompt_tokens_details", {}).get("cached_tokens", 0)
uncached = usage["prompt_tokens"] - cached
self.cached_tokens += cached
self.uncached_tokens += uncached
self.total_cost += (cached * cached_price + uncached * input_price) / 1_000_000
def summary(self):
total_input = self.cached_tokens + self.uncached_tokens
hit_rate = (self.cached_tokens / total_input * 100) if total_input else 0
return {
"requests": self.requests,
"total_input_tokens": total_input,
"cached_tokens": self.cached_tokens,
"cache_hit_rate": f"{hit_rate:.1f}%",
"estimated_cost": f"${self.total_cost:.4f}"
}
metrics = CacheMetrics()
# Input price: $2.00/M, Cached price: $0.20/M (90% discount)
INPUT_PRICE = 2.00
CACHED_PRICE = 0.20
for msg in customer_messages:
response = client.chat.completions.create(
model="claude-sonnet-5",
messages=build_prompt(msg["text"], msg["name"], msg["tier"]),
max_tokens=300
)
metrics.record(response.usage.to_dict(), INPUT_PRICE, CACHED_PRICE)
print(json.dumps(metrics.summary(), indent=2))Step 5: Optimize Your Cache Hit Rate
Several techniques can maximize your cache hit rate:
1. Keep the prefix truly static. Do not embed timestamps, random IDs, or session-specific data in the system prompt. Put all dynamic content in the last user message.
2. Batch similar requests. If you are processing multiple documents with the same instructions, send them in sequence within the cache TTL window (typically 5 minutes). Each request after the first benefits from the cache.
3. Order matters. The cache is prefix-based. If your system prompt is 3000 tokens and you add a 500-token context block after it, the first 3000 tokens are cached. If you swap the order, only the first 500 tokens are cached.
4. Warm the cache. For applications with predictable traffic patterns, send a "warmup" request at the start of a session to populate the cache before real users arrive.
def warm_cache():
"""Send a minimal request to populate the KV cache."""
client.chat.completions.create(
model="claude-sonnet-5",
messages=build_prompt(
user_message="warmup",
customer_name="system",
account_tier="system"
),
max_tokens=1 # We don't need a real response
)Real-World Cost Example
Let's say you run a customer support bot that handles 10,000 queries per day. Each request has a 3,000-token system prompt (instructions + product docs) and a 200-token user query.
Without caching:
- Input tokens per request: 3,200
- Total daily input tokens: 32,000,000
- At $2.00/M (Claude Sonnet 5 input price): $64.00/day
With caching (90% of input cached at 10% of price):
- Cached tokens per request: 3,000 (at $0.20/M)
- Uncached tokens per request: 200 (at $2.00/M)
- Daily cached cost: 30,000,000 × $0.20/M = $6.00
- Daily uncached cost: 2,000,000 × $2.00/M = $4.00
- Total: $10.00/day — an 84% reduction
Over a month, that is $1,200 saved — on a single application, from a single optimization.
Common Pitfalls
- Changing even one character in the prefix invalidates the entire cache. Be especially careful with templating engines that might add invisible whitespace or vary formatting.
- The cache TTL is short. If your traffic is bursty with long gaps between requests, you may not benefit. Consider cache warming or choosing a provider with longer TTLs.
- Not all models support caching. Always check the provider's documentation. On Qubax AI, model cards indicate caching support.
- Cache write premiums. The first request costs more (up to 25% extra). If you only send one request per cache window, you actually pay more, not less.
Conclusion
Prompt caching is the highest-ROI optimization available to developers building on AI APIs. It requires no model changes, no quality degradation, and can be implemented in an afternoon. If your application has a significant static prompt prefix — and most do — you are leaving money on the table by not using it.
Start by auditing your prompts: what percentage is static? If it is more than 50%, caching will save you money. Then restructure your prompts to put static content first, choose a caching-supported model, and measure the results.
Ready to implement prompt caching? Explore caching-supported models on [Qubax AI](https://qubax.ai/models) and check the [API docs](https://qubax.ai/docs) for implementation details.
FAQ
What is prompt caching in AI APIs?
Prompt caching is a feature where the API provider stores the computed key-value cache of your prompt's prefix, so repeated requests with the same prefix are processed faster and at a lower cost.
How much does prompt caching save?
Cached input tokens are typically billed at 10–50% of the standard input price, depending on the provider. If 80% of your prompt is static, you can save 70–80% on input token costs.
Which AI models support prompt caching?
Most frontier models from Anthropic (Claude), OpenAI (GPT), and DeepSeek support prompt caching. Check the model catalog on Qubax AI for caching support indicators.
How long does the prompt cache last?
Cache TTL varies by provider, typically 5 to 60 minutes of inactivity. If no request uses the cache within the TTL window, it is evicted and must be recomputed.
Does prompt caching affect output quality?
No. Prompt caching produces identical outputs to non-cached requests. It only affects the speed and cost of processing the input — the model's behavior is unchanged.
What happens if I change one word in my system prompt?
The entire cache is invalidated. Prompt caching is prefix-based and requires an exact match from the beginning of the prompt. Any change, even a single character, forces a full recompute.
