How to Build a Semantic Cache for LLM APIs in Python (Cut Your Token Bill by 40-60%)

Every serious LLM application answers the same questions over and over — slightly reworded. "What's your refund policy?", "How do refunds work?", "Can I get my money back?" Same intent, same answer, and today, same full-price API call every time. A semantic cache fixes that, and in this tutorial you'll build one in under 100 lines of Python.

Semantic caching stores previously answered questions along with their meaning (an embedding vector), not just their exact text. When a new question arrives, you check whether a similar question was already answered; if the similarity is high enough, you return the stored answer instantly — zero tokens, zero latency, near-zero cost. For support bots, documentation assistants, and RAG pipelines, teams routinely see 40–60% of requests served from cache.

This tutorial uses the OpenAI-compatible API format, so the same code works with any model accessible through Qubax — from flagship models to bargain-tier options like DeepSeek V4 Flash at roughly $0.005 per million input tokens.

Architecture in 60 Seconds

User question
   |
   v
Embed the question --> Vector store (in-memory or SQLite)
   |                        |
   |              nearest neighbor + similarity score
   |                        |
   +-- similarity >= threshold --> return cached answer  (cost: $0)
   |
   +-- similarity < threshold  --> call the LLM --> store (question, embedding, answer)

Two design decisions matter:

1. The similarity threshold. Too low, and you answer wrong questions from cache. Too high, and the cache never hits. We'll start at 0.90 cosine similarity and show you how to tune it.
2. Cheap embeddings. Cache lookups must cost a tiny fraction of an LLM call. Use a small embedding model — you'll pay microseconds, not cents.

Prerequisites

pip install openai numpy

You'll need a secret key and a base URL pointing at your provider. Keep the key in a .env file (never commit it) rather than pasting it into code or shell history:

# .env
SECRET_KEY = sk-...                        # your secret API key, loaded via dotenv
BASE_URL   = https://api.qubax.ai/v1       # OpenAI-compatible endpoint

import os
from dotenv import load_dotenv
load_dotenv()  # loads SECRET_KEY and BASE_URL from .env

from openai import OpenAI
client = OpenAI(api_key=os.environ["SECRET_KEY"],
                base_url=os.environ["BASE_URL"])

Step 1: The Cache Core

import json, time
import numpy as np

class SemanticCache:
    def __init__(self, threshold=0.90, path="cache.json"):
        self.threshold = threshold
        self.path = path
        self.entries = []  # [{question, embedding, answer, model, ts}]
        try:
            with open(path) as f:
                self.entries = json.load(f)
        except FileNotFoundError:
            pass

    def embed(self, text: str) -> np.ndarray:
        r = client.embeddings.create(model="text-embedding-3-small", input=text)
        v = np.array(r.data[0].embedding, dtype=np.float32)
        return v / np.linalg.norm(v)  # normalize -> cosine similarity = dot product

    def lookup(self, question: str):
        if not self.entries:
            return None
        q = self.embed(question)
        matrix = np.array([e["embedding"] for e in self.entries], dtype=np.float32)
        sims = matrix @ q
        i = int(np.argmax(sims))
        if sims[i] >= self.threshold:
            return self.entries[i]["answer"]
        return None

    def store(self, question: str, answer: str, model: str):
        self.entries.append({
            "question": question,
            "embedding": self.embed(question).tolist(),
            "answer": answer,
            "model": model,
            "ts": time.time(),
        })
        with open(self.path, "w") as f:
            json.dump(self.entries, f)

cache = SemanticCache()

Step 2: Wrap Your LLM Calls

CHAT_MODEL = "deepseek-v4-flash"   # swap for any model on qubax.ai/models

def ask(question: str) -> dict:
    hit = cache.lookup(question)
    if hit:
        return {"answer": hit, "source": "cache", "cost_usd": 0.0}

    resp = client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": "You are a concise support assistant."},
            {"role": "user", "content": question},
        ],
    )
    answer = resp.choices[0].message.content
    cache.store(question, answer, CHAT_MODEL)
    return {"answer": answer, "source": "llm", "cost_usd": 0.00001}

That's the whole system. Try it:

print(ask("What is your refund policy?"))        # -> source: llm
print(ask("How does your refund policy work?"))  # -> source: cache
print(ask("Can I get a refund?"))                # -> source: cache

Step 3: Tune the Threshold Like an Engineer, Not a Gambler

The threshold is your precision/recall dial. Test it against your real traffic instead of guessing:

TEST_PAIRS = [
    # (question_a, question_b, should_match)
    ("What is your refund policy?", "How do I get a refund?", True),
    ("What is your refund policy?", "How do I reset my password?", False),
    ("Do you ship to Canada?", "Is shipping available to Canada?", True),
    ("Do you ship to Canada?", "Do you ship to Germany?", True),  # borderline!
]

def evaluate(threshold):
    cache.threshold = threshold
    correct = 0
    for a, b, expected in TEST_PAIRS:
        cache.store(a, "answer:" + a, CHAT_MODEL)
        hit = cache.lookup(b) is not None
        correct += (hit == expected)
    return correct / len(TEST_PAIRS)

for t in (0.80, 0.85, 0.90, 0.95):
    print(t, evaluate(t))

Rules of thumb:

• 0.90–0.93 works well for support/FAQ traffic with one embedding model.
• 0.95+ for high-stakes answers (medical, legal, financial) where a stale or wrong cache hit is worse than an extra API call.
• Below 0.85 you will start serving confidently wrong answers. Don't.

Step 4: Make It Production-Ready

The toy cache above needs four upgrades before real traffic:

1. Namespace by context. A refund question in the billing docs and the same question in the returns policy may need different answers. Include the namespace in the embedding key:

def lookup_ns(self, question, namespace):
    tagged = f"{namespace}:: {question}"
    ...

2. TTL and invalidation. Prices, policies, and docs change. Add a ts check in lookup — skip entries older than your freshness budget:

if time.time() - self.entries[i]["ts"] > 7 * 86400:  # 7 days
    return None

3. Real vector storage. Past ~50k entries, load embeddings once into FAISS or use SQLite with sqlite-vec. Don't reload a JSON file per request.

4. Log every cache decision. Store the similarity score in the response so you can audit near-misses. The most valuable metric is your false-hit rate — how often users rephrase the cached answer because it was wrong for their variant.

Step 5: Combine With Routing for Maximum Savings

Caching is one layer of a cost strategy. The full stack looks like this:

1. Exact-match cache (a plain dict) for literally repeated prompts — free.
2. Semantic cache (this tutorial) for reworded repeats — free.
3. Model routing: send what's left to the cheapest model that can handle it. Classification and extraction tasks rarely need a flagship; DeepSeek V4 Flash or GLM 5.3 Flash handle them at 1–2% of flagship cost on Qubax's marketplace.
4. Prompt caching for long, stable system prompts (your provider may discount repeated prefixes automatically).

A support bot using all four layers typically ends up paying full price on fewer than half its requests — and that's before you negotiate volume.

Common Pitfalls

• Caching personal data. If users embed PII in questions, your cache becomes a PII store. Redact before embedding, or namespace per user.
• One global threshold. Different intents tolerate different error rates. Consider per-namespace thresholds.
• Silent staleness. Pair every cache with an invalidation event from your source-of-truth system ("policy updated -> purge refund entries").
• Benchmarking with synthetic traffic. Hit rates on invented questions are meaningless. Measure on real logs.

Conclusion

Semantic caching is the highest-ROI optimization in the LLM stack: ~100 lines of code, no model changes, and instantly fewer tokens billed. Combined with cheap-model routing on an open marketplace, it's how lean teams serve flagship-quality answers at commodity costs.

Build it, measure your hit rate on real traffic, then tune the threshold with the evaluation script above. And when you're ready to route the remaining cache misses to the cheapest capable model, browse live per-token pricing on Qubax — the API docs cover the OpenAI-compatible endpoint used in this tutorial.

FAQ

How much can a semantic cache actually save?

It depends entirely on your traffic redundancy. Support bots and FAQ assistants often see 40–60% cache hits; creative-generation workloads may see under 10%. Measure on real logs before projecting savings.

Is a cached answer as good as a fresh one?

The cached answer is identical to what the model gave the first time. The risk isn't quality — it's appropriateness: a slightly different question may deserve a slightly different answer. That's what the threshold and namespaces manage.

Which embedding model should I use?

A small, fast one — embedding lookups must stay far cheaper than LLM calls. text-embedding-3-small or an equivalent open embedding model is fine for most English traffic.

Does this work with any LLM provider?

Yes, as long as the provider offers an OpenAI-compatible chat and embeddings API. The code in this tutorial runs unmodified against any such endpoint, including Qubax.