How to Build a Two-Tier Cost Router for AI APIs (Python Tutorial, ~60 Lines)

Most teams pay flagship-model prices for tasks a budget model could do perfectly well. The fix is a two-tier router: cheap requests go to a small model, hard requests get escalated to a frontier one. In this tutorial you'll build one in Python that typically cuts inference bills 60–80% — with a real fallback, cost tracking, and zero dependencies beyond the standard requests library.

Why routing works

The trick is that model quality isn't linear with price. A $0.10/M-token model handles summarization, extraction, classification, and casual chat nearly as well as a $10/M flagship — but flagships still win clearly on hard reasoning and long agentic tasks. If you can classify difficulty cheaply, you only pay frontier prices for the small share of requests that need them.

Typical production traffic breaks down roughly:

• 70–80% "easy" (short prompts, structured output, simple Q&A)
• 15–25% "medium" (multi-paragraph reasoning, larger context)
• 5–10% "hard" (agentic loops, novel math, subtle code)

Routing the first group down is where the savings live.

Step 1: Pick your tiers

For this tutorial we'll use concrete, widely-available models:

• Budget tier: a fast, cheap model — e.g. DeepSeek V4 Flash or GPT-5.6 Luna class. Target: under $0.10/M input tokens.
• Premium tier: a frontier model — GPT-5.6 Sol or Claude Sonnet 5 class, for escalations.

Pricing varies a lot by provider. Before wiring anything, check live rates — on Qubax's model list you can compare both tiers across providers at open-market prices, then plug your numbers into the router below.

Step 2: Score difficulty with heuristics (not another LLM)

Don't use an LLM to decide which LLM to use — that adds latency and cost. Start with cheap heuristics:

import re

def difficulty_score(prompt: str) -> int:
    """0-10 heuristic difficulty score."""
    score = 0
    words = len(prompt.split())

    # Long prompts tend to be harder
    if words > 400:  score += 2
    elif words > 150: score += 1

    # Hard-task signals
    hard_signals = [
        r"\b(prove|derive|optimize|refactor|architecture|debug)\b",
        r"\b(step[- ]by[- ]step|why|explain in depth)\b",
        r"

", # code involved r"\b(math|algorithm|complexity)\b", ] score += 2 * sum(bool(re.search(p, prompt, re.I)) for p in hard_signals)

# Easy-task signals if re.search(r"\b(summarize|translate|classify|extract|format)\b", prompt, re.I): score -= 2 if words < 60: score -= 1

return max(0, min(10, score))

Tune these signals against *your* traffic. The point is a starting split — you'll refine with real data in Step 5.

## Step 3: Route

python import os, time, requests

BUDGETMODEL = "deepseek-v4-flash" # example slugs — use your provider's PREMIUMMODEL = "gpt-5.6-sol" ROUTER_THRESHOLD = 4 # score >= 4 goes premium

def callmodel(model: str, messages: list, apikey: str, baseurl: str): resp = requests.post( f"{baseurl}/chat/completions", headers={"Authorization": f"Bearer {apikey}"}, json={"model": model, "messages": messages}, timeout=120, ) resp.raisefor_status() return resp.json()

def route(messages: list, apikey: str, baseurl: str, log: dict): prompt = messages[-1]["content"] score = difficulty_score(prompt)

tier = "premium" if score >= ROUTERTHRESHOLD else "budget" model = PREMIUMMODEL if tier == "premium" else BUDGET_MODEL

t0 = time.time() result = callmodel(model, messages, apikey, base_url) log[model] = log.get(model, 0) + 1

print(f"[router] score={score} -> {model} ({time.time()-t0:.1f}s)") return result

## Step 4: Add quality-based escalation

Heuristics misclassify sometimes. The safety net: let the budget model try first, and escalate when the output looks bad — either by a quick automated check or by your own post-processing:

python def looksbad(text: str, minlen: int = 80) -> bool: if len(text) < minlen: return True refusalmarkers = ["as an ai", "i cannot", "i'm sorry"] return any(m in text.lower() for m in refusal_markers)

def routewithescalation(messages, apikey, baseurl): prompt = messages[-1]["content"] score = difficulty_score(prompt)

if score < ROUTERTHRESHOLD: try: r = callmodel(BUDGETMODEL, messages, apikey, baseurl) text = r["choices"][0]["message"]["content"] if not looksbad(text): return r print("[router] budget output weak — escalating") except requests.RequestException as e: print(f"[router] budget failed ({e}) — escalating")

return callmodel(PREMIUMMODEL, messages, apikey, baseurl)

For structured-output tasks, a stronger check is schema validation: if the JSON doesn't parse or fails a Pydantic model, escalate. For code tasks, "does it run / do tests pass" is the ultimate escalation signal.

## Step 5: Track the savings

You can't improve what you don't measure. Tag every request with the model used and price it:

python

$/M tokens — replace with your live rates (check qubax.ai/models)

PRICES = { BUDGETMODEL: {"in": 0.03, "out": 0.18}, PREMIUMMODEL: {"in": 1.03, "out": 4.12}, }

def usagecost(model, usage) -> float: p = PRICES[model] return (usage["prompttokens"] p["in"] + usage["completion_tokens"] p["out"]) / 1000000 ```

Log usage_cost per request alongside the tier. After a week you'll know your actual easy/hard split and can tune ROUTER_THRESHOLD. In most real deployments, 70%+ of traffic lands on the budget tier — which is where the 60–80% total savings come from.

Hardening checklist before production

• Timeouts and retries with backoff on both tiers; on budget-tier repeated failure, escalate rather than retry the same model forever.
• Per-day cost caps per API key — abort or degrade to budget-only mode when exceeded.
• Cache identical prompts (a simple Redis semantic cache cuts bills another 10–30% for repetitive workloads).
• Log difficulty scores vs. final tier — misclassified-hard requests going to the budget model are your quality risk; review a sample weekly.
• Pin model versions. Providers update models; a silent swap can change both quality and price under you.

Where to go from here

Once the two-tier router is stable, natural upgrades are: a third tier for embeddings/classification, an LLM-as-judge for automatic escalation decisions on ambiguous cases, and per-customer routing policies. All of them are variations on the same loop you just built: classify cheaply, spend only where it matters.

Ready to wire it up? Compare budget and premium models side by side — with live open-market pricing — on Qubax's model index, and check the API docs to connect any model through one OpenAI-compatible endpoint.

FAQ

How much can a two-tier router actually save?

Most teams see 60–80% total savings when 70%+ of traffic qualifies as "easy." Your number depends on your workload's difficulty mix — measure it with Step 5's logging before projecting savings.

Won't cheaper models hurt output quality?

Only on hard tasks — which is exactly what the router escalates. The quality risk is misclassification, which is why the escalation check in Step 4 and weekly sampling of budget-tier outputs matter.

Should I use an LLM to classify difficulty instead of heuristics?

Rarely. A classifier call adds latency, cost, and its own error rate. Heuristics plus quality-based escalation give most of the benefit at near-zero overhead. Upgrade to an LLM judge only if your misclassification rate stays high.

Does this work with any API provider?

Yes — the router is provider-agnostic since it just swaps the model string on an OpenAI-compatible endpoint. Marketplaces like Qubax expose GPT, Claude, DeepSeek, GLM and more through one endpoint, which makes tier-swapping a one-line change.