How to Build a Cost-Aware LLM Router in Python: Route Each Request to the Right Model Automatically

Most AI apps send 100% of requests to one flagship model. That's like shipping every package overnight express — including the ones going across town.

In reality, a typical production workload looks like this:

• 60–70% of requests are easy: summarization, formatting, simple Q&A
• 20–30% are medium: multi-step reasoning, code generation
• 5–10% genuinely need frontier-level intelligence

If you route each request to the cheapest model that can handle it, you routinely cut costs 70–90% with minimal quality loss. In this tutorial, you'll build exactly that: a cost-aware LLM router in Python with difficulty classification, automatic escalation, fallbacks, and per-request cost tracking.

What You'll Build

A routing layer that:

1. Scores each request's difficulty (cheaply, using a tiny model)
2. Picks the right model from a tier list
3. Falls back to the next tier if the primary fails
4. Escalates to a stronger model if the output fails validation
5. Tracks cumulative spend and enforces a budget cap

Prerequisites

• Python 3.10+
• A Qubax API key (sign up free, then see the quickstart docs)
• pip install httpx

Qubax exposes an OpenAI-compatible endpoint, so the same code works whether your models come from a wholesale marketplace or any standard provider.

Step 1: Define Model Tiers

Think in tiers, not individual models. On Qubax, prices are dramatically lower than retail, which makes tiering even more effective:

TIERS = [
    {
        "name": "budget",
        "model": "glm-5.3-flash",
        "max_output": 1024,
    },
    {
        "name": "mid",
        "model": "deepseek-v4-flash",
        "max_output": 2048,
    },
    {
        "name": "frontier",
        "model": "gpt-5.6-sol",
        "max_output": 4096,
    },
]

API_URL = "https://api.qubax.ai/v1/chat/completions"

The philosophy: start cheap, escalate only on evidence of need.

Step 2: A Single Completion Helper

import httpx

def complete(model: str, messages: list, max_tokens: int = 1024) -> dict:
    """Call the API and return content + usage."""
    resp = httpx.post(
        API_URL,
        headers={"Authorization": "Bearer YOUR_QUBAX_KEY"},
        json={
            "model": model,
            "messages": messages,
            "max_tokens": max_tokens,
            "temperature": 0.2,
        },
        timeout=60,
    )
    resp.raise_for_status()
    data = resp.json()
    return {
        "content": data["choices"][0]["message"]["content"],
        "usage": data.get("usage", {}),
        "model": model,
    }

Step 3: Cheap Difficulty Classification

The trick: use your cheapest model as the classifier. One short prompt, ~50 tokens, costs a fraction of a cent:

CLASSIFIER_PROMPT = ("""Rate this request's difficulty 1-5.
1 = simple rewriting, formatting, extraction, basic facts
3 = multi-step reasoning, moderate coding, analysis
5 = hard math, complex architecture, subtle logic
Reply with ONLY the digit.

Request: {req}""")

def classify_difficulty(request: str) -> int:
    tier = TIERS[0]  # always use the cheapest model
    r = complete(tier["model"], [
        {"role": "user", "content": CLASSIFIER_PROMPT.format(req=request[:1500])}
    ], max_tokens=4)
    try:
        return min(5, max(1, int(r["content"].strip()[0])))
    except (ValueError, IndexError):
        return 3  # default to mid on classifier failure

Step 4: The Router with Fallback and Escalation

Now wire it together — route by difficulty, fall back on errors, escalate if the output looks bad:

def route(request: str, force_tier: int | None = None) -> dict:
    if force_tier is not None:
        start = force_tier
    else:
        difficulty = classify_difficulty(request)
        # difficulty 1-2 -> budget, 3 -> mid, 4-5 -> frontier
        start = 0 if difficulty <= 2 else (1 if difficulty == 3 else 2)

    attempts = []
    for tier in TIERS[start:]:
        try:
            result = complete(tier["model"], [{"role": "user", "content": request}],
                              max_tokens=tier["max_output"])
            attempts.append(tier["name"])

            # Escalate if output is suspiciously truncated or refuses a hard task
            if quality_looks_bad(result["content"], request) and tier is not TIERS[-1]:
                continue
            result["tiers_tried"] = attempts
            return result
        except httpx.HTTPError:
            continue  # fallback to next tier

    raise RuntimeError(f"All tiers failed. Tried: {attempts}")

def quality_looks_bad(output: str, request: str) -> bool:
    """Cheap heuristics — replace with your own validators."""
    if len(output.strip()) < 20:
        return True
    if "as an ai" in output.lower() and "?" in request:
        return True
    return False

Try it

easy = "Fix the grammar: 'their going to the store tommorow'"
hard = "Design a rate limiter for a multi-tenant API with token bucket semantics."

print(route(easy)["model"])   # -> glm-5.3-flash  (cheap)
print(route(hard)["model"])   # -> gpt-5.6-sol    (frontier)

Step 5: Cost Tracking and a Budget Cap

Add a tiny ledger so no single runaway prompt can drain your balance:

class Budget:
    def __init__(self, cap_usd: float):
        self.cap = cap_usd
        self.spent = 0.0

    def record(self, usage: dict, in_price: float, out_price: float):
        cost = (usage.get("prompt_tokens", 0) * in_price
                + usage.get("completion_tokens", 0) * out_price) / 1_000_000
        self.spent += cost
        if self.spent > self.cap:
            raise RuntimeError(f"Budget exceeded: ${self.spent:.4f} > ${self.cap}")
        return cost

Record after every call in complete() and log tier + cost per request. Within a day you'll have real data on your actual difficulty mix — most teams are surprised how few requests truly land in the frontier tier.

Production Tips

• Cache the classifier's decisions. Similar prompts get the same difficulty score; a small LRU cache saves thousands of classification calls.
• Log tier distribution. If >30% of requests escalate to frontier, your classifier prompt needs tuning.
• Let users force a tier. Power users know when they want frontier; give them an override (force_tier=2).
• Re-check prices monthly. The whole point of an open model marketplace is that prices shift — GLM 4.7 Flash at $0.0018/M input today was unthinkable for this quality two years ago. Compare live pricing on Qubax's models page.
• Test escalation thresholds. quality_looks_bad is deliberately dumb; swap in task-specific validators (JSON schema checks, unit tests for code, citation checks).

What This Saves in Practice

With a realistic 65/25/10 easy/medium/hard mix and Qubax pricing (budget tier ~$0.016–0.06 per million tokens vs frontier ~$1–5), blended cost per request typically drops 5–15x versus sending everything to a flagship model — and that's before prompt caching discounts.

Cheap models, expensive models, right job, right price. That's the whole game.

Ship it: Grab a free API key at qubax.ai, browse model pricing at qubax.ai/models, and read the full API documentation.

FAQ

Does routing add latency?

Yes, one extra short call (~100–300ms) for classification. Cache it, batch it, or skip classification for known request types (e.g., internal admin tools are always easy). For latency-critical paths, classify with a rule-based pre-filter first.

Won't cheap models hurt quality?

For genuinely easy requests (grammar fixes, extraction, formatting), frontier and budget models are both near 100% correct — you're paying 50x more for identical output. Escalation catches the cases where the cheap model fails.

Can I use this pattern with OpenAI-compatible providers other than Qubax?

Yes — the code only assumes an OpenAI-style /chat/completions endpoint and per-model pricing. Qubax makes it especially cheap to run multi-tier routing because you can access many providers' models under one key and one bill.

How do I pick which models go in each tier?

Pick one model per tier with the best quality-per-dollar at that level — check community benchmarks plus live prices. Revisit monthly; the leaderboard changes fast.