Build a Multi-Model Fallback Router in Python: Never Let an AI Outage Hit Your Users

Every production AI feature needs a fallback. Your primary model will rate-limit, timeout, or return garbage at 3am — and if your code has no plan B, your users see the error. In this guide we build a resilient multi-model router in Python that retries across models, tracks cost, and fails gracefully.

What You Will Build

A ~100-line Python module that:

1. Calls a list of candidate models in priority order
2. Retries transient failures with exponential backoff
3. Falls back to the next model on persistent errors
4. Tracks token usage and estimated spend per request

We will use the OpenAI-compatible API surface, which Qubax exposes for every model — so swapping models is a one-line change.

Prerequisites

• Python 3.10+
• A Qubax API key (grab one from your Qubax dashboard)

pip install openai

Step 1: The Basic Client

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["QUBAX_API_KEY"],
    base_url="https://api.qubax.ai/v1",
)

def complete(prompt: str, model: str) -> str:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

Step 2: Retry With Exponential Backoff

Transient errors (429s, 5xx, timeouts) deserve a retry before a fallback. Don't retry validation errors — a bad request will stay bad.

import random

RETRYABLE = {429, 500, 502, 503, 504}

def complete_with_retry(prompt: str, model: str, max_retries: int = 2) -> str:
    for attempt in range(max_retries + 1):
        try:
            return complete(prompt, model)
        except Exception as e:
            status = getattr(e, "status_code", None)
            if status not in RETRYABLE or attempt == max_retries:
                raise
            sleep = (2 ** attempt) + random.random()
            print(f"[{model}] retry {attempt + 1} after {sleep:.1f}s: {e}")
            time.sleep(sleep)

Step 3: The Fallback Chain

Now the core: try each model in order. Put your best/cheapest model first; put the reliable workhorse last.

FALLBACK_CHAIN = [
    "glm-5.3-flash",      # fast + cheap primary
    "deepseek-v4-flash",  # ultra-cheap backup
    "claude-sonnet-5",    # premium last resort
]

def complete_any(prompt: str) -> tuple[str, str]:
    last_err = None
    for model in FALLBACK_CHAIN:
        try:
            return complete_with_retry(prompt, model), model
        except Exception as e:
            last_err = e
            print(f"[{model}] FAILED, falling back: {e}")
    raise RuntimeError(f"All models failed") from last_err

Step 4: Cost Tracking

Read usage from every response so you know exactly what each model costs you in practice — not in theory.

def complete_with_cost(prompt: str) -> dict:
    resp = client.chat.completions.create(
        model=FALLBACK_CHAIN[0],
        messages=[{"role": "user", "content": prompt}],
    )
    u = resp.usage
    return {
        "text": resp.choices[0].message.content,
        "model": resp.model,
        "input_tokens": u.prompt_tokens,
        "output_tokens": u.completion_tokens,
    }

Log these numbers. After a week you will know your real cost per request, which models burn budget, and whether your fallback chain is even being exercised.

Step 5: Fail Gracefully

Finally, wrap it for production — return a degraded experience instead of a 500:

def safe_complete(prompt: str) -> str:
    try:
        text, model = complete_any(prompt)
        return text
    except RuntimeError:
        return "Our AI service is temporarily unavailable. Please try again shortly."

Testing Your Fallbacks

Do not wait for a real outage. Temporarily put a nonsense model name first in the chain and run your test suite — verify the request lands on model #2, latency stays acceptable, and your logs say what happened. Fallback code that has never been executed is fallback code that does not work.

Going Further

• Routing by task: send coding prompts to a coding-tuned model, cheap classification to a nano-tier model. Compare options on the Qubax models page.
• Latency-aware routing: track p95 latency per model and demote slow ones automatically.
• Budget guards: kill-switch a model when its daily spend crosses a threshold.

Resilience is mostly bookkeeping. A candidate list, a retry loop, and honest cost logging will carry you further than any exotic orchestration framework.

All models in this tutorial are available on [Qubax AI](https://qubax.ai/models) at wholesale prices through one OpenAI-compatible API — see the [docs](https://qubax.ai/docs) for authentication details.

FAQ

Does an OpenAI SDK work with other providers?

Yes — any OpenAI-compatible endpoint works with the official SDK. You only change base_url and api_key, which is exactly what makes multi-provider fallback chains this simple.

What order should my fallback chain be?

Cheapest model that satisfies your quality bar first, most expensive last. That way fallbacks are rare, and when they happen you are paying up for reliability, not by default.

How do I know which models are cheapest right now?

Check the pricing tables on qubax.ai/models — prices update as compute providers compete, so a chain that was optimal last month may not be today.