How to Build Multi-Model AI Failover with Circuit Breakers (TypeScript Tutorial)

Your AI-powered app is only as reliable as the provider serving your model. When OpenAI has an outage — and it will — your users see errors. When Anthropic rate-limits you during a traffic spike, your checkout flow breaks. The fix isn't hoping providers stay up. The fix is automatic failover.

In this tutorial, you'll build a production-grade multi-model fallback client in TypeScript: a circuit breaker per provider, automatic retry on the next-best model, and a health tracker that stops hammering dead endpoints. This is the same resilience pattern that gateways like Qubax AI implement internally — but here you'll own the logic, so you understand every moving part.

What You're Building

By the end, you'll have a client that:

1. Tries your primary model (say, GPT-5.6 Sol)
2. On failure, degrades gracefully to a fallback chain (Claude Opus 4.8 → DeepSeek V4 Pro)
3. Uses a circuit breaker so a dead provider gets skipped instantly, not retried into
4. Tracks per-provider latency and error rates
5. Exposes a single chat() method your app calls everywhere

The beauty of using an OpenAI-compatible aggregator: all three models share one request format, so failover is a model-name change, not a re-integration.

Prerequisites

• Node.js 18+ and TypeScript
• A Qubax API key (get one here)
• Basic familiarity with fetch and async/await

Step 1: Project Setup

mkdir ai-failover && cd ai-failover
npm init -y
npm install typescript tsx @types/node
npx tsc --init

Create a .env with your key:

QUBAX_API_KEY=qx_live_your_key_here

Step 2: The Provider Chain Model

First, define your fallback chain. Each entry is a model plus its role in the chain:

// chain.ts
export interface ChainEntry {
  model: string;
  label: string;        // human-readable name for logs
  timeoutMs: number;    // per-provider timeout
}

export const CHAIN: ChainEntry[] = [
  { model: "gpt-5.6-sol",     label: "GPT-5.6 Sol",      timeoutMs: 30_000 },
  { model: "claude-opus-4.8", label: "Claude Opus 4.8",  timeoutMs: 30_000 },
  { model: "deepseek-v4-pro", label: "DeepSeek V4 Pro",  timeoutMs: 45_000 },
];

The chain order is your degradation strategy. Ours goes frontier → frontier → budget-frontier. If you're cost-sensitive, invert the logic: put the cheap model first and only escalate to expensive models on failure (or on a quality signal).

Step 3: The Circuit Breaker

The circuit breaker is the heart of the system. It has three states:

• CLOSED — everything is fine, requests flow through
• OPEN — the provider has failed too much; requests skip it entirely for a cooldown period
• HALF-OPEN — cooldown elapsed; one trial request is allowed through to test recovery

// breaker.ts
export class CircuitBreaker {
  private failures = 0;
  private openedAt = 0;
  private state: "closed" | "open" | "half-open" = "closed";

  constructor(
    private readonly threshold = 3,       // failures before opening
    private readonly cooldownMs = 30_000  // how long to stay open
  ) {}

  canRequest(): boolean {
    if (this.state === "closed") return true;
    if (this.state === "open") {
      if (Date.now() - this.openedAt >= this.cooldownMs) {
        this.state = "half-open";
        return true; // allow one trial request
      }
      return false;
    }
    return false; // half-open: only the trial request goes through
  }

  recordSuccess(): void {
    this.failures = 0;
    this.state = "closed";
  }

  recordFailure(): void {
    this.failures++;
    if (this.failures >= this.threshold || this.state === "half-open") {
      this.state = "open";
      this.openedAt = Date.now();
    }
  }

  get status() {
    return { state: this.state, failures: this.failures };
  }
}

Why this matters: without a breaker, a dead provider means every request wastes its full timeout (often 30+ seconds) before failing over. With a breaker, the dead provider is skipped in microseconds. Your users go from 30-second hangs to imperceptible failover after three failures.

Step 4: The Failover Client

Now the client that ties it together:

// client.ts
import { CircuitBreaker } from "./breaker";
import { CHAIN, ChainEntry } from "./chain";

interface ChatMessage {
  role: "system" | "user" | "assistant";
  content: string;
}

interface ChatOptions {
  messages: ChatMessage[];
  maxTokens?: number;
  temperature?: number;
}

interface ChatResult {
  content: string;
  servedBy: string;      // which model actually answered
  attempts: number;      // how many providers we tried
}

const API_URL = "https://api.qubax.ai/v1/chat/completions";
const API_KEY = process.env.QUBAX_API_KEY!;

export class FailoverClient {
  private breakers = new Map<string, CircuitBreaker>();

  private breakerFor(model: string): CircuitBreaker {
    if (!this.breakers.has(model)) {
      this.breakers.set(model, new CircuitBreaker());
    }
    return this.breakers.get(model)!;
  }

  async chat(opts: ChatOptions): Promise<ChatResult> {
    const errors: string[] = [];

    for (let i = 0; i < CHAIN.length; i++) {
      const entry = CHAIN[i];
      const breaker = this.breakerFor(entry.model);

      if (!breaker.canRequest()) {
        errors.push(`${entry.label}: circuit open, skipped`);
        continue;
      }

      try {
        const content = await this.callModel(entry, opts);
        breaker.recordSuccess();
        return { content, servedBy: entry.label, attempts: i + 1 };
      } catch (err) {
        breaker.recordFailure();
        errors.push(`${entry.label}: ${(err as Error).message}`);
        console.warn(`[failover] ${entry.label} failed, trying next...`);
      }
    }

    throw new Error(
      `All providers failed. Attempts:\n${errors.join("\n")}`
    );
  }

  private async callModel(
    entry: ChainEntry,
    opts: ChatOptions
  ): Promise<string> {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), entry.timeoutMs);

    try {
      const res = await fetch(API_URL, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${API_KEY}`,
        },
        body: JSON.stringify({
          model: entry.model,
          messages: opts.messages,
          max_tokens: opts.maxTokens ?? 1024,
          temperature: opts.temperature ?? 0.7,
        }),
        signal: controller.signal,
      });

      if (!res.ok) {
        const body = await res.text();
        throw new Error(`HTTP ${res.status}: ${body.slice(0, 200)}`);
      }

      const data = await res.json();
      const content = data.choices?.[0]?.message?.content;
      if (!content) throw new Error("Empty response");
      return content;
    } finally {
      clearTimeout(timer);
    }
  }

  healthReport() {
    return Object.fromEntries(
      [...this.breakers.entries()].map(([model, b]) => [model, b.status])
    );
  }
}

Three details worth noticing:

• Per-model timeouts. A budget model might legitimately take longer than a frontier one; each chain entry gets its own budget.
• AbortController everywhere. Without an explicit timeout, a hung connection blocks the entire chain. The abort guarantees you move on.
• `servedBy` in the result. You always know which model answered — essential for debugging quality regressions ("why did this response style change?") and for cost attribution.

Step 5: Use It

// index.ts
import { FailoverClient } from "./client";

const client = new FailoverClient();

async function main() {
  const result = await client.chat({
    messages: [
      { role: "system", content: "You are a concise technical assistant." },
      { role: "user", content: "Explain circuit breakers in two sentences." },
    ],
    maxTokens: 200,
  });

  console.log(`Served by: ${result.servedBy} (attempt ${result.attempts})`);
  console.log(result.content);

  console.log("Health:", client.healthReport());
}

main();

Run it:

npx tsx index.ts

Step 6: Test the Failover

Simulate an outage by setting the first model to a nonsense name — the API will 404, the breaker will trip after enough failures, and you'll watch requests flow to Claude and DeepSeek instead:

export const CHAIN: ChainEntry[] = [
  { model: "invalid-model-x",  label: "Broken (test)",     timeoutMs: 10_000 },
  { model: "gpt-5.6-sol",      label: "GPT-5.6 Sol",       timeoutMs: 30_000 },
];

Within a few requests, the health report shows the breaker open for the dead entry, and latency for subsequent calls stays flat — because canRequest() rejects instantly instead of waiting on a timeout. That's the whole point.

Step 7: Production Hardening

For real production use, add these:

• Streaming support. Failover must happen before the first streamed token; after that, you can't transparently switch. Buffer nothing, but decide fast.
• Exponential backoff with jitter for transient errors (429, 503) before counting them as breaker failures — a rate limit isn't an outage.
• Latency-based demotion. Track a rolling p95 per provider; if it creeps past your SLO, deprioritize that chain entry even if it's technically healthy.
• Quality-aware escalation. The reverse pattern: start on a cheap model, and escalate to a frontier model only when a cheap-model response fails a validation check (JSON schema, length, confidence signal). This combines resilience with cost routing.
• Persistent breaker state. In-memory breakers reset on deploy. If you run multiple instances, move breaker state to Redis so one instance's bad experience teaches the fleet.

What This Buys You

• Availability. One provider's outage becomes invisible to your users.
• Latency protection. Dead providers are skipped in microseconds, not timeout-seconds.
• Cost control. Degradation order is yours — failover to a cheaper model can actually save money during incidents, especially with Qubax's platform pricing.
• Observability. servedBy + health reports make every request auditable.

The full code is ~150 lines with no dependencies beyond Node's built-in fetch. Build it once, and provider outages stop being your pager's problem.

Explore models for your fallback chain at qubax.ai/models — full API docs at qubax.ai/docs.

FAQ

What is automatic failover in AI APIs?

Automatic failover means that when one AI model provider fails (timeout, 500 error, rate limit), your client automatically retries the request on a different model — without your application or user seeing an error. It's implemented with a fallback chain of models and retry logic.

Why do I need a circuit breaker?

Without one, every request to a dead provider waits for a full timeout (often 30+ seconds) before failing over. A circuit breaker "opens" after repeated failures and skips the dead provider instantly, keeping user-perceived latency flat during outages.

Can I do failover across different providers?

Yes — and an OpenAI-compatible aggregator like Qubax makes it trivial. Since all models share one request format and one API key, failing over from GPT-5.6 Sol to Claude Opus 4.8 to DeepSeek V4 Pro is just changing the model field.

Should failover go to a cheaper or more expensive model?

Depends on your priority. For revenue-critical features, fail up to the most reliable frontier model. For cost-sensitive pipelines, fail down to a cheaper model — incidents become savings. Most teams use both: fail up for user-facing, fail down for background jobs.

How many models should be in a fallback chain?

Three is the sweet spot: primary, backup, and emergency. Beyond that, you add configuration complexity for rapidly diminishing reliability returns. Prioritize chain diversity — don't put three models that share one upstream provider in a chain.

Does failover work with streaming responses?

Partially. You can fail over transparently only before the first token reaches the client. After streaming starts, a mid-stream failure can't be retried without the user noticing. Decide fast (short timeouts on first-token latency) and consider sending a graceful "regenerating" message if a stream dies.

How is this different from a model router?

A router chooses the model before the request based on rules (cost, task type). Failover switches models after a failure. Production systems combine both: routing picks the starting point, and failover handles the exceptions. See our earlier guide on building a cost-based router, then layer this failover logic on top.