Cut your LLM bill 80-95% with a confidence-based router: cheap model first, flagship only when it matters.
Introduction
Here's the pattern that separates developers with a $50/month AI bill from those with a $5,000/month bill: the expensive ones send every request to a flagship model; the cheap ones route.
Most production traffic — simple chat, extraction, classification, straightforward code — doesn't need a frontier model. It needs a cheap model 90% of the time, and a smart escalation path for the 10% that's hard. In this tutorial, you'll build a production-grade confidence-based LLM router in TypeScript: cheap model first, automatic self-review, and escalation to a flagship only when the cheap model isn't confident.
Total time: ~30 minutes. Result: the single highest-ROI optimization you can make to an AI stack.
The Architecture
The router works in three stages:
- Attempt the task with a cheap model (e.g., a Flash-class model).
- Review: a lightweight grader step checks whether the cheap model's answer meets your requirements (format, completeness, confidence).
- Escalate: if the review fails, retry with a flagship model.
user request ──▶ cheap model ──▶ grader ──▶ pass ──▶ return answer
│
└──▶ fail ──▶ flagship model ──▶ return answerThis "cascade" pattern routinely cuts costs 80–95% on realistic workloads because most requests pass the cheap tier.
Prerequisites
- Node.js 20+
- An API key for an OpenAI-compatible endpoint. We'll use Qubax, which exposes one API for models from every major provider — grab a key at qubax.ai, and see the API docs for details.
npm install openai(the official SDK works with any OpenAI-compatible endpoint)
Step 1: Set Up the Client
// router.ts
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.QUBAX_API_KEY,
baseURL: "https://api.qubax.ai/v1",
});
// Pick models for your workload from https://qubax.ai/models
const CHEAP_MODEL = "gpt-5.6-luna"; // ~$0.017/$0.069 per 1M tokens
const FLAGSHIP_MODEL = "gpt-5.6-sol"; // ~$0.235/$0.94 per 1M tokensThe key insight: because the two models are ~13x apart in price, every request that doesn't escalate saves ~92%.
Step 2: Build the Grader
The grader is a second, tiny LLM call that judges the draft. It must return structured JSON so your code can branch reliably:
type GraderVerdict = {
pass: boolean;
reason: string;
confidence: number; // 0-1
};
export async function gradeDraft(
task: string,
draft: string
): Promise<GraderVerdict> {
const res = await client.chat.completions.create({
model: CHEAP_MODEL, // the grader can itself be cheap
messages: [
{
role: "system",
content: `You are a strict QA reviewer. Judge whether the DRAFT
fully satisfies the TASK. Check: (1) it answers exactly what was asked,
(2) format requirements are met, (3) no fabricated facts or code bugs.
Respond with JSON only:
{"pass": boolean, "reason": string, "confidence": number}`,
},
{ role: "user", content: `TASK:\n${task}\n\nDRAFT:\n${draft}` },
],
response_format: { type: "json_object" },
temperature: 0,
});
return JSON.parse(res.choices[0].message.content ?? "{}");
}Why a grader instead of asking the cheap model "how confident are you?" — because self-reported confidence is notoriously unreliable. A separate reviewer call with a fresh context catches real failures far better, and at cheap-tier prices it costs fractions of a cent.
Step 3: Build the Router
export type RouteResult = {
answer: string;
usedModel: string;
escalated: boolean;
attempts: number;
};
export async function routeRequest(task: string): Promise<RouteResult> {
// Stage 1: cheap attempt
const cheap = await client.chat.completions.create({
model: CHEAP_MODEL,
messages: [{ role: "user", content: task }],
temperature: 0.3,
});
const draft = cheap.choices[0].message.content ?? "";
// Stage 2: grade
const verdict = await gradeDraft(task, draft);
if (verdict.pass && verdict.confidence >= 0.75) {
return { answer: draft, usedModel: CHEAP_MODEL, escalated: false, attempts: 1 };
}
// Stage 3: escalate to flagship
console.log(`Escalating: ${verdict.reason} (confidence ${verdict.confidence})`);
const flagship = await client.chat.completions.create({
model: FLAGSHIP_MODEL,
messages: [
{
role: "user",
content: `${task}\n\n(A previous attempt was rejected for: ${verdict.reason}.
Previous draft, for reference only:\n${draft})`,
},
],
temperature: 0.3,
});
return {
answer: flagship.choices[0].message.content ?? "",
usedModel: FLAGSHIP_MODEL,
escalated: true,
attempts: 2,
};
}Step 4: Track the Economics
You can't optimize what you don't measure. Add usage logging:
// usage.ts
export function logUsage(result: RouteResult, inTok: number, outTok: number) {
const prices: Record<string, [number, number]> = {
[CHEAP_MODEL]: [0.017, 0.069],
[FLAGSHIP_MODEL]: [0.235, 0.94],
};
const [pIn, pOut] = prices[result.usedModel];
const cost = (inTok / 1e6) * pIn + (outTok / 1e6) * pOut;
console.log(
`${result.usedModel} escalated=${result.escalated} cost=$${cost.toFixed(6)}`
);
}Step 5: Test It
// Two tasks at opposite ends of the difficulty spectrum
await routeRequest("Extract the email addresses from this text: ...");
await routeRequest("Design a distributed rate limiter with exactly-once semantics...");Expected behavior: the extraction task passes the cheap tier; the distributed-systems question escalates. Run this against 50–100 of your real prompts and log the escalation rate — that number is your savings. An 85% pass rate on the cheap tier with a 13x price gap means roughly ~70% savings overall, including grader overhead.
Production Hardening
Before this goes live, add:
- Timeouts and retries on both tiers, with exponential backoff.
- A maximum escalation depth (cap at 2 tiers unless you enjoy surprises).
- Per-task routing rules: simple regex-able tasks (format conversions, extractions) can skip the cheap model's grader entirely.
- Caching: identical prompts should never cost you money twice. A semantic cache in front of the router is free money.
- Streaming: the flagship retry can stream to the user while the cheap draft is shown as "refining…" for perceived latency.
Choosing Your Tiers
The right pair depends on your workload. Good rules of thumb:
- Cheap tier: Flash-class models — GLM 5.3 Flash, DeepSeek V4.1 Flash, GPT-5.6 Luna, Gemini Flash variants. Look for sub-$0.10/1M input pricing.
- Flagship tier: Claude Opus 5, GPT-5.6 Sol/Terra, Gemini 3.x Pro — for genuinely hard reasoning and agentic loops.
- Grader: same cheap tier as stage 1, at temperature 0.
Browse live pricing for all of these on Qubax's model marketplace — wholesale competition between providers means the same model often costs several times less than retail, which makes the cheap tier even cheaper.
Conclusion
The cascade router is the highest-leverage cost optimization in applied AI: ~30 minutes of work for 70–95% savings on typical workloads. Cheap model first, structured grading, honest escalation — and log everything so your savings are measurable, not anecdotal.
Try the models from this tutorial on Qubax → qubax.ai/models, and check the API docs to plug in your key.
FAQ
How much can I actually save with a router?
On workloads where 70–90% of requests are routine, savings of 70–95% are typical, even after accounting for grader calls. The exact number depends on your escalation rate — measure it with real prompts.
Doesn't the grader call add cost and latency?
Yes, but the grader runs on the cheap tier and costs a small fraction of one flagship call. Net savings stay strongly positive as long as most requests pass the cheap tier.
What if the cheap model is confidently wrong?
That's the main risk of any cascade. Mitigate it by making the grader check factual grounding for high-stakes tasks, or by routing regulated/safety-critical traffic straight to the flagship tier.
Can I use this with any provider?
Yes — the pattern only needs an OpenAI-compatible chat completions endpoint, which Qubax provides for every model on the marketplace with a single API key.
Should I route hard tasks directly to the flagship?
If you can classify difficulty up front (e.g., by prompt length, task type, or user tier), bypass the cheap tier for known-hard traffic. The router still handles the ambiguous middle.
Variations on the Cascade Pattern
Once the basic router works, several upgrades are worth considering:
1. Task-type routing (before the first call)
Use a classifier — even a keyword matcher — to pre-sort traffic: extractions and format conversions go cheap-tier-only; multi-step agent tasks go straight to the flagship. This removes one grader call per simple task.
2. Majority-vote for medium stakes
For tasks where the cheap model sometimes fails but correctness matters (data extraction, structured output), generate 2–3 cheap drafts and have the grader pick the best. Three cheap calls still cost less than one flagship call, and disagreement itself is a useful escalation signal.
3. Feedback loops
Log every escalation and its reason. After a few weeks, patterns emerge — e.g., "requests over 2,000 tokens always escalate" — which you can convert into deterministic routing rules and eliminate wasted cheap attempts.
4. Per-customer tiers
Combine the router with user plans: free users get cheap-tier-only, paid users get the full cascade. This turns cost control into a pricing lever.
Common Pitfalls
- Grader drift: if you later swap the cheap model, re-test the grader — a model change can silently shift pass rates.
- Over-strict graders: a grader that rejects everything inflates costs by escalating too much. Track your escalation rate weekly; 10–30% is a healthy band for mixed workloads, and anything above ~40% means your cheap tier is mis-picked or your grader is too harsh.
- Forgetting token accounting: the grader's input includes the whole draft, so long outputs double the cheap-tier cost. Factor that into your price math.
- One router for everything: different task types (code vs. chat vs. extraction) deserve different model pairs. Parameterize the router by task type rather than hardcoding one pair.
