How to Evaluate LLMs Before Production: A Practical Guide (With Code)

Shipping an LLM feature on vibes is how you end up with angry users and a rollback at 2 AM. Here's a practical, code-first evaluation workflow you can run in an afternoon — before your users find the failure modes for you.

GitHub's engineering team recently published guidance on this exact problem, and their advice matches what production teams learn the hard way: evaluate before production, with datasets, scorers, and a repeatable pipeline. This article turns that into a concrete tutorial.

Why "I Tried It a Few Times" Isn't Evaluation

Manual testing fails for three reasons:

1. Sample size. Ten prompts tell you almost nothing about 100,000 monthly requests.
2. No regression detection. A new model version can silently break the exact case that worked last week.
3. No comparison rigor. "Model B felt better" is not a decision; "Model B won 71% of blind pairwise judgments on my dataset" is.

Evaluation turns model selection into engineering. The three pillars: a dataset, a scoring method, and automation.

Step 1: Build a Golden Dataset (50–200 Examples)

Collect real inputs your app will see, and for each one define what "correct" looks like. Sources: your support tickets, your logs, edge cases users complain about, plus adversarial examples (prompt injection, empty inputs, non-English).

Save as JSONL:

{"id": 1, "input": "Summarize this refund policy for a 10-year-old", "context": "<policy text>", "must_contain": ["30 days", "receipt"], "must_not_contain": ["legal jargon"]}
{"id": 2, "input": "My invoice says I was charged twice", "context": "", "must_contain": ["apolog"], "must_not_contain": []}

Tips:

• Cover your failure history — every bug a user has reported becomes a test case.
• Keep 20% of the dataset "held out" so you don't overfit to your own tests.
• Version it in git like code.

Step 2: Pick Scorers That Match the Task

There are three families of automated checks:

Deterministic checks (fast, free, reliable): does the output contain required strings, valid JSON, correct format, no banned words, within latency/cost budget?

LLM-as-judge: a strong model grades outputs against a rubric. Great for tone, helpfulness, and summarization quality. Calibrate it against ~30 human-labeled examples first.

Pairwise comparison: for model selection, showing the judge two outputs and asking "which is better?" is more reliable than absolute scoring.

Step 3: Wire It Up in Python

We'll evaluate three candidate models through one OpenAI-compatible endpoint. On Qubax AI you can swap model= between providers without changing anything else.

import json, os, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["QUBAX_API_KEY"],
    base_url="https://api.qubax.ai/v1"
)

CANDIDATES = ["gpt-6-astra", "claude-opus-5", "glm-5.3"]

def load_cases(path="golden.jsonl"):
    return [json.loads(l) for l in open(path)]

def run_case(client, model, case):
    start = time.time()
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a concise support assistant."},
            {"role": "user", "content": f"{case['input']}\n\nContext:\n{case.get('context','')}"}
        ],
        temperature=0.2,
    )
    return {
        "output": resp.choices[0].message.content,
        "latency_s": round(time.time() - start, 2),
        "tokens_in": resp.usage.prompt_tokens,
        "tokens_out": resp.usage.completion_tokens,
    }

def check_rules(case, out):
    text = out["output"].lower()
    ok = all(k.lower() in text for k in case["must_contain"])
    ok &= not any(b.lower() in text for b in case["must_not_contain"])
    return ok

results = {m: [] for m in CANDIDATES}
for model in CANDIDATES:
    for case in load_cases():
        out = run_case(client, model, case)
        out["passed"] = check_rules(case, out)
        results[model].append(out)

for model, rows in results.items():
    passed = sum(r["passed"] for r in rows) / len(rows)
    avg_lat = sum(r["latency_s"] for r in rows) / len(rows)
    avg_cost_tokens = sum(r["tokens_out"] for r in rows) / len(rows)
    print(f"{model:15} pass={passed:.0%}  latency={avg_lat:.2f}s  out_tokens={avg_cost_tokens:.0f}")

Run it for every candidate model. Now model selection is a table, not a feeling.

Step 4: Add an LLM Judge for Quality

For criteria rules can't check (helpfulness, tone), add a judge pass:

JUDGE_PROMPT = """Rate the assistant answer below from 1-5 on helpfulness for the user's question.
Respond with JSON: {"score": n, "reason": "..."}"""

def judge(case, output_text):
    resp = client.chat.completions.create(
        model="claude-opus-5",  # use your strongest model as judge
        messages=[{"role": "user", "content": f"{JUDGE_PROMPT}\n\nQuestion: {case['input']}\n\nAnswer: {output_text}"}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Calibration checklist for the judge:

• Label 30 outputs yourself; the judge should agree ≥80% of the time.
• Randomize answer order in pairwise mode to kill position bias.
• Never let a candidate model judge itself.

Step 5: Automate and Gate Releases

• Run the suite in CI on every prompt change, model upgrade, or provider swap.
• Set thresholds: "no model ships below 92% rule-check pass rate."
• Track cost per request — a model that's 2% better but 5x pricier is usually the wrong call. With Qubax pricing at a fraction of retail rates, the cost spread between candidates can be dramatic; check live numbers at qubax.ai/models.
• Re-run weekly against the same golden dataset so regressions surface immediately.

Common Mistakes

• Testing only happy paths. Half your dataset should be messy, adversarial, or ambiguous.
• One judge, one model. Judges are models; they have biases. Calibrate and vary them.
• Ignoring latency and cost. Users feel timeouts more than they feel a 2% quality delta.
• Never re-evaluating. Model providers update checkpoints constantly. Last quarter's winner can regress this month.

FAQ

How many test cases do I need?

Start with 50–100 real examples; 200+ once you're stable. Quality and coverage of cases matters far more than raw count.

Which model makes the best judge?

Use the strongest model you can afford — typically a frontier model like Claude Opus 5 or GPT-6 Astra — with temperature 0 and structured JSON output.

Can I evaluate models from different providers in one pipeline?

Yes — that's the point of an OpenAI-compatible gateway. On Qubax AI, changing one model= string swaps between GPT, Claude, Gemini, GLM, DeepSeek, and Kimi with identical code.

How do I evaluate an agentic (multi-step) application?

Score the final outcome against the goal, plus intermediate checks: did it call the right tools, respect permissions, stay within step budget? Tool-trace logs are your dataset.

Where do I get API access?

Create a key at Qubax AI, point your client at the OpenAI-compatible endpoint, and run the exact code above.