How to Build an LLM-as-Judge Evaluation Pipeline in Python (Tutorial)

You just improved your system prompt. Did it actually help? If your answer is "I tried it a few times and it felt better," you are gambling, not engineering. Every serious LLM application eventually needs the same thing: a way to measure output quality automatically, on every change, without a room full of annotators.

That tool is the LLM-as-judge - a strong model scoring your application's outputs against a rubric you define. Done right, it turns prompt iteration from vibes into a regression suite. Done wrong, it produces confident-looking numbers that mean nothing.

This tutorial builds a complete evaluation pipeline in Python: a golden dataset, a judge with a structured rubric, scoring, and a CLI to run it all. By the end you will have a harness you can run before every deploy.

The full code is below, and you can run it against any model in the Qubax catalog by changing one string.

Why you need automated evaluation

LLM outputs are non-deterministic and open-ended. Classic unit tests cannot check "is this a good summary." Your options are:

Human review - gold standard, does not scale past a dozen changes per week.
Exact-match / heuristic checks - work for structure ("did it output valid JSON"), blind to quality.
LLM-as-judge - a model reads output + rubric, returns a score. Orders of magnitude cheaper than humans, consistent enough to compare runs, and correlates surprisingly well with human judgment when the rubric is concrete.

The judge pattern slots into three places in a development loop:

Prompt iteration - score candidate prompts against the golden set; ship the winner.
Model migration - run the same eval when swapping GPT for Claude or a budget model; know the cost of the switch.
Production monitoring - sample real traffic, score it, alert when quality drifts.

Architecture: four components

code

golden_dataset.json  ->  candidate (your app logic + prompt)  ->  judge (strong model + rubric)  ->  report

Golden dataset: 30-100 real inputs with ideal answers or checklists. This is the foundation - garbage here poisons everything downstream.
Candidate: the system you are testing. In this tutorial, a summarization function; substitute your RAG pipeline, agent, or classifier.
Judge: a strong model with a strict, structured prompt that outputs scores in a parseable format.
Report: aggregate scores, per-example results, and the diff against the last run.

Step 1: The golden dataset

Aim for real inputs, not synthetic ones. Twenty real examples beat two hundred invented ones. Each entry has the input, context, and a reference answer or checklist:

json

[
  {
    "id": "faq-001",
    "input": "Summarize this support ticket: Customer upgraded to Pro plan yesterday, was charged twice, wants a refund for the duplicate charge.",
    "reference": "Customer double-charged after Pro upgrade; requests refund of duplicate.",
    "checklist": ["mentions double charge", "mentions refund request"]
  },
  {
    "id": "faq-002",
    "input": "Summarize: User reports API returning 429 errors since 9am UTC, only on the /v2/generate endpoint, other endpoints fine.",
    "reference": "Rate-limit errors on /v2/generate since 09:00 UTC; other endpoints unaffected.",
    "checklist": ["mentions 429/rate limit", "identifies affected endpoint"]
  }
]

Keep a private holdout set that you never tune against. When your numbers plateau on the dev set, the holdout tells you whether you actually improved or just memorized.

Step 2: The judge prompt

Judge quality lives and dies on rubric concreteness. Vague rubrics ("is it good?") give you noise. Each dimension gets a strict definition and anchored score levels:

python

JUDGE_SYSTEM = "You are a strict, impartial evaluation judge. You score AI outputs against a rubric. You are not the author; be critical. Score each dimension 1-5: accuracy (does the output state only facts supported by the input/reference? 5 = every claim verifiable, 1 = fabricates), completeness (are the key points from the checklist covered? 5 = all, 3 = most, 1 = critical omissions), concision (free of padding? 5 = tight, 1 = bloated). Respond with ONLY valid JSON: {\"accuracy\": <int>, \"completeness\": <int>, \"concision\": <int>, \"reasoning\": \"<one sentence>\"}"

Two rules that separate usable judges from toys:

Force reasoning before the score. Ask for brief justification in the same response - models self-correct when they must articulate why.
Force structured output. JSON-only responses make parsing reliable and scores comparable across runs.

Step 3: The pipeline

Here is the complete implementation, using the OpenAI-compatible interface that works with Qubax AI and any standard SDK:

python

import json, os, statistics
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["QUBAX_API_KEY"],
    base_url="https://api.qubax.ai/v1"
)

CANDIDATE_MODEL = "gpt-5.6-luna"    # cheap model under test
JUDGE_MODEL     = "claude-opus-4-6"  # strong model as judge

def summarize(text):
    resp = client.chat.completions.create(
        model=CANDIDATE_MODEL,
        messages=[
            {"role": "system", "content": "Summarize support tickets in one sentence. Facts only."},
            {"role": "user", "content": text},
        ],
        temperature=0.2,
    )
    return resp.choices[0].message.content

def judge(input_text, output, reference):
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM},
            {"role": "user", "content": json.dumps({
                "input": input_text, "output": output, "reference": reference
            })},
        ],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

def run_eval(dataset_path):
    dataset = json.load(open(dataset_path))
    results = []
    for ex in dataset:
        output = summarize(ex["input"])
        scores = judge(ex["input"], output, ex["reference"])
        scores["id"] = ex["id"]
        results.append(scores)

    report = {
        "n": len(results),
        "mean_accuracy": round(statistics.mean(r["accuracy"] for r in results), 2),
        "mean_completeness": round(statistics.mean(r["completeness"] for r in results), 2),
        "mean_concision": round(statistics.mean(r["concision"] for r in results), 2),
    }
    report["overall"] = round(
        (report["mean_accuracy"] * 0.5
         + report["mean_completeness"] * 0.3
         + report["mean_concision"] * 0.2), 2
    )
    report["results"] = results
    return report

if __name__ == "__main__":
    report = run_eval("golden_dataset.json")
    print(json.dumps({k: v for k, v in report.items() if k != "results"}, indent=2))
    lows = [r for r in report["results"] if r["accuracy"] <= 2]
    print(f"{len(lows)} low-accuracy examples - inspect these first:")
    for r in lows:
        print(f"  {r['id']}: {r['reasoning']}")

Run it:

bash

pip install openai
export QUBAX_API_KEY="your-key"
python eval.py

Output looks like:

code

{
  "n": 50,
  "mean_accuracy": 4.41,
  "mean_completeness": 4.12,
  "mean_concision": 4.63,
  "overall": 4.34
}
3 low-accuracy examples - inspect these first:
  faq-017: invents a refund amount not present in ticket

Now "did my prompt change help?" is a one-command question with a numeric answer.

Step 4: Make the judge trustworthy

A judge that mirrors your candidate's errors is worse than no judge. Harden it:

Cross-vendor judging. If your candidate is a GPT model, judge with Claude (as above); if the candidate is open-weight, judge with a frontier model. Shared training biases correlate errors and inflate scores. On Qubax, switching judge model is a one-line change.

Validate against humans. Hand-score 20 examples yourself. If judge-human agreement is under roughly 80 percent, your rubric is too vague - tighten the anchors until it rises.

Watch position and verbosity bias. Judges favor longer answers and the first option in comparisons. Fixed rubric position and explicit concision scoring (as above) mitigate both.

Temperature 0 for the judge, always. You are building a measurement instrument, not a creative partner. Determinism within runs is the entire point.

Track cost. Judging 50 examples with a frontier model is cheap (pennies to a few dollars depending on length), but production monitoring at scale adds up. A common pattern: frontier judge on the 1 percent sample that triggers alerts, budget model judge for routine scoring.

Step 5: Wire it into CI

The eval becomes a deployment gate with a few lines in your CI config:

yaml

- name: Run LLM eval suite
  run: |
    python eval.py > report.json
    OVERALL=$(python -c "import json; print(json.load(open('report.json'))['overall'])")
    python -c "import sys; sys.exit(0 if float('$OVERALL') >= 4.0 else 1)"

Now a prompt regression fails the build instead of reaching production. This is the moment evals stop being an experiment and become infrastructure.

Cost engineering: judge smart, not big

Since judges run on every change, their cost compounds. Three levers:

Judge dimensions selectively. Accuracy on every run; concision weekly.
Cascade judges. A budget model judges everything; anything scoring in the ambiguous 3-range gets escalated to a frontier judge. Typically cuts judge spend 5-10x with negligible accuracy loss.
Exploit pricing spreads. Judge models at retail can be expensive; routed through Qubax, the same frontier judges cost a fraction of list price, and budget candidates like GPT-5.6 Luna cost less than a coffee per full eval run. Check current pricing at qubax.ai/models.

Common pitfalls

Tuning against the judge. If you optimize prompts specifically to please the judge, you have built an expensive random number generator. Keep the human-validated holdout.
One giant rubric. Five well-defined dimensions beat fifteen fuzzy ones. Split concerns (safety, factuality, style) into separate judge calls if needed.
Ignoring variance. Rerun the eval 3 times; if overall swings more than 0.15 between identical runs, your dataset is too small or the rubric too vague.
Evaluating in the wrong context. Judge the exact outputs your production system produces, including your real system prompt and tools - not a simplified demo version.

Conclusion

Automated evaluation is the difference between an LLM demo and an LLM product. The judge pattern gives you a quality signal that is cheap enough to run constantly and stable enough to act on. Start small: thirty golden examples, two rubric dimensions, one judge model, and a CI gate. Grow the dataset as real failures teach you what to measure.

Want to put this into practice? Grab an API key at Qubax AI and point the pipeline above at any model in the catalog - budget candidates for your system, frontier models for the judge, one integration for all of it.

FAQ

What is LLM-as-judge?

Using a strong language model to automatically score the outputs of another model against a rubric you define. It replaces slow human review with a cheap, consistent, automated quality signal you can run on every change.

Which model should be the judge?

A model stronger than the one you are testing, ideally from a different vendor to avoid correlated biases - for example, judge a GPT candidate with Claude. On Qubax you can switch judge models with a one-line change.

How accurate are LLM judges?

With a concrete rubric and structured scoring, well-built judges typically reach 75-85 percent agreement with human raters. Validate yours by hand-scoring 20 examples; below 80 percent agreement means your rubric needs tightening.

How many examples do I need in a golden dataset?

Start with 30-50 real examples and grow from there. Rerun the eval on identical code - if overall scores swing more than 0.15, add examples or sharpen the rubric.

How much does an eval pipeline cost?

Far less than most teams expect: judging 50 short examples with a frontier model typically costs pennies to a few dollars at retail - and less when routed through Qubax. Cascading to a budget judge for routine runs cuts it further.

Does this replace human review?

No - it complements it. Automate the routine 95 percent so your humans review the interesting failure cases, validate the judge periodically, and keep a holdout set only humans ever score.