Building an LLM-as-Judge Evaluation Pipeline with Gemini — Production-Grade Design and Implementation

Implementation 1: Pointwise judge

Here's a minimal Pointwise judge using the google-genai SDK with Gemini 2.5 Pro. The single most important detail is to never let the judge respond in free text — always force structured JSON output. Asking for "just a number" is brittle; the model will sneak in explanations and you'll write fragile parsers.

# eval_pointwise.py
# Pointwise judge: score factuality / politeness / specificity from 0-5
# Required env: GEMINI_API_KEY
import os
import json
from google import genai
from google.genai import types
from pydantic import BaseModel, Field
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
class JudgeScore(BaseModel):
    factuality: int = Field(ge=0, le=5, description="Factual correctness 0-5")
    politeness: int = Field(ge=0, le=5, description="Politeness 0-5")
    specificity: int = Field(ge=0, le=5, description="Specificity 0-5")
    rationale: str = Field(description="Brief reason, max 80 chars")
 
JUDGE_PROMPT = """You are a strict evaluator. Score the assistant's response to the user question on three axes from 0-5.
 
Axes:
- factuality: Is the response factually correct? If unverifiable, default to 3.
- politeness: Is the tone respectful and professional?
- specificity: Does the response include concrete examples, numbers, or code?
 
User question:
{question}
 
Assistant response:
{answer}
 
Return JSON only. No commentary."""
 
def judge_pointwise(question: str, answer: str) -> JudgeScore:
    prompt = JUDGE_PROMPT.format(question=question, answer=answer)
    try:
        resp = client.models.generate_content(
            model="gemini-2.5-pro",
            contents=prompt,
            config=types.GenerateContentConfig(
                temperature=0.0,  # deterministic judging
                response_mime_type="application/json",
                response_schema=JudgeScore,
                max_output_tokens=512,
            ),
        )
        return JudgeScore.model_validate_json(resp.text)
    except Exception as e:
        # Don't silently null-out failures. Log + neutral fallback.
        print(f"[judge error] {type(e).__name__}: {e}")
        return JudgeScore(factuality=3, politeness=3, specificity=3, rationale="judge_failed")
 
if __name__ == "__main__":
    q = "Show me the smallest example of calling Gemini API from Cloudflare Workers."
    a = "Just use fetch."
    score = judge_pointwise(q, a)
    print(json.dumps(score.model_dump(), indent=2))

Three details matter here. First, temperature=0.0 makes the judge deterministic — without that, the same input yields different scores on every call, which kills any time-series tracking. Second, response_schema lets the SDK build a JSON Schema for you, so the model's output is validated against your Pydantic model and you don't need to beg the prompt for "JSON only." Third, on judge failure, fall back to a neutral midpoint and log loudly rather than silently returning null. Otherwise your aggregate will be full of NaNs and the average will lie to you.

Expected output:

{
  "factuality": 2,
  "politeness": 2,
  "specificity": 1,
  "rationale": "fetch alone insufficient — missing endpoint, auth, model"
}

A subtle point about temperature=0.0: it does not literally mean "always produce identical output" with frontier LLMs — there's still a small amount of nondeterminism from sampling tie-breaks and infrastructure-level caching. In practice, identical inputs produce identical outputs more than 95% of the time at temperature 0, which is enough for stable trend tracking. If you need bit-exact reproducibility for compliance reasons, store the judge's output alongside the scored case so you can replay it later.

Implementation 2: Pairwise judge

For A/B comparisons, the judge picks "A wins," "B wins," or "tie." Pairwise tends to agree with humans more often than Pointwise, but it has a stubborn position bias: the response shown first is more likely to be picked. The fix is to randomize the order on every call and undo the swap when interpreting the verdict.

# eval_pairwise.py
# Pairwise judge with position-bias mitigation
import os
import random
from typing import Literal
from google import genai
from google.genai import types
from pydantic import BaseModel, Field
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
class PairwiseVerdict(BaseModel):
    winner: Literal["A", "B", "TIE"]
    rationale: str = Field(max_length=200)
 
PAIRWISE_PROMPT = """You compare assistant responses. Two responses to the same question are shown below. Decide which is better.
 
Criteria (weighted equally):
1. Factual correctness
2. Directness in answering the question
3. Adequate but not excessive specificity (penalize filler and repetition)
 
Question:
{question}
 
Response A:
{a}
 
Response B:
{b}
 
Pick A, B, or TIE."""
 
def judge_pairwise(question: str, ans_a: str, ans_b: str) -> PairwiseVerdict:
    # Position-bias mitigation: 50% chance to swap A and B
    swap = random.random() < 0.5
    a_text, b_text = (ans_b, ans_a) if swap else (ans_a, ans_b)
 
    prompt = PAIRWISE_PROMPT.format(question=question, a=a_text, b=b_text)
 
    try:
        resp = client.models.generate_content(
            model="gemini-2.5-pro",
            contents=prompt,
            config=types.GenerateContentConfig(
                temperature=0.0,
                response_mime_type="application/json",
                response_schema=PairwiseVerdict,
                max_output_tokens=400,
            ),
        )
        verdict = PairwiseVerdict.model_validate_json(resp.text)
 
        # Undo the swap if needed
        if swap and verdict.winner == "A":
            return PairwiseVerdict(winner="B", rationale=verdict.rationale)
        if swap and verdict.winner == "B":
            return PairwiseVerdict(winner="A", rationale=verdict.rationale)
        return verdict
    except Exception as e:
        print(f"[pairwise judge error] {type(e).__name__}: {e}")
        return PairwiseVerdict(winner="TIE", rationale="judge_failed")

That random swap is just a few lines, but in my experience it lifts Pairwise reliability by twenty percentage points or more. It's a well-known trick from the MT-Bench paper and various Anthropic eval write-ups, but it's easy to forget when you're scaffolding your own pipeline — so I leave it explicitly in the code as a reminder. There's also a stronger variant: run the judge twice with deterministic ordering (A then B, then B then A), and only declare a winner when both runs agree. This costs 2x but eliminates positional bias almost entirely. I use the cheap randomized version for daily evaluation and the double-pass for high-stakes release gates.

Three biases that always show up

Run an LLM judge in production for any length of time and you'll meet these three.

Position bias

In Pairwise, the first response shown wins more often than it should. The randomization above takes most of the sting out. To fully eliminate it, you can run the judge twice — once in each order — and only declare a winner when both agree. That doubles the cost, so I save the double-judge pattern for high-stakes release gates. A useful sanity check is to compute the win rate of "A always shown first" vs "B always shown first" on a small held-out sample; if those win rates differ by more than 5 percentage points, your judge has noticeable position bias and you need at least the random-swap mitigation.

Length bias

The judge rewards longer responses, especially in Pointwise. Two countermeasures: first, explicitly tell the judge that verbosity is a deduction in the prompt. Second, log the response length alongside the score and watch for a tightening linear relationship — if you suddenly see "average length ≈ average score × 200," your judge has stopped reading content and started measuring volume. A more robust mitigation is to normalize: compute the score's residual after regressing on response length, and track that residual over time instead of the raw score. This way, a global shift in response verbosity (which can happen when a system prompt changes) doesn't masquerade as a quality shift.

Self-preference bias

When the judge and the system under test are from the same model family, the judge tends to favor its own outputs. Gemini judging Gemini exhibits this. The cleanest fix is to use a different family for the judge (e.g., Gemini outputs evaluated by GPT, and vice versa). When budget or operations forbid that, measure the judge against human raters every quarter and watch for the gap widening. A practical compromise is to use the same family but a different size — for example, evaluating Gemini Flash outputs with Gemini Pro. The size difference creates enough independence to dampen self-preference, and you keep the operational simplicity of staying within one provider.

Productionizing the pipeline

Single-call code is the easy part; the daily-regression and release-gate workflows are where teams stall. Here's a simplified version of the structure I run in practice.

# eval_pipeline.py
# Run a dataset through the judge concurrently; persist results
import asyncio
import os
import sqlite3
import time
from typing import List
from google import genai
from google.genai import types
from pydantic import BaseModel
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
class EvalRecord(BaseModel):
    case_id: str
    question: str
    answer: str
 
async def judge_async(record: EvalRecord, semaphore: asyncio.Semaphore) -> dict:
    """Evaluate a single case under a concurrency cap."""
    async with semaphore:
        for attempt in range(3):
            try:
                resp = await asyncio.to_thread(
                    client.models.generate_content,
                    model="gemini-2.5-pro",
                    contents=JUDGE_PROMPT.format(question=record.question, answer=record.answer),
                    config=types.GenerateContentConfig(
                        temperature=0.0,
                        response_mime_type="application/json",
                        response_schema=JudgeScore,
                        max_output_tokens=512,
                    ),
                )
                score = JudgeScore.model_validate_json(resp.text)
                return {"case_id": record.case_id, **score.model_dump()}
            except Exception as e:
                # Backoff on 429 / 503 and retry
                wait = 2 ** attempt
                print(f"[retry {attempt+1}/3] {record.case_id}: {e} — wait {wait}s")
                await asyncio.sleep(wait)
        # 3 strikes: return a missing-data marker
        return {"case_id": record.case_id, "factuality": 3, "politeness": 3, "specificity": 3, "rationale": "max_retry"}
 
async def run_pipeline(dataset: List[EvalRecord], concurrency: int = 8) -> List[dict]:
    sem = asyncio.Semaphore(concurrency)
    return await asyncio.gather(*[judge_async(r, sem) for r in dataset])
 
def save_to_sqlite(results: List[dict], path: str = "eval_results.db") -> None:
    conn = sqlite3.connect(path)
    conn.execute("""CREATE TABLE IF NOT EXISTS scores (
        case_id TEXT, ts INTEGER, factuality INT, politeness INT,
        specificity INT, rationale TEXT
    )""")
    ts = int(time.time())
    conn.executemany(
        "INSERT INTO scores VALUES (?, ?, ?, ?, ?, ?)",
        [(r["case_id"], ts, r["factuality"], r["politeness"], r["specificity"], r["rationale"]) for r in results],
    )
    conn.commit()
    conn.close()

Tune concurrency to your Gemini API tier — 8 is comfortable on Tier 1, and you can push to 16–32 on Tier 2+. Higher concurrency means more 429s, so the exponential-backoff retry isn't optional. In one project, concurrency 16 chewed through 1,000 cases in roughly eight minutes for under $2 of API spend.

A few production-grade extensions you'll want once this scaffold is running. First, write to a real database (Postgres or BigQuery) instead of SQLite — you'll quickly want to join scores against deployment metadata, prompt versions, and rater info. Second, attach a "run id" to every batch so you can compare runs. Third, set up a simple dashboard (Looker Studio, Grafana, or a one-page Streamlit app) that plots score trends over time, broken down by category. Even an ugly dashboard will surface drift faster than scrolling through a database. Fourth, alert on aggregate movement: if the median factuality score drops by more than 0.3 points week-over-week, page someone.

Measuring the judge itself

The thorniest question with LLM-as-Judge is "how much can I trust this judge?" The most direct answer is to periodically measure agreement with human raters using accuracy or rank correlation (Spearman / Cohen's kappa).

A workable cadence:

1. Sample 100–200 cases from your normal evaluation set, stratified across categories.
2. Have humans score them on the same axes — at least two raters, with disagreements adjudicated by a third.
3. Compute correlation between judge scores and human scores. For Pointwise scores, Spearman's rank correlation is usually the right metric. For Pairwise, weighted Cohen's kappa or simple agreement rate.
4. If correlation drops noticeably from last quarter, redesign the judge prompt and run the calibration again.

In my experience, a Spearman rank correlation above 0.6 is enough to keep using the judge for regression tracking. Below 0.4, it's a sign the axis you're trying to score isn't well-suited to LLM judging — at that point I split the axis into finer-grained sub-axes or switch to Pairwise. There's also a more sophisticated technique called "ranking-based calibration" where you periodically re-anchor the judge against a small set of human-labeled cases, but for most teams the quarterly check is enough; over-engineering this loop tends to consume more time than it saves.

One operational tip: keep the calibration set strictly separate from the regression set. If they overlap, the judge can effectively memorize the calibration cases and the correlation number will be inflated. I usually keep two non-overlapping samples drawn from production traffic.

Cost optimization

LLM-as-Judge calls add up fast. A few combinable techniques:

• Tiered model use: Don't run all cases through 2.5 Pro. Use Flash as a first-pass filter, then re-judge only the borderline ones with Pro. In one of my projects this cut the monthly bill by 65–75%.
• Context caching: If your judge prompt has a long fixed preamble (system instructions, evaluation rubric), use Gemini's Context Caching so the preamble is billed at the cache rate.
• Batched questions: Group similar questions per request — be careful here, as it amplifies length bias and you'll need to validate that scores remain comparable to single-case scoring.
• Sampling: Skip full-coverage evaluation. Compute the sample size needed for a reasonable confidence interval and stop there.

The Flash-then-Pro tiered pattern is the easiest win — small implementation cost, large bill reduction — so it's the first knob I reach for when costs creep up. The implementation is mechanical: run all cases through Flash, mark anything with a score in the middle of the scale (say 2 or 3 on a 0–5 axis) as "uncertain," and re-run only those through Pro. The intuition is that high and low scores are usually robust across model sizes; the disagreements happen in the middle. In practice, only 15–25% of cases land in the uncertain zone, so the Pro spend is similarly reduced.

Context caching deserves special attention if your judge prompt has a long fixed preamble. A typical evaluation rubric I've used was about 1,500 tokens; cached, that costs roughly 25% of the per-token rate. Multiply that by thousands of evaluation calls per week and the savings are substantial.

Common mistakes I've made

The traps I've personally hit, in rough order of how often they bite people.

Trap 1: Criteria defined only in words

A prompt that says "score politeness from 0 to 5" leaves the judge guessing what 4 vs 3 means each time. Anchor each score level with a concrete one-line example: "4 = polite but uses generic templates; 5 = polite and visibly tailored to the user's situation." This single change tightens score variance noticeably. The technical name for this pattern is "anchored rubric" and it's one of the cheapest interventions you can make — a few hundred extra tokens in the prompt for a meaningful drop in score variance.

Trap 2: Trusting the rationale

Judges produce surprisingly creative rationale strings to justify whatever score they emitted. They post-hoc rationalize. It's fine to surface the rationale in dashboards for spot-checking, but never let the rationale drive a decision — let the numerical aggregates do that. A useful sanity check: occasionally take a small batch of cases, modify the response slightly to be obviously worse, re-judge, and compare the rationale. If the rationale doesn't change much even though the response did, you're seeing post-hoc rationalization in action.

Trap 3: Drifting evaluation sets

If you keep adding cases to your eval dataset week over week and removing old ones, you can't compare scores across time. Split your sets: a "regression" dataset that stays frozen for six to twelve months, and a "feature evaluation" dataset that's rebuilt per release. This separation is what makes long-term quality tracking possible. In one team I worked with, we discovered after six months that we had been comparing scores from different eval sets without realizing it, and our supposedly improving quality trend was an artifact of an easier dataset replacing a harder one. Don't let this happen.

Trap 4: Ignoring inter-rater agreement during calibration

When you have humans score the calibration set, you'll find the humans don't fully agree with each other either. If your raters' inter-rater agreement is, say, 70%, you cannot expect your LLM judge to exceed 70% agreement with any single rater — that's an upper bound set by human disagreement itself. Always measure inter-rater agreement first and use it as the ceiling against which you compare the judge.

Trap 5: Treating "TIE" as failure in Pairwise

In Pairwise, ties are informative — they tell you the change had no measurable effect. Some teams treat any tie as judge failure and re-run with different settings until they get a winner. Don't. If your judge says TIE on 40% of cases for a particular axis, that's the signal that the axis isn't sensitive enough to detect the change you made. Either redesign the axis or accept the result.

Designing the judge prompt

The judge prompt is the single most leveraged artifact in this whole system — small wording changes can shift scores by entire percentage points of correlation with humans. I don't have a one-size-fits-all template, but I've settled on a small set of patterns that consistently work better than the alternatives.

Lead with the role and the strictness level. "You are a strict evaluator" outperforms "You are an evaluator" in nearly every test I've run. The strictness framing makes the judge less likely to give middle-of-the-scale benefit-of-the-doubt scores. If your domain has a known professional standard ("You are a senior software engineer reviewing pull requests"), naming it tightens score variance further.

Define the scale with anchors, not adjectives. Rather than "5 = excellent, 4 = good, 3 = average," write "5 = the response is correct, complete, and demonstrably actionable; 4 = correct but missing one specific piece; 3 = partially correct or with one factual slip..." Anchored rubrics shave noise off your scores faster than any other prompt change.

Tell the judge what to ignore explicitly. If you don't want the judge influenced by response length, write "Length is not a quality signal — do not reward longer responses." If you don't want it influenced by formatting, write "Markdown formatting is not part of the score." Models genuinely follow these "do not consider X" instructions when they're stated explicitly, but they default to considering everything if you don't.

End with a re-statement of the output format. Even when you're using response_schema, putting a brief reminder ("Return JSON only with the keys factuality, politeness, specificity, rationale.") at the end of the prompt is an extra layer of safety. The structured output enforcement in the SDK is robust, but defense in depth costs almost nothing.

A small case study to make this concrete. On one project, our initial Pointwise prompt produced Spearman correlation of 0.41 against human ratings — borderline unusable. The interventions that lifted us above 0.65 over about three weeks were: (1) anchored rubric examples for each score level, (2) an explicit "verbose responses are not better" instruction, (3) a calibration set of 50 cases used to A/B test prompt variations weekly, and (4) switching from a generic "evaluator" framing to a domain-specific "senior support engineer reviewing customer responses" framing. None of these are tricks — they're just deliberate prompt engineering applied to the prompt that does the judging, with the same care you'd apply to the prompt being judged.

Measuring judge reproducibility (flakiness)

Setting temperature=0.0 does not make a judge fully deterministic. Because of backend parallelism and small tokenization differences, the same input can score one point apart across runs. In the evaluation pipeline I run as an indie developer on my own apps, sending 1,000 cases through the judge five times at temperature=0 produced a ±1 score difference on roughly 3–5% of cases.

The danger is concluding "the score went up by 0.2" without knowing the floor noise. If your improvement is buried inside the flakiness band, that delta might just be noise. So before wiring a judge into production, I add a step that judges each case multiple times, measures agreement, and flags the cases the judge can't decide reliably.

# eval_flakiness.py
# Judge the same case N times and measure score flakiness.
# Output: per-case mode, agreement, spread, and an "unreliable" flag.
import os
import statistics
from collections import Counter
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
def judge_once(question: str, answer: str) -> int:
    resp = client.models.generate_content(
        model="gemini-2.5-pro",
        contents=JUDGE_PROMPT.format(question=question, answer=answer),
        config=types.GenerateContentConfig(
            temperature=0.0,
            response_mime_type="application/json",
            response_schema=JudgeScore,
            max_output_tokens=512,
        ),
    )
    return JudgeScore.model_validate_json(resp.text).factuality
 
def measure_flakiness(question: str, answer: str, n: int = 5) -> dict:
    scores = [judge_once(question, answer) for _ in range(n)]
    counter = Counter(scores)
    mode, mode_count = counter.most_common(1)[0]
    agreement = mode_count / n            # share of the most common score
    spread = max(scores) - min(scores)    # min-to-max range
    return {
        "mode": mode,
        "agreement": round(agreement, 2),
        "spread": spread,
        "stdev": round(statistics.pstdev(scores), 2),
        # Unreliable if agreement < 0.6 or the spread is 2 or more
        "unreliable": agreement < 0.6 or spread >= 2,
        "raw": scores,
    }
 
if __name__ == "__main__":
    q = "Show me the minimal code to call the Gemini API from a Cloudflare Worker."
    a = "POST to the endpoint with fetch and authenticate via x-goog-api-key; the model name goes in the URL."
    print(measure_flakiness(q, a, n=5))

Rather than silently dropping the unreliable cases from your averages, route them to a separate human-review queue. High-flakiness cases are usually a sign that the judge is genuinely torn — meaning the evaluation axis doesn't fit that case well. In practice, inspecting the flakiest cases is one of the most reliable ways I've found to discover where an axis needs to be split.

Operationally, judging every case N times multiplies cost by N. So run the flakiness measurement only on a 100–200 case sample before a release decision, then use the observed noise band as a threshold in your daily regression tracking: anything that moves within that band is treated as noise. Once you know the band, you stop overreacting to small score wiggles and your evaluation decisions settle down considerably.

Where to go next

LLM-as-Judge works best when you treat it not as a human-evaluator replacement but as a way to cover the volume and frequency of evaluation that human raters can't reach. Aim for "shows the same direction as humans" rather than "perfectly matches humans" and you'll avoid the most painful disappointment.

The single concrete next step I'd recommend: take the Pointwise implementation here, run it against a small set of about 50 examples, and measure agreement with your own scoring. If you get above 60% agreement on top-2 categories, the judge is worth investing in. If not, that's a useful signal too — either split the axes finer or switch to Pairwise from the start. A weekend of work to get this scaffold in place will pay back through the next several prompt-iteration cycles.

If you want to go deeper on evaluation design, pair this with Prompt Evaluation and Optimization Pipeline for the Gemini API and Quality Monitoring with a Golden Dataset and an LLM Judge for a clearer view of the whole operations phase.