Quietly Catching Wrong Answers in Your Gemini-Powered App — A Production Auto-Eval Loop

A Concrete Judge Prompt

Each lens runs on a separate, cheaper model (Haiku-class or Gemini's smaller sibling) as the Judge. Here's my prompt for factuality:

You are an LLM response evaluator. You're given:
 
[Context]
{source_document}
 
[User question]
{user_query}
 
[Answer under review]
{model_answer}
 
Evaluate as follows:
1. List every factual claim made in the answer
2. For each claim, check whether it is contradicted by the context
3. Produce an overall verdict: "consistent" / "partially_inconsistent" / "inconsistent"
 
Quote evidence from the context whenever possible.
Do not guess. If the context doesn't contain the needed info,
return "insufficient_context".
 
Return JSON:
{
  "overall": "...",
  "claims": [
    {"text": "...", "verdict": "...", "evidence": "..."}
  ]
}

The key line is "do not guess, return insufficient_context if the info isn't there." Without it, the judge itself hallucinates. Once the judge starts inventing verdicts, the whole evaluation is corrupted.

Avoiding the Self-Evaluation Bias

If you generate with Gemini and judge with Gemini, the same model biases toward favorable verdicts of its own output. This is documented in research and I see it clearly in my own data.

My countermeasures:

• Use a different model family as the judge. Generate with Gemini, judge with a small Claude or GPT model.
• Two-judge majority vote. Only responses where the two judges disagree go to human review.
• Validate the judge weekly. Run it against a known-answer dataset to detect judge quality drift.

The second tactic has the best ROI. Always running two judges doubles cost, but restricting deep review to disagreement cases pushes anomaly-detection precision up dramatically at acceptable cost.

Sampling Strategy to Keep Cost Sustainable

Evaluating every response isn't economical. I layer sampling:

• Always evaluate: responses with anomalous length, containing specific keywords, or that received negative user feedback — 100% coverage
• Random sample: evaluate 1–5% of all responses (for daily reports)
• Deep-dive evaluation: any response judged inconsistent is re-evaluated on a bigger model

The first tier matters most. Outliers are already signal — sampling them loses information. The rest can be sampled aggressively without much loss.

With this tiered sampling, evaluation cost lands around 15% of generation cost. That's operable.

Typical Failure Modes the Loop Surfaces

Six months in, these are the Gemini failure patterns the loop surfaces most often:

• Slow instruction-following decay: as prompts accumulate additions over time, older instructions get ignored. Newer clauses tend to dominate.
• Factual errors at long context: the longer the context, the more the model confuses information from the first half.
• Subtle format drift: JSON output preceded by a one-line explanation (which breaks downstream parsing).
• Edge-case safety filter overreach: in medical or legal topics, legitimate informational responses sometimes get blocked.

The fourth is the hardest to catch. From the user's view it's just "no response came back," and normal error monitoring won't flag it. I track safety-filter block reasons and block rates as a separate metric.

Choosing the Threshold — Calibrate Against Past Logs

The boundary that decides "send to human review or not" will always be wrong if you set it by feel. About two weeks into running this, I inserted a one-time calibration against past logs.

The procedure isn't elaborate. Pull roughly 300 recent production responses and hand-label 120 of them as "wrong" or "fine." It's tedious, but skipping it leaves the threshold floating. The rest stay unlabeled, used only to gauge cost.

Here's what happened when I applied three different decision boundaries to those 120 hand-labeled cases. "Miss rate" is the share of truly-wrong responses that slipped through; "false-positive rate" is the share of fine responses needlessly sent to the human queue.

Laid out as numbers, it's clear there's no single "correct" answer. "Both inconsistent" is light on human load but lets over 20% of real errors through — too leaky for a quality guardrail. "Either inconsistent" cuts misses to 6% but inflates false positives toward 20%, turning the human queue into a mixed bag.

I settled on the third option, a score gap of 2 or more. Both miss rate and false-positive rate stay in single digits, and the queue volume fits what one person can clear in a day. The key is to work backward from how many cases you can actually review per day. Choosing a sustainable precision over an ideal one is what keeps the loop running long term.

This calibration isn't a one-and-done. When product requirements or topics shift, the same threshold drifts in its miss rate. I re-pull about 50 cases once a quarter and check how far the table has moved.

Dashboarding and Operations

I roll up evaluation results into a dashboard:

• Daily inconsistent rate by lens
• Top 3 "problem prompt patterns"
• Judge-to-judge agreement rate (if this falls, suspect judge quality drift)
• Safety-filter block rate trend

Alerts fire when the inconsistent rate exceeds 1.5× the trailing 7-day average. That catches sudden regressions early.

What This System Costs You

In fairness, there are costs too.

The biggest is that I can no longer take a single response at face value. However strong the accuracy, I keep thinking "how would the eval loop judge this?" That's healthy skepticism, but it does slow me down a little.

Second, the evaluation prompts themselves need maintenance. When product requirements change, evaluation criteria change. If they drift out of sync with reality, the loop starts lying to you.

Why It's Still Worth Building

Even so, I'm glad I built and operate this. Knowing what's happening is enormous for the sanity of a service operator.

If you're running Gemini in production, start with a minimal sampling-based evaluation. Don't try to write perfect criteria upfront — even a single-axis judge that only returns inconsistent vs. not will surface surprising issues. That becomes the starting point for your next improvement cycle.