◈ API / SDK/2026-04-22Advanced

Quietly Catching Wrong Answers in Your Gemini-Powered App — A Production Auto-Eval Loop

Running Gemini in production eventually shows you responses that are 'kind of wrong.' I want to catch them before users do. This is the exact auto-eval loop I run over live traffic, with the prompts I use and the mistakes I had to learn my way through.

Gemini API¹⁹² Evaluation³ Quality² LLM as judge Production³²

✦ Premium Article

With Gemini in production, reaching "mostly correct" responses is actually pretty quick. The harder state to escape is "wrong sometimes, quietly." I operate a service where I built and maintain an auto-eval loop that runs over live traffic to catch exactly that kind of quiet failure.

This article shares the design, the prompts, and the traps I stepped in. It's Gemini-specific in places, but the shape applies to any LLM in production.

As an indie developer I run a handful of my own apps, with Gemini wired into their support replies and description generation. I started building this eval loop the week AdMob revenue dipped a few percent after a feature release, with no obvious explanation. Tracing it, the cause wasn't a server error or a crash — the responses themselves had quietly degraded. Users just close the app; nothing reaches you. That faintly tilted-floor feeling was the starting point for this whole system.

Why Auto-Eval in Production Matters

Pre-deployment evaluation on a fixed test set is important, but production introduces factors you can't capture upfront:

Users send inputs you never anticipated
The UI changes, so the prompts drift subtly
The Gemini model updates under the hood
Documents and FAQs change, changing context

All of these bypass your pre-eval coverage. Trying to catch everything pre-deployment leads to bloated test sets that drift out of sync with reality. A lightweight but continuous evaluation over real traffic is the practical answer.

The moment I was most grateful I had this system: a feature's accuracy was drifting slowly downward, no user complained, but the evaluation dashboard showed the trend clearly.

Four Lenses to Evaluate in Production

I automatically score each production response on:

Factuality — do factual claims in the answer contradict the given context?
Instruction following — does the output obey system prompt requirements (format, length, prohibitions)?
Relevance — does the answer actually address the user's question?
Safety — is there any leakage of sensitive info, harmful content, or bias?

Evaluating all four on every request is rarely economical. I route evaluation based on request type and risk.

For example, a FAQ bot leans on 1 and 3. A generative response leans on 2 and 4. Trying to measure all four for every request can push evaluation cost above generation cost.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦A two-judge majority setup using a different model family from the generator, sending only disagreements to humans — tiered sampling that keeps eval cost near 15% of generation cost

✦A concrete calibration procedure on 120 hand-labeled past responses, with a measured table of miss rate, false-positive rate, and human-queue share per threshold

✦Alert design that pings Slack when the inconsistent rate exceeds 1.5x the 7-day average, plus the four failure modes that actually surfaced while running my own indie apps

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

A Concrete Judge Prompt

Each lens runs on a separate, cheaper model (Haiku-class or Gemini's smaller sibling) as the Judge. Here's my prompt for factuality:

You are an LLM response evaluator. You're given:
 
[Context]
{source_document}
 
[User question]
{user_query}
 
[Answer under review]
{model_answer}
 
Evaluate as follows:
1. List every factual claim made in the answer
2. For each claim, check whether it is contradicted by the context
3. Produce an overall verdict: "consistent" / "partially_inconsistent" / "inconsistent"
 
Quote evidence from the context whenever possible.
Do not guess. If the context doesn't contain the needed info,
return "insufficient_context".
 
Return JSON:
{
  "overall": "...",
  "claims": [
    {"text": "...", "verdict": "...", "evidence": "..."}
  ]
}

The key line is "do not guess, return insufficient_context if the info isn't there." Without it, the judge itself hallucinates. Once the judge starts inventing verdicts, the whole evaluation is corrupted.

Avoiding the Self-Evaluation Bias

If you generate with Gemini and judge with Gemini, the same model biases toward favorable verdicts of its own output. This is documented in research and I see it clearly in my own data.

My countermeasures:

Use a different model family as the judge. Generate with Gemini, judge with a small Claude or GPT model.
Two-judge majority vote. Only responses where the two judges disagree go to human review.
Validate the judge weekly. Run it against a known-answer dataset to detect judge quality drift.

The second tactic has the best ROI. Always running two judges doubles cost, but restricting deep review to disagreement cases pushes anomaly-detection precision up dramatically at acceptable cost.

Sampling Strategy to Keep Cost Sustainable

Evaluating every response isn't economical. I layer sampling:

Always evaluate: responses with anomalous length, containing specific keywords, or that received negative user feedback — 100% coverage
Random sample: evaluate 1–5% of all responses (for daily reports)
Deep-dive evaluation: any response judged inconsistent is re-evaluated on a bigger model

The first tier matters most. Outliers are already signal — sampling them loses information. The rest can be sampled aggressively without much loss.

With this tiered sampling, evaluation cost lands around 15% of generation cost. That's operable.

Typical Failure Modes the Loop Surfaces

Six months in, these are the Gemini failure patterns the loop surfaces most often:

Slow instruction-following decay: as prompts accumulate additions over time, older instructions get ignored. Newer clauses tend to dominate.
Factual errors at long context: the longer the context, the more the model confuses information from the first half.
Subtle format drift: JSON output preceded by a one-line explanation (which breaks downstream parsing).
Edge-case safety filter overreach: in medical or legal topics, legitimate informational responses sometimes get blocked.

The fourth is the hardest to catch. From the user's view it's just "no response came back," and normal error monitoring won't flag it. I track safety-filter block reasons and block rates as a separate metric.

Choosing the Threshold — Calibrate Against Past Logs

The boundary that decides "send to human review or not" will always be wrong if you set it by feel. About two weeks into running this, I inserted a one-time calibration against past logs.

The procedure isn't elaborate. Pull roughly 300 recent production responses and hand-label 120 of them as "wrong" or "fine." It's tedious, but skipping it leaves the threshold floating. The rest stay unlabeled, used only to gauge cost.

Here's what happened when I applied three different decision boundaries to those 120 hand-labeled cases. "Miss rate" is the share of truly-wrong responses that slipped through; "false-positive rate" is the share of fine responses needlessly sent to the human queue.

Condition to send to human queue	Share queued	Miss rate (measured)	False-positive rate (measured)
Both judges say inconsistent	~3%	22%	4%
Either judge says inconsistent	~11%	6%	19%
5-point score gap of 2 or more	~7%	9%	8%

Laid out as numbers, it's clear there's no single "correct" answer. "Both inconsistent" is light on human load but lets over 20% of real errors through — too leaky for a quality guardrail. "Either inconsistent" cuts misses to 6% but inflates false positives toward 20%, turning the human queue into a mixed bag.

I settled on the third option, a score gap of 2 or more. Both miss rate and false-positive rate stay in single digits, and the queue volume fits what one person can clear in a day. The key is to work backward from how many cases you can actually review per day. Choosing a sustainable precision over an ideal one is what keeps the loop running long term.

This calibration isn't a one-and-done. When product requirements or topics shift, the same threshold drifts in its miss rate. I re-pull about 50 cases once a quarter and check how far the table has moved.

Dashboarding and Operations

I roll up evaluation results into a dashboard:

Daily inconsistent rate by lens
Top 3 "problem prompt patterns"
Judge-to-judge agreement rate (if this falls, suspect judge quality drift)
Safety-filter block rate trend

Alerts fire when the inconsistent rate exceeds 1.5× the trailing 7-day average. That catches sudden regressions early.

What This System Costs You

In fairness, there are costs too.

The biggest is that I can no longer take a single response at face value. However strong the accuracy, I keep thinking "how would the eval loop judge this?" That's healthy skepticism, but it does slow me down a little.

Second, the evaluation prompts themselves need maintenance. When product requirements change, evaluation criteria change. If they drift out of sync with reality, the loop starts lying to you.

Why It's Still Worth Building

Even so, I'm glad I built and operate this. Knowing what's happening is enormous for the sanity of a service operator.

If you're running Gemini in production, start with a minimal sampling-based evaluation. Don't try to write perfect criteria upfront — even a single-axis judge that only returns inconsistent vs. not will surface surprising issues. That becomes the starting point for your next improvement cycle.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.