Building a RAG Evaluation Framework with Gemini API: RAGAS, LLM-as-Judge, and Custom Metrics Production Masterclass

Here's a pattern I've seen repeatedly across RAG projects: you spend weeks tuning prompts, adjusting retrieval parameters, swapping embedding models — and at some point you're making decisions based on gut feeling rather than data. Then a user reports that answer quality has dropped, and tracing back the cause takes longer than the original improvement did.

The fix isn't more tuning. It's building a measurement system first.

This guide covers how to build a complete RAG evaluation framework using Gemini API as the judge model, integrating RAGAS for automated metrics, LLM-as-Judge for semantic quality, and custom domain-specific criteria — all wired into CI/CD for continuous quality monitoring.

Why RAG Quality Is Hard to Measure

RAG differs from classification tasks in a fundamental way: there's no clean ground truth for "did this answer well?"

A search engine can be evaluated by whether relevant documents rank highly. But a RAG system has three failure modes that interact:

Retrieval failure: The right context wasn't fetched
Augmentation failure: The model didn't use the context correctly
Generation failure: The answer wasn't well-formed for the question

Evaluating only the final answer without decomposing these stages means you know that something is wrong without knowing where.

An evaluation framework makes each stage measurable independently.

The Four Core Metrics

Before writing any code, here's the conceptual model for what we're measuring.

Faithfulness measures whether the generated answer is grounded in the retrieved context — specifically, what fraction of claims in the answer are actually supported by the context. A score of 1.0 means every statement traces back to retrieved content. Anything below 0.7 in production typically causes user trust issues.

Answer Relevancy measures whether the answer addresses the question asked. Even a perfectly faithful answer can be irrelevant if it answers the wrong question. This metric is computed by generating reverse questions from the answer and comparing their embeddings to the original question.

Context Precision measures what fraction of the retrieved context was actually useful for generating the answer. High precision means your retrieval is targeted. Low precision means you're fetching noise that may confuse the model.

Context Recall measures whether all information needed for a correct answer was actually retrieved. This requires a ground truth answer, making it the most expensive metric to compute — but the most diagnostic for retrieval-stage problems.

from dataclasses import dataclass
from typing import Optional
 
@dataclass
class RAGEvalMetrics:
    faithfulness: float
    answer_relevancy: float
    context_precision: float
    context_recall: Optional[float] = None  # requires ground truth
 
    @property
    def overall_score(self) -> float:
        scores = [self.faithfulness, self.answer_relevancy, self.context_precision]
        return sum(scores) / len(scores)
 
    def is_acceptable(self, threshold: float = 0.70) -> bool:
        return self.overall_score >= threshold

RAGAS Setup with Gemini as the Evaluator

RAGAS automates the computation of these metrics. Configuring it to use Gemini rather than OpenAI takes one configuration step.

# requirements: ragas>=0.2.0, langchain-google-genai>=2.0.0
 
import os
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_google_genai import ChatGoogleGenerativeAI
from datasets import Dataset
 
def setup_gemini_evaluator(model: str = "gemini-2.5-flash-preview-05-20"):
    """
    Use Gemini Flash as the RAGAS judge model.
    Flash is cost-efficient for bulk evaluation — use Pro only for
    high-stakes manual audits.
    """
    llm = ChatGoogleGenerativeAI(
        model=model,
        google_api_key=os.environ["GOOGLE_API_KEY"],
        temperature=0.0,  # deterministic evaluation
    )
    return LangchainLLMWrapper(llm)
 
evaluator_llm = setup_gemini_evaluator()
for metric in [faithfulness, answer_relevancy, context_precision, context_recall]:
    metric.llm = evaluator_llm
 
def run_ragas_evaluation(
    rag_pipeline,
    test_cases: list[dict],
) -> list[dict]:
    """
    Evaluate a RAG pipeline using RAGAS metrics.
 
    Args:
        rag_pipeline: callable(question: str) -> {"answer": str, "contexts": list[str]}
        test_cases: list of {"question": str, "ground_truth": str}
 
    Returns:
        list of per-sample metric dicts
    """
    questions, answers, contexts, ground_truths = [], [], [], []
 
    for case in test_cases:
        result = rag_pipeline(case["question"])
        questions.append(case["question"])
        answers.append(result["answer"])
        contexts.append(result["contexts"])
        ground_truths.append(case.get("ground_truth", ""))
 
    eval_dataset = Dataset.from_dict({
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths,
    })
 
    result = evaluate(
        dataset=eval_dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )
    return result.to_pandas().to_dict(orient="records")

A common surprise when first running RAGAS: Faithfulness scores are often lower than expected. If your system prompt encourages the model to "add helpful background context," it will generate claims not grounded in the retrieved context — which is exactly what Faithfulness penalizes.

LLM-as-Judge for Semantic Quality

RAGAS metrics are good at structural quality — but they don't capture things like "is this explanation clear?" or "does this answer match the expected tone for our product?"

LLM-as-Judge uses Gemini itself to evaluate the response against a rubric. The key is using a different model or configuration for generation and evaluation to reduce self-affirmation bias.

import google.generativeai as genai
from pydantic import BaseModel, Field
import json
 
class JudgmentResult(BaseModel):
    score: float = Field(ge=0.0, le=1.0)
    reasoning: str
    passed: bool
    issues: list[str] = Field(default_factory=list)
 
JUDGE_PROMPT = """
You are a strict quality auditor for a RAG system. Evaluate the response below
against these criteria:
 
1. Factual grounding: does every claim trace back to the provided context?
2. Question coverage: does the response actually answer what was asked?
3. Information balance: is the response neither missing key facts nor padded with noise?
4. Clarity: would a non-expert reader understand this response?
 
Input:
Question: {question}
Retrieved Context: {contexts}
Generated Answer: {answer}
 
Return a JSON object with this structure:
{{
  "score": <float 0.0-1.0>,
  "reasoning": "<one sentence rationale>",
  "passed": <true if score >= 0.75>,
  "issues": ["<issue 1>", "<issue 2>"]
}}
"""
 
def llm_as_judge(
    question: str,
    contexts: list[str],
    answer: str,
    judge_model: str = "gemini-2.5-pro-preview-05-06",
) -> JudgmentResult:
    """
    Use Gemini Pro as judge for semantic quality evaluation.
 
    Using Pro as judge when generation runs on Flash creates
    an intentional asymmetry that reduces self-affirmation bias.
    """
    client = genai.GenerativeModel(judge_model)
 
    prompt = JUDGE_PROMPT.format(
        question=question,
        contexts="\n---\n".join(contexts[:3]),  # top 3 only to control cost
        answer=answer,
    )
 
    response = client.generate_content(
        prompt,
        generation_config=genai.GenerationConfig(
            temperature=0.0,
            response_mime_type="application/json",
        ),
    )
 
    return JudgmentResult(**json.loads(response.text))
 
# Example output:
# JudgmentResult(
#   score=0.88,
#   reasoning="All claims are grounded in context; response directly addresses the question",
#   passed=True,
#   issues=[]
# )

Building Custom Domain Metrics

Generic metrics won't catch domain-specific failure modes. A RAG system over financial documents has different quality requirements than one over product documentation.

from typing import Callable
import re
 
def build_custom_metric(
    name: str,
    evaluator: Callable[[str, str, list[str]], float],
    description: str,
) -> dict:
    return {"name": name, "description": description, "evaluator": evaluator}
 
# Example 1: Numeric accuracy (finance / data-heavy domains)
def numeric_accuracy_check(question: str, answer: str, contexts: list[str]) -> float:
    """
    Check that numbers in the answer appear in the retrieved context.
    Catches cases where the model "rounds" or "estimates" figures not in the source.
 
    Returns: fraction of answer numbers that exist in context (0.0-1.0)
    """
    answer_numbers = set(re.findall(r'\d+(?:\.\d+)?', answer))
    if not answer_numbers:
        return 1.0  # no numbers = no numeric grounding issues
 
    context_numbers = set(re.findall(r'\d+(?:\.\d+)?', " ".join(contexts)))
    matched = answer_numbers & context_numbers
    return len(matched) / len(answer_numbers)
 
# Example 2: Source citation quality (documentation RAG)
def citation_quality_check(question: str, answer: str, contexts: list[str]) -> float:
    """
    Check whether the answer properly signals when it's citing source material.
    """
    citation_phrases = [
        "according to", "as stated in", "per the documentation",
        "the documentation notes", "as described in",
    ]
    has_citation = any(phrase in answer.lower() for phrase in citation_phrases)
    has_context = len(contexts) > 0
 
    if has_context and has_citation:
        return 1.0
    elif not has_context:
        return 0.5
    else:
        return 0.65  # context exists but answer doesn't signal it
 
numeric_metric = build_custom_metric(
    name="numeric_accuracy",
    evaluator=numeric_accuracy_check,
    description="Fraction of numeric claims in the answer that are grounded in context",
)

Design custom metrics by starting from the question: "what would a domain expert consider a failure?" Work backward from those failure modes to measurable signals.

Dataset Management

The evaluation dataset is as important as the evaluation code. Starting with 20–30 hand-crafted question/answer pairs is sufficient for initial calibration.

import json
from pathlib import Path
from datetime import datetime
 
class EvalDatasetManager:
    def __init__(self, dataset_path: str = "data/eval_dataset.json"):
        self.path = Path(dataset_path)
        self.dataset = self._load()
 
    def _load(self) -> list[dict]:
        if self.path.exists():
            return json.loads(self.path.read_text())
        return []
 
    def add_case(
        self,
        question: str,
        ground_truth: str,
        category: str = "general",
        difficulty: str = "medium",
    ) -> None:
        case = {
            "id": f"{len(self.dataset) + 1:04d}",
            "question": question,
            "ground_truth": ground_truth,
            "category": category,
            "difficulty": difficulty,
            "created_at": datetime.now().isoformat(),
        }
        self.dataset.append(case)
        self.path.write_text(
            json.dumps(self.dataset, ensure_ascii=False, indent=2)
        )
 
    def get_summary(self) -> dict:
        from collections import Counter
        return {
            "total": len(self.dataset),
            "by_category": dict(Counter(c["category"] for c in self.dataset)),
            "by_difficulty": dict(Counter(c["difficulty"] for c in self.dataset)),
        }

The most effective dataset questions come from real user interactions — support tickets, search logs, or Intercom conversations. Questions you invent tend to cluster around the cases you already handle well.

CI/CD Integration: Automated Quality Gates

Wiring evaluation into GitHub Actions ensures that every change to the RAG pipeline is validated before merge.

# .github/workflows/rag-quality-gate.yml
name: RAG Quality Gate
 
on:
  pull_request:
    paths:
      - 'src/rag/**'
      - 'prompts/**'
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
 
      - run: pip install ragas langchain-google-genai datasets
 
      - name: Run evaluation
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: python scripts/eval_rag.py --threshold 0.72 --output report.json
 
      - name: Enforce quality gate
        run: |
          python -c "
          import json, sys
          r = json.load(open('report.json'))
          print(f'Score: {r[\"overall_score\"]:.3f}')
          sys.exit(0 if r['passed'] else 1)
          "
 
      - name: Post PR comment
        uses: actions/github-script@v7
        if: always()
        with:
          script: |
            const r = JSON.parse(require('fs').readFileSync('report.json'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## RAG Quality Report\n| Metric | Score |\n|--------|-------|\n| Faithfulness | ${r.faithfulness.toFixed(3)} |\n| Answer Relevancy | ${r.answer_relevancy.toFixed(3)} |\n| Context Precision | ${r.context_precision.toFixed(3)} |\n| **Overall** | **${r.overall_score.toFixed(3)}** |\n\nThreshold: 0.72 | ${r.passed ? '✅ PASSED' : '❌ FAILED'}`
            });

# scripts/eval_rag.py
import argparse, json, sys
from pathlib import Path
 
def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--threshold", type=float, default=0.70)
    parser.add_argument("--output", default="report.json")
    args = parser.parse_args()
 
    test_cases = json.loads(Path("data/eval_dataset.json").read_text())[:20]
 
    from src.rag.pipeline import run_rag
    results = run_ragas_evaluation(run_rag, test_cases)
 
    scores = {k: sum(r.get(k, 0) for r in results) / len(results)
              for k in ["faithfulness", "answer_relevancy", "context_precision"]}
    overall = sum(scores.values()) / 3
 
    report = {**scores, "overall_score": overall, "passed": overall >= args.threshold,
              "sample_count": len(results)}
 
    Path(args.output).write_text(json.dumps(report, indent=2))
    sys.exit(0 if report["passed"] else 1)
 
if __name__ == "__main__":
    main()

Production Sampling Monitor

Evaluating 100% of production traffic is prohibitively expensive. A 5% sampling rate with async evaluation captures quality trends without impacting latency.

import random
import asyncio
from datetime import datetime
 
class ProductionRAGMonitor:
    def __init__(self, sample_rate: float = 0.05, alert_threshold: float = 0.65):
        self.sample_rate = sample_rate
        self.alert_threshold = alert_threshold
        self.buffer: list[dict] = []
 
    def should_evaluate(self) -> bool:
        return random.random() < self.sample_rate
 
    async def evaluate_async(self, question, answer, contexts, request_id):
        """Fire-and-forget: doesn't block user response"""
        if self.should_evaluate():
            asyncio.create_task(
                self._run_evaluation(question, answer, contexts, request_id)
            )
 
    async def _run_evaluation(self, question, answer, contexts, request_id):
        try:
            result = llm_as_judge(question, contexts, answer)
            metric = {
                "request_id": request_id,
                "timestamp": datetime.utcnow().isoformat(),
                "score": result.score,
                "passed": result.passed,
                "issues": result.issues,
            }
            self.buffer.append(metric)
 
            if result.score < self.alert_threshold:
                print(f"⚠️ Quality alert [{request_id}]: score={result.score:.2f}, issues={result.issues}")
                # wire this to Slack/PagerDuty/BigQuery in production
 
        except Exception as e:
            # evaluation failures are silent — never affect user responses
            print(f"Eval error for {request_id}: {e}")
 
    def get_rolling_stats(self, window: int = 100) -> dict:
        recent = self.buffer[-window:]
        if not recent:
            return {}
        scores = [m["score"] for m in recent]
        return {
            "avg_score": round(sum(scores) / len(scores), 3),
            "pass_rate": round(sum(1 for m in recent if m["passed"]) / len(recent), 3),
            "min_score": round(min(scores), 3),
            "sample_count": len(recent),
        }

Three Before/After Case Studies

Here are the most impactful improvements this framework revealed in practice.

Case 1: System prompt wording caused Faithfulness to drop

Before: "Answer helpfully and add relevant background context where useful." → Faithfulness: 0.58 (model added unretrieved "background" information)

After: "Answer using only the information in the provided context. If something is not in the context, say so explicitly." → Faithfulness: 0.89 (+31 points)

Case 2: Retrieving too many chunks hurt Context Precision

Before: Fetching top-10 chunks → Context Precision: 0.41 After: Filtering to top-3 by relevance score → Context Precision: 0.78

More context is not always better. Irrelevant chunks create noise that reduces the quality of the generation step.

Case 3: Query rewriting improved Answer Relevancy

Before: Passing raw user queries to vector search → Answer Relevancy: 0.71 After: Rewriting queries with Gemini before retrieval → Answer Relevancy: 0.87

def rewrite_query_for_retrieval(question: str) -> str:
    """
    Rewrite a conversational user question into a retrieval-optimized form.
 
    Example:
      Input:  "how do I make it stop doing that?"
      Output: "Gemini API stop sequence configuration parameter"
    """
    client = genai.GenerativeModel("gemini-2.5-flash-preview-05-20")
    response = client.generate_content(
        f"Convert this user question into a search query optimized for "
        f"technical documentation retrieval. Return only the query.\n\n"
        f"Question: {question}"
    )
    return response.text.strip()

Managing Evaluation Costs

A realistic cost estimate for one evaluation run (20 test cases, Gemini Flash as judge):

RAGAS metrics: ~$0.003–$0.01 per test case
LLM-as-Judge: ~$0.0003–$0.001 per test case
Total for 20 cases: ~$0.07–$0.22

The practical approach: use Flash for all automated CI/CD evaluation, and run a weekly deep audit with Pro on the 50 lowest-scoring samples from production monitoring. This keeps CI costs under $0.25/run while ensuring high-fidelity quarterly reviews.

The framework itself needs maintenance — thresholds that made sense at launch may need adjustment as the underlying documents or query distribution shifts. A quarterly review of threshold calibration is worth scheduling explicitly.

For building the underlying RAG system, see Gemini API RAG System Complete Implementation. For deeper evaluation patterns, Gemini API LLM-as-Judge Evaluation Production Guide covers more advanced judging architectures.

The most important thing is to start. A 20-case evaluation dataset and one automated CI check will reveal more about your RAG system's actual behavior than weeks of manual testing. Build the measurement infrastructure first, then let the data guide what to improve.