Automating App Localization QA with the Gemini API: A Structured-Output Pipeline That Catches Translation Drift Early

Implementation 1: scoring translations with structured output

The core of the evaluator is a function that calls Gemini 2.5 Pro with a responseSchema. Instead of free-form natural language, the model returns a strict JSON with four numerical axes plus an issues array and a suggested rewrite.

# localization_qa.py
import os
import google.generativeai as genai
 
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
 
EVAL_SCHEMA = {
    "type": "object",
    "properties": {
        "accuracy":     {"type": "integer", "minimum": 1, "maximum": 5},
        "fluency":      {"type": "integer", "minimum": 1, "maximum": 5},
        "consistency":  {"type": "integer", "minimum": 1, "maximum": 5},
        "context_fit":  {"type": "integer", "minimum": 1, "maximum": 5},
        "issues":       {"type": "array", "items": {"type": "string"}},
        "suggested_translation": {"type": "string"},
    },
    "required": ["accuracy", "fluency", "consistency", "context_fit", "issues"],
}
 
EVAL_INSTRUCTIONS = """\
You are a localization QA engineer for mobile apps.
Score the translation on four axes (1-5).
- accuracy: preserves the meaning of the source
- fluency: reads naturally to a native speaker of the target language
- consistency: aligns in terminology and register with the prior examples provided
- context_fit: matches the provided UI context (screen, control kind)
Use the issues array for concrete problems and put your proposed
rewrite in suggested_translation.
"""
 
def evaluate_translation(
    source_text: str,
    translated_text: str,
    target_lang: str,
    glossary: dict[str, str],
    prior_examples: list[dict],
    ui_context: dict,
) -> dict:
    model = genai.GenerativeModel(
        "gemini-2.5-pro",
        generation_config={
            "response_mime_type": "application/json",
            "response_schema": EVAL_SCHEMA,
            "temperature": 0.1,
        },
        system_instruction=EVAL_INSTRUCTIONS,
    )
 
    prompt = f"""\
[Target Language] {target_lang}
[UI Context] screen_id={ui_context['screen_id']}, control={ui_context['control_kind']}
[Glossary] {glossary}
[Prior Examples in this language]
{prior_examples}
 
[Source (en)] {source_text}
[Translation ({target_lang})] {translated_text}
"""
    response = model.generate_content(prompt)
    return response.parsed if hasattr(response, "parsed") else \
           __import__("json").loads(response.text)

Returning a schema-validated JSON means the downstream gating code (threshold checks, issue-count rules) is just plain dict access. Keeping temperature low (0.1) matters in practice: if the same input yields different scores from day to day, CI becomes flaky and the team stops trusting it.

A small detail that is not spelled out in the official docs: when you supply response_schema, raising temperature occasionally produces schema-violating output. I have run this safely in the 0.0 - 0.2 range; above that I started seeing rare malformed outputs.

Implementation 2: change detection and incremental QA

Diff extraction is pure Python. Parse Localizable.strings line by line and compare against the previous commit:

# diff_extractor.py
import re
import subprocess
from pathlib import Path
 
STRING_RE = re.compile(r'^"([^"]+)"\s*=\s*"((?:[^"\\]|\\.)*)";\s*$')
 
def parse_strings(path: Path) -> dict[str, str]:
    out = {}
    for line in path.read_text(encoding="utf-8").splitlines():
        m = STRING_RE.match(line.strip())
        if m:
            out[m.group(1)] = m.group(2)
    return out
 
def extract_changed_keys(repo: Path, file_rel: str, base_ref: str = "HEAD~1"):
    new = parse_strings(repo / file_rel)
    try:
        old_raw = subprocess.check_output(
            ["git", "show", f"{base_ref}:{file_rel}"],
            cwd=repo, text=True
        )
        old = {}
        for line in old_raw.splitlines():
            m = STRING_RE.match(line.strip())
            if m:
                old[m.group(1)] = m.group(2)
    except subprocess.CalledProcessError:
        old = {}
    changed = {k: v for k, v in new.items() if old.get(k) != v}
    return changed

One tuning detail: using base_ref="HEAD~1" picks up intra-branch edits that get later reverted, inflating the evaluation set. I switched to the output of git merge-base HEAD origin/main, which alone cut the volume by roughly 30-50%.

Implementation 3: making the consistency scan affordable

Asking Gemini to perform full consistency scans against the entire string catalog grows cost linearly. Instead, I prefilter with cheap embeddings to retrieve the five closest existing translations in the same target language, then ask Gemini to evaluate consistency against those.

# consistency.py
import google.generativeai as genai
import numpy as np
 
def embed(texts: list[str]) -> np.ndarray:
    res = genai.embed_content(
        model="models/text-embedding-004",
        content=texts,
        task_type="SEMANTIC_SIMILARITY",
    )
    return np.array(res["embedding"])
 
def top_k_similar(query: str, corpus: list[tuple[str, str]], k: int = 5):
    """corpus: [(key, translation), ...] all in the same target language"""
    if not corpus:
        return []
    q = embed([query])[0]
    matrix = embed([t for _, t in corpus])
    sims = matrix @ q / (np.linalg.norm(matrix, axis=1) * np.linalg.norm(q) + 1e-9)
    idx = np.argsort(-sims)[:k]
    return [corpus[i] for i in idx]

At $0.025 per million tokens, text-embedding-004 is essentially free for this workload. Across my portfolio (17 languages × ~2,800 keys, around 5,000 evaluations per month), the embedding step costs less than $0.50/month, and it cuts the input size that Gemini 2.5 Pro has to read by more than half.

Implementation 4: CI integration and PR comments

I wire this into GitHub Actions and run it on every PR that touches localization files. Crucially, I do not fail the build on a bad score; I post a comment and apply a label. Hard-blocking makes translators avoid opening PRs, which is worse than catching the issue at review.

# .github/workflows/localization-qa.yml
name: Localization QA
on:
  pull_request:
    paths:
      - "Localizable/**/*.strings"
      - "app/src/main/res/values-*/strings.xml"
 
jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install google-generativeai numpy
      - name: Run QA
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: python scripts/localization_qa.py --base-ref ${{ github.event.pull_request.base.sha }}
      - name: Comment on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require("fs");
            const body = fs.readFileSync("qa-report.md", "utf-8");
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body
            });

The report is plain Markdown, formatted as a key | lang | avg_score | issues table. Putting entries with avg_score < 3.5 at the top and collapsing the rest behind a <details> block drops the cognitive load enough that PRs actually get read.

Operational lessons: how to choose thresholds

Setting the gating threshold is the part that worries me most before a release. A few rules that I had to learn the hard way.

First, run for two weeks in observation mode before gating anything. Without seeing the score distribution, picking 3.5 as a universal cutoff produces flurries of false positives in specific languages (Polish and Turkish were the worst for me) where Gemini's scoring tendencies diverge from English or Japanese.

Second, keep per-language thresholds. I run 3.7 for CJK (Japanese, Chinese, Korean), 3.5 for Latin-script languages (English, French, Spanish), and 3.3 elsewhere. Trying to "calibrate Gemini" so a single threshold works across all languages turned out to be a losing battle.

Third, review the issues array weekly and fold the recurring false-positive patterns back into system_instruction. After about three months you end up with a domain-specific evaluator that hits a much better signal-to-noise ratio than the day-one prompt.

Cost: 5,000 monthly evaluations for about $4

Concrete numbers, all based on Gemini 2.5 Pro list pricing as of May 2026 (input $1.25/1M tokens, output $10/1M tokens):

A typical evaluation uses roughly 600 input tokens (system instruction + glossary + five prior examples + source + translation) and 150 output tokens (the JSON scores + issues). That works out to roughly $0.00225 per evaluation, or about $11 per 5,000 evaluations.

Context Caching is where the cost actually collapses. The system_instruction, the per-language glossary, and the top-100 prior examples per language do not change between requests within a session. Putting those into a Cached Content object drops the input cost on cache hits substantially. In my setup, 70-80% of evaluation requests hit the cache, and total spend lands around $4/month.

You can take this further: route the first pass through Gemini 2.5 Flash, and only escalate to Pro when the Flash score lands in the grey zone (3.0-4.0). For most localization workloads this halves the cost again.

Common pitfalls

Three years in, the same handful of pitfalls keep showing up.

The first is running QA on ambiguous source strings. Short labels like "Save" cannot be translated unambiguously into Japanese (保存 / 保管 / 救う are all plausible). Punishing Gemini for the resulting low score is the wrong response. The fix lives upstream: always pass screen_id and control_kind, and use the comment field (Apple .stringsdict supports this) to disambiguate at the source.

The second is over-trusting the score. An evaluation can return 4.5 and still produce text that overflows the UI container. Always run a separate length-ratio check (len(target) / len(source)) outside the model.

The third is running every language inside a single CI job. I did that initially and hit Gemini API rate limits often enough that the whole pipeline would fall over. Splitting with matrix: per language and capping max-parallel: 4 made the pipeline boringly reliable.

One next step

If you suspect your app has silent localization drift right now, the highest-leverage first move is to run this pipeline for a week in observation-only mode. Looking at the actual score distribution for your specific app and language mix is the only honest way to set a threshold that stays useful as the catalog grows.

I am still refining this pipeline, especially for non-Latin-script languages where the evaluator is less stable than I would like. If you are working on the same problem, I hope this is a useful starting point. Thank you for reading.