⬡ Advanced/2026-07-17Advanced

A Japanese query won't surface its English twin — when embeddings notice language before meaning

Embed a translation pair with gemini-embedding-2 and the two halves won't be nearest neighbours, because language itself inflates similarity. Here is how I measured cross-lingual recall using translation pairs as ground truth, and what happened when I subtracted the language centroid.

Gemini API¹⁸⁹ gemini-embedding-2⁷ multilingual⁸ vector search evaluation⁴

✦ Premium Article

Across four sites I publish every article in Japanese and English as a matched pair. If one half is missing, switching languages returns a 404, so the counts always match. One site currently sits at 725 JA and 725 EN. Which means I hold 725 documents whose meaning is identical and whose language is the only difference — ground truth, already labelled, for free.

I wanted to use those pairs to build a pre-publication duplicate check. Embed a new Japanese draft, retrieve the closest existing articles, and since both languages live in the same index, catch cross-language overlap too.

The first query returned ten Japanese articles. Every one of them. And the draft I queried with had an English counterpart that already existed — one whose content matched almost exactly. It lost to Japanese articles about entirely different topics.

My task_type values were correct. My vectors were normalised. The problem sat further upstream than either.

Cosine similarity is also measuring what language you wrote in

An embedding model turns text into a vector. You would like that vector to carry meaning. In practice it also carries the language the text was written in.

Multilingual models are trained to bring meanings closer across languages, true. They are not trained to align them perfectly. What you get instead is a space where Japanese text loosely clusters in one region and English text clusters in another.

Pair	Semantic distance	Same language?	Observed similarity
Japanese draft ↔ its English translation	near-identical	no	medium
Japanese draft ↔ unrelated Japanese article	far	yes	medium to high

The bonus that language agreement pays out is large enough to cover the semantic gap. Those ten Japanese results were not a model failure. I had been measuring language and meaning together while believing I measured only meaning.

Whether that matters depends entirely on your usage. If you only ever search within one language, the language component loads onto every vector equally and rankings are unaffected. It bites at exactly one moment: when Japanese and English share an index and you query across both. That is the moment I walked into.

Translation pairs are already an evaluation set

Measurement first. The nice property of translation pairs is that nobody has to label anything. When I query with the Japanese version of gemini-3-multitool-agent-guide.mdx, the correct answer is the English file with the same slug. Decided in advance.

A hundred pairs is enough to see the shape. The harness below embeds both halves and reports JA→EN recall and MRR.

import os, json, math, random
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
MODEL = "gemini-embedding-2"
 
def embed(texts, task_type):
    """Embed a list of texts, batching to stay under the API limit."""
    out = []
    for i in range(0, len(texts), 32):
        chunk = texts[i:i + 32]
        res = client.models.embed_content(
            model=MODEL,
            contents=chunk,
            config=types.EmbedContentConfig(task_type=task_type),
        )
        out.extend([e.values for e in res.embeddings])
    return out
 
def l2_normalize(v):
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v] if n else v
 
def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))
 
# pairs: [{"slug": "...", "ja": "first 2000 chars of the JA body", "en": "English body ..."}, ...]
pairs = json.load(open("pairs.json"))
random.seed(42)
pairs = random.sample(pairs, min(100, len(pairs)))
 
ja_vecs = [l2_normalize(v) for v in embed([p["ja"] for p in pairs], "RETRIEVAL_QUERY")]
en_vecs = [l2_normalize(v) for v in embed([p["en"] for p in pairs], "RETRIEVAL_DOCUMENT")]
 
def evaluate(queries, docs):
    """queries[i] should retrieve docs[i]. Collect rank statistics."""
    hits = {1: 0, 5: 0, 10: 0}
    rr_sum = 0.0
    for i, q in enumerate(queries):
        scored = sorted(
            ((cosine(q, d), j) for j, d in enumerate(docs)),
            reverse=True,
        )
        rank = next(r for r, (_, j) in enumerate(scored, start=1) if j == i)
        for k in hits:
            if rank <= k:
                hits[k] += 1
        rr_sum += 1.0 / rank
    n = len(queries)
    return {
        "recall@1": hits[1] / n,
        "recall@5": hits[5] / n,
        "recall@10": hits[10] / n,
        "MRR": rr_sum / n,
    }
 
print("baseline:", evaluate(ja_vecs, en_vecs))

At this point the candidate pool holds English documents only. The language bonus applies uniformly to every candidate, so recall comes out clean. On my corpus baseline recall@1 landed around 0.9.

The interesting part is the next step. Add the Japanese articles to the pool.

# Put the Japanese articles into the same index (excluding the query itself)
ja_docs = [l2_normalize(v) for v in embed([p["ja"] for p in pairs], "RETRIEVAL_DOCUMENT")]
 
def evaluate_mixed(queries, en_docs, ja_docs):
    hits = {1: 0, 5: 0, 10: 0}
    for i, q in enumerate(queries):
        cands = [(cosine(q, d), ("en", j)) for j, d in enumerate(en_docs)]
        cands += [(cosine(q, d), ("ja", j)) for j, d in enumerate(ja_docs) if j != i]
        cands.sort(reverse=True)
        rank = next(r for r, (_, key) in enumerate(cands, start=1) if key == ("en", i))
        for k in hits:
            if rank <= k:
                hits[k] += 1
    n = len(queries)
    return {f"recall@{k}": v / n for k, v in hits.items()}
 
print("mixed:", evaluate_mixed(ja_vecs, en_vecs, ja_docs))

recall@1 collapsed the moment Japanese candidates entered the pool. The gap between the two runs is the size of the language component in your data. An index that looked healthy on a single-language benchmark became a different animal once it went multilingual. Being able to put a number on that drop was the real payoff of the harness.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Use translation pairs you already own as a labelled evaluation set and measure your index's cross-lingual recall today

✦Restore cross-lingual ranking with language-centroid removal — vector math only, zero extra API calls at query time

✦Choose between centroid removal, a translation bridge, and per-language indexes on latency and cost grounds

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

What did not work, first

Roughly in the order I tried them. Sharing the dead ends up front will save you the afternoon it cost me.

Changing task_type. I worked through the combinations — aligning both sides on SEMANTIC_SIMILARITY, forcing both to RETRIEVAL_DOCUMENT, and so on. Ranks shifted a little. Japanese candidates still owned the top of the list. task_type addresses the asymmetry between queries and documents; it has nothing to say about a language skew. Getting it right still matters, so if you haven't, read why task_type is the first thing to suspect when retrieval is close but wrong first.

Raising the dimension count. I pushed output_dimensionality from 768 to 1536 and then 3072. Essentially no change. The language component is not an artefact of insufficient capacity — it is structural. More resolution scales the same structure up. Dimension count is a storage trade-off, a separate axis covered in cutting dimensions to shrink storage.

Preprocessing. Normalising full-width punctuation, stripping heading markers — good hygiene, irrelevant here. A language difference is not a formatting inconsistency.

Lowering the similarity threshold. Rather than cutting at top-N, why not accept everything above a threshold? This made things worse. Lower the threshold and unrelated same-language articles flood in. A ranking problem does not yield to a threshold when the ordering itself is unchanged.

Subtract the mean vector of each language

This is what worked.

The idea is plain. If the language component is an offset shared by every Japanese embedding, then the mean of the Japanese vectors approximates that offset well. Subtract each vector's own language mean and the shared part cancels, leaving only how that text differs from other texts in the same language.

def centroid(vecs):
    n = len(vecs)
    dim = len(vecs[0])
    return [sum(v[i] for v in vecs) / n for i in range(dim)]
 
def remove_centroid(vecs, c):
    """Subtract the language centroid, then re-normalise. Skipping the
    re-normalisation distorts cosine similarity."""
    return [l2_normalize([x - ci for x, ci in zip(v, c)]) for v in vecs]
 
# Build centroids per language from the document-side set
c_ja = centroid(ja_docs)
c_en = centroid(en_vecs)
 
ja_q2 = remove_centroid(ja_vecs, c_ja)     # the query is Japanese, so use the JA centroid
ja_d2 = remove_centroid(ja_docs, c_ja)
en_d2 = remove_centroid(en_vecs, c_en)
 
print("centered:", evaluate_mixed(ja_q2, en_d2, ja_d2))

On my corpus, mixed-pool recall@1 climbed back to roughly the baseline level. The original symptom — the twin never showing up — was gone.

A handful of lines, once you know the trick. But this is not a procedure documented anywhere in the Gemini docs. It lives in the space after the API returns, where you post-process according to the shape of your own corpus.

Where this bites in production

Centroids depend on data. That turned out to be harder than the code.

Trap	What happens	What to do
Centroid built from too few samples	Noise leaks in and rankings wobble	Use at least ~200 documents per language
Applying a document centroid to queries	Short and long text distributions differ; residual skew remains	Keep a query-side centroid if you have the volume; otherwise reuse the document one and measure the gap
Forgetting to re-normalise	Norms diverge and cosine stops meaning anything	L2-normalise immediately after subtracting
Model swapped, centroid stale	You subtract a centroid from a different space entirely; quality decays silently	Store the centroid tagged with its model name and rebuild on every re-index
Corpus grew, centroid untouched	The population drifts and the offset slowly stops fitting	Refresh it alongside re-indexing

The last two are the dangerous ones. Centroid subtraction never raises an error. A stale centroid computes fine and returns results. Only the ranking quietly degrades. I now write model, built_at, and n_docs into the centroid file and cross-check it against the index metadata. The mechanics of the model swap itself follow the procedure in re-indexing an embedding migration without downtime.

One more caveat. Centroid removal can shave a little accuracy off same-language search. Within one language the offset was never in the way, so removing it is pure information loss. If you serve both cross-lingual and same-language queries, consider storing both the raw and the centred vectors. Storage doubles; the embedding API call count stays at one.

Centroid removal, translation bridge, per-language indexes

Three workable answers. Initially I could only see the second one.

Approach	Mechanism	Extra calls	Added latency	Strength	Weakness
Language-centroid removal	Subtract the per-language mean, re-normalise	none	near zero (vector math)	small implementation, no runtime cost	corpus bookkeeping; slight same-language regression
Translation bridge	Translate the query into the target language, then embed	1 generation + 1 embedding	full generation latency	predictable accuracy; scales conceptually to more languages	per-query cost and delay; translation variance moves ranks
Per-language index + RRF	Search each language, fuse by rank	one search per language	small if parallelised	avoids cross-language comparison; per-language tuning	always injects each language's top hits, relevant or not

The translation bridge gives you accuracy you can reason about, at the price of one generation call per query. For a pre-publication check firing a few dozen times a day, that cost rounds to nothing. Put it behind a related-articles widget that runs on every page view and the delay lands directly on your readers.

RRF is conceptually tidy — it never compares scores across languages. It didn't fit my case. The moment you say "top three from Japanese, top three from English," you get three English articles even when no English article is relevant. The scheme has no way to express there is nothing here.

What I picked, and why

I split it by usage.

The pre-publication duplicate check uses the translation bridge. It runs a few times a day, accuracy dominates, and no human is waiting. The Japanese draft gets translated into English before hitting the English index; the Japanese index is queried directly. Cross-language comparison never occurs. That workflow as a whole lives in the pre-publication gate for topic cannibalisation.

Related articles on the page uses centroid removal. I could not justify inserting a generation call somewhere a reader is waiting. Subtracting a centroid is matrix arithmetic and finishes below perception. Zero additional API calls is, for someone running this as an indie developer and watching the monthly bill, reason enough on its own.

Put plainly: translation bridge where there is a latency budget, centroid removal where there is none. The question is where you are willing to trade a little accuracy for zero runtime cost.

I have stopped trying to settle these questions once and for all. The same problem, called three orders of magnitude more often, is not the same problem.

The order I would introduce this

If I were putting it into your environment:

Freeze 50-100 translation pairs. Sample randomly and record the seed. Never change the set afterwards — changing it destroys comparability with earlier runs.
Measure with single-language candidates. If recall is poor here, language is not your issue. Revisit task_type and chunking first.
Mix the other language into the pool and measure again. The gap between steps 2 and 3 is your language component. If it's small, stop here.
Build centroids and re-measure. At least 200 documents per language, and never skip the re-normalisation.
Also measure same-language recall. Confirm centroid removal hasn't hurt it. If the regression is unacceptable, look at storing both vector sets.
Stamp the centroid with model name, build date, and document count. Cross-check against index metadata and fail loudly at startup on a mismatch.
Put centroid rebuilding into the re-index runbook. Anything not written into the procedure eventually gets skipped.

Wrapping up

If you run a multilingual index, measure cross-lingual recall with translation pairs before anything else. If you already publish the same content in two languages, your evaluation set is sitting there right now. Labelling cost: zero.

Something that looks healthy in one language can wear a completely different face once you mix. I learned that from a screen showing ten Japanese results.

Whether it happens in your environment is an open question — and having that number is what separates the next decision from a guess.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.