After the Vertex AI Rename, I Measured My Search Stack. Ranking Held. Aggregation Did Not.

The First Surprise

I started with Recall@3: does a correct document appear anywhere in the top three?

For topical queries, all three conditions scored 1.000. No difference whatsoever.

In hindsight the reason is plain. "How do I handle a quota overage" carries "quota", "overage", "handle". Those terms have lower document frequency than "vertex" or "ai", so BM25 weights them higher. The product name was never doing the discriminating work. Seeing that in the table, the dread I'd carried overnight turned into something closer to anticlimax, and I still remember that feeling.

I had assumed the scary part of a rename was retrieval failure. For any query that carries a topic, the name is a supporting actor.

Tightening the metric to P@5 — how many of the top five are actually relevant — finally separates the conditions.

Doing nothing has exactly one weak spot: 0.440 on name-only queries using the new terminology. Barely two of five results are relevant.

The Second Surprise

Look at column B.

The bulk rewrite pushes new-name queries to a perfect 1.000. Then the same edit drops legacy-name queries from 0.920 to 0.080. Topical legacy queries fall from 0.760 to 0.400 — roughly half.

Nobody stops using the old name on rename day. Existing users are precisely the population most attached to the old vocabulary. In my own help search, I would expect months before the new phrasing dominates.

So B is a trade where you sacrifice the readers you have for readers who have not arrived yet. That is a net loss, not a migration.

Condition C lands everything in a 0.840–0.920 band. It never wins a column outright, and it never collapses in one either. It also requires no reindex, which means it can ship the same day the rename lands.

C is not free. Averaged over 200 runs, a plain query took 102.7 µs and an expanded one 125.6 µs — about 22% more. The relative number is not nothing, but 23 µs of absolute overhead disappears into any architecture with a network hop in it. Measure it against your own stack before deciding it matters.

What Actually Broke

Everything above concerns retrieval. Here is the number that made me stop.

Take the twelve platform-side documents and apply a naive exact-match gate:

The moment you update that string to the new name, ten of twelve documents vanish. And unlike search, this layer hands back two plausible-looking results, so nothing announces that it emptied out.

A ranker degrades visibly. Something wrong stays near the top and a human notices. An exact-match gate returns two rows and exits zero.

I counted the places in my own setup with that shape:

1. Cost aggregation — monthly billing grouped by product name. Renamed spend silently falls into "other".
2. Alert routing — notification targets chosen by matching product names in log messages. Unmatched alerts flow to the default channel and get buried.
3. Document tagging — a batch job that files runbooks by product. Only documents written after the rename land in a different bucket.
4. A CI banned-terms check — the very mechanism meant to catch deprecated product names had those names hardcoded.

None of the four raise an exception. The numbers just get a little smaller. This is the kind of breakage you discover weeks later, staring at a monthly report that feels slightly off.

Where the Alias Layer Belongs

The measurements make the placement decision concrete.

That last row is the conclusion I came away with. Document bodies should keep the name they were written with. A chunk is a record of what was true when someone wrote it, not a surface to repaint every time marketing moves.

Name resolution belongs to the reader, not the record. Structure it that way and the next rename costs you nothing at index time.

Implementation

One source of truth for aliases, consumed by all three layers.

# -*- coding: utf-8 -*-
"""Single source of truth for product aliases: search, filtering and aggregation all read this."""
from dataclasses import dataclass
from datetime import date
 
@dataclass(frozen=True)
class ProductAlias:
    canonical: str                    # the aggregation key, distinct from any display name
    surfaces: tuple                   # every spelling that actually appears in docs or queries
    renamed_on: date | None = None    # rename date, kept so aliases can expire
    note: str = ""
 
ALIASES: list[ProductAlias] = [
    ProductAlias(
        canonical="google.ai.platform",
        surfaces=("vertex ai", "vertex-ai", "vertexai", "aiplatform",
                  "gemini enterprise agent platform"),
        renamed_on=date(2026, 7, 30),
        note="Renamed from Vertex AI on 2026-07-30. Keep matching the old name for now.",
    ),
    ProductAlias(
        canonical="google.ai.studio",
        surfaces=("ai studio", "google ai studio", "aistudio"),
    ),
]
 
# Build the reverse map once at startup
_SURFACE_TO_CANONICAL = {
    s: a.canonical for a in ALIASES for s in a.surfaces
}
# Longest surface first, otherwise short aliases eat substrings of longer names
_SURFACES_BY_LENGTH = sorted(_SURFACE_TO_CANONICAL, key=len, reverse=True)
 
 
def canonical_keys(text: str) -> set[str]:
    """For filtering and aggregation: which products does this text mention?"""
    tl = text.lower()
    found, consumed = set(), tl
    for surface in _SURFACES_BY_LENGTH:
        if surface in consumed:
            found.add(_SURFACE_TO_CANONICAL[surface])
            # Mask what we matched so it cannot be counted twice
            consumed = consumed.replace(surface, " " * len(surface))
    return found
 
 
def expand_query(q: str) -> str:
    """For search: append every other spelling of any product named in the query."""
    ql = q.lower()
    extra = []
    for alias in ALIASES:
        if any(s in ql for s in alias.surfaces):
            extra += [s for s in alias.surfaces if s not in ql]
    return q if not extra else q + " " + " ".join(dict.fromkeys(extra))
 
 
def stale_aliases(today: date, months: int = 18) -> list[ProductAlias]:
    """Aliases are trivial to add and hard to retire.
    Let the machine remember which ones are due for review."""
    out = []
    for a in ALIASES:
        if a.renamed_on and (today - a.renamed_on).days > months * 30:
            out.append(a)
    return out
 
 
if __name__ == "__main__":
    doc = "Vertex AI quota overages return 429. Retry with exponential backoff."
    print(canonical_keys(doc))
    # {'google.ai.platform'}  <- legacy-name doc still resolves
 
    print(canonical_keys("Gemini Enterprise Agent Platform skills are defined in YAML"))
    # {'google.ai.platform'}  <- new name lands on the same key
 
    print(expand_query("Gemini Enterprise Agent Platform IAM roles"))
    # returns the query with legacy spellings appended
 
    print(stale_aliases(date(2026, 7, 31)))
    # []  <- nothing due for review yet

The _SURFACES_BY_LENGTH ordering is where I tripped. I reasoned that "vertex ai" is not a substring of "gemini enterprise agent platform", so dictionary order would be fine. It broke from the other side: the moment I added the short alias aiplatform, it started colliding with longer names and double-counting. The two masking lines are what fixed it.

What This Measurement Cannot Show

Rereading the draft before publishing, I wanted to write down where these numbers are biased by my own setup. Three points.

First, I did not test vector search. An embedding model may place "Vertex AI" and the new name close together, in which case the legacy-query drop I saw with BM25 could be much smaller. Or the new name, barely present in training data, could land farther away. I can't say which without measuring against my own embedding model.

Second, the corpus was my own runbooks. The style and vocabulary are skewed, and product names cluster near the opening lines. Build the same table from someone else's documents and the numbers will move.

Third, the four aggregation spots are simply a count of my setup. Your environment may not produce the same four.

Don't let the range you measured become the range of your conclusion. That is the line I tried to hold in this piece.

Operating This Over Time

Aliases accumulate. Left alone, the surface set grows until it starts dragging in unrelated documents.

That is what renamed_on is for. Anything more than 18 months past its rename gets surfaced as a review candidate — a human still makes the call, but the machine handles the remembering. There is nothing principled about 18 months. My apps take a little over a year for users to migrate off old versions, so I padded that.

Gemini naming is in motion right now. Pretraining for Gemini 4 has been announced, and the 3.5 and 3.6 model families are running side by side. Reading model IDs from configuration is well-trodden advice. The places that aggregate on the string in that config file are usually outside the blast radius anyone checks.

Names and availability keep shifting, so verify the rename details against the Gemini Enterprise release notes rather than this article.

What to Do Today

Start with one command against your own repository.

# Find hardcoded product names in comparisons, filters and aggregations
grep -rn --include='*.py' --include='*.ts' --include='*.sql' \
  -e '"Vertex AI"' -e "'Vertex AI'" -e 'Vertex AI' . \
  | grep -Ev '^\s*#|^\s*//' \
  | grep -E '==|in |LIKE|GROUP BY|filter|includes'

Of the lines that come back, fix the ones that fail without raising first. Search ranking can wait. What this measurement showed me is that the silent shortfall deserves attention before the visible degradation.

If you run the same benchmark, do not stop at 30 documents. Document frequency shifts with corpus size, and so does how much discriminating power a product name carries. Every number here holds only under "30 docs, 2-grams, BM25".

Measure before you reindex. That ordering is the part I will keep.