Trimming for Real with output_dimensionality

Trimming is specified with a single API parameter: you pass the dimension count you want to output_dimensionality.

from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_GEMINI_API_KEY")
 
def embed_3072(text: str) -> list[float]:
    """Receive the default 3072 dimensions as-is."""
    res = client.models.embed_content(
        model="gemini-embedding-001",
        contents=text,
        config=types.EmbedContentConfig(task_type="RETRIEVAL_DOCUMENT"),
    )
    return res.embeddings[0].values  # length 3072, already normalized
 
def embed_768(text: str) -> list[float]:
    """Receive only the leading 768 dimensions."""
    res = client.models.embed_content(
        model="gemini-embedding-001",
        contents=text,
        config=types.EmbedContentConfig(
            task_type="RETRIEVAL_DOCUMENT",
            output_dimensionality=768,
        ),
    )
    return res.embeddings[0].values  # length 768, but NOT normalized

This part is easy. Pass output_dimensionality=768 and you get a length-768 vector back. The problem is what comes next. As the comment in the code above already says, 768-dimensional vectors are not normalized. Use them without knowing this, and your search results break silently.

At 768 Dimensions, Always Re-normalize: Skip This and Search Breaks Silently

In the gemini-embedding-001 spec, only the 3072-dimensional output comes pre-normalized (norm of 1); any other dimension is not normalized. Vectors trimmed to the leading dimensions come back with L2 norms all over the place.

Why is this a problem? Many vector databases compute distance by default with dot product rather than cosine similarity. For normalized vectors, dot product and cosine similarity coincide. But take a dot product over unnormalized vectors and a vector with a large norm rises to the top regardless of its content. A document that should be entirely unrelated sits high in the results purely because its norm happens to be large. Because this happens with no error and no warning, the cause is extremely hard to pin down.

The fix is simple: always L2-normalize trimmed vectors before storing and searching.

import numpy as np
 
def l2_normalize(vec: list[float]) -> list[float]:
    """Scale the L2 norm to 1. Mandatory for trimmed vectors."""
    arr = np.asarray(vec, dtype=np.float32)
    norm = np.linalg.norm(arr)
    if norm == 0.0:
        return arr.tolist()  # return zero vectors unchanged
    return (arr / norm).tolist()
 
def embed_768_normalized(text: str) -> list[float]:
    res = client.models.embed_content(
        model="gemini-embedding-001",
        contents=text,
        config=types.EmbedContentConfig(
            task_type="RETRIEVAL_DOCUMENT",
            output_dimensionality=768,
        ),
    )
    return l2_normalize(res.embeddings[0].values)

I once skipped this single step and spent half a day chasing a search that was "somehow off." The code ran fine, threw no exceptions, and returned plausible-looking results. That is exactly what makes it hard to catch. Trim the dimensions, and you must re-normalize. Routing both the storage side and the query side through the same function is the one sure way to enforce it.

One operational note: the most common accident is an asymmetry where you normalize on storage but forget on query, or vice versa. Funneling the embedding entry point through a single abstraction layer, as below, prevents that asymmetry structurally.

Choosing Among 768, 1536, and 3072

Choosing a dimension is a judgment about where to put the center of gravity in the triangle of accuracy, cost, and latency. Here are the rules of thumb I actually use.

768 dimensions suits tasks with high record counts where search is about "scooping up roughly nearby things." In my case, semantic clustering of store reviews and classifying wallpapers by visual similarity were well served by it. The benchmark accuracy gap is tiny, while the storage and latency benefits are largest.

1536 dimensions is the middle choice for when you want one more notch of accuracy but 3072 is overkill. Storage is half of 3072, and thanks to MRL the accuracy nearly matches 3072. If you are unsure, starting here is the safe move.

3072 dimensions I reserve for cases where search results carry the core of the business and a slight accuracy gap maps directly to experience: a paid document-search feature, say, or a domain where mis-retrieval erodes trust. Alternatively, you can take the pragmatic stance of using 3072 only for the "refinement" stage of the two-stage search described below.

Summarized on one sheet, at 300,000 records in float32:

• 3072 dimensions: 12 KB each, about 3.5 GB total
• 1536 dimensions: 6 KB each, about 1.8 GB total
• 768 dimensions: 3 KB each, about 0.9 GB total

This difference scales nearly linearly into the managed DB monthly cost. The more records you have, the heavier that first dimension choice weighs.

Migrating Dimensions Without Stopping the Existing Index

If you are already running an index at 3072 and want to move to 768, you are not changing the model itself, so the embedding space is shared. Even so, you cannot mix 3072 vectors and 768 vectors in the same index for search, because the dimension counts differ. So migration means standing up a separate index for the new dimension, rebuilding in the background, confirming quality, and then switching the read path over.

For zero-downtime reindexing when you swap the embedding model itself, I covered the full design in a separate article, a zero-downtime reindexing design for switching Gemini embedding models. Switching dimensions follows the same logic: you can reuse the dual-write and shadow-verification framework as-is. Here I will add only the dimension-specific points.

In the regeneration job, always record which dimension each vector was built at.

def build_record(doc_id: str, text: str, dim: int) -> dict:
    res = client.models.embed_content(
        model="gemini-embedding-001",
        contents=text,
        config=types.EmbedContentConfig(
            task_type="RETRIEVAL_DOCUMENT",
            output_dimensionality=dim,
        ),
    )
    vec = res.embeddings[0].values
    if dim != 3072:
        vec = l2_normalize(vec)  # re-normalize anything other than 3072
    return {
        "id": doc_id,
        "embedding": vec,
        "embedding_dim": dim,            # stamp the dimension into metadata
        "embedding_model": "gemini-embedding-001",
    }

Carrying embedding_dim on the record lets you tell at a glance, mid-migration, whether a vector is the old or new dimension, and lets you detect dual-write misses. It is humble, but it is one line of metadata like this that makes a migration safe.

Coarse Filtering with 768, Refinement with 3072

Trimming dimensions lowers cost, but there are situations where you do not want to compromise accuracy at all. For those, the design I favor is coarse-to-fine retrieval. I consider it the design that best exploits the Matryoshka property.

The idea goes like this. First, gather a wide set of candidates from the 768-dimensional index. That stage is fast and light. Then re-rank only those candidates using 3072-dimensional vectors. Compared with searching all records at 3072, the refinement targets only a few dozen items, so the cost and latency are negligible.

def two_stage_search(query: str, coarse_index, fine_store, top_k: int = 10):
    """Cast a wide net with 768, re-rank precisely with 3072."""
    # Stage 1: narrow to 100 with the 768-dim index
    q768 = embed_768_normalized(query)
    candidates = coarse_index.search(q768, top_k=100)  # [(id, _score768), ...]
 
    # Stage 2: re-rank only the candidates with 3072-dim vectors
    q3072 = embed_query_3072(query)  # 3072 query vector (normalized)
    rescored = []
    for doc_id, _ in candidates:
        v3072 = fine_store.get_vector(doc_id)  # fetch the candidate's 3072 vector
        score = float(np.dot(q3072, v3072))    # normalized, so dot == cosine
        rescored.append((doc_id, score))
 
    rescored.sort(key=lambda x: x[1], reverse=True)
    return rescored[:top_k]

With this structure you do not need to keep the huge 3072 index resident in memory. Keep only the 768 index in a fast tier, and pull the 3072 vectors from cheaper storage for just the candidates. Shifting from "hold everything at high precision" to "keep most of it light and only the critical part heavy" is what makes cost and accuracy compatible.

One note on the query-side 3072 vector embed_query_3072: build it with task_type="RETRIEVAL_QUERY". Using different task_type values on the document side and the query side is a search-accuracy fundamental independent of the dimension discussion.

The Configuration I Currently Choose

Finally, let me record the configuration actually running across my six sites and several apps, along with the reasons behind it.

For high-count tasks with a wide accuracy tolerance (review classification, wallpaper classification), I use single-stage search at 768 dimensions. I bury the re-normalization inside the embedding function and always route both storage and query through the same function. That way the asymmetry accident cannot happen by construction.

For the more paid-leaning features where search quality maps directly to experience, I use two-stage search that filters coarsely with 768 and refines with 3072. It greatly reduces resident memory compared with all-records-at-3072, while the search quality the user touches barely changes.

There is no single right answer for choosing a dimension. But if you keep stacking up records at "the default 3072 for now," a future version of you, with far more records, will be holding their head in front of a quietly inflated bill. Since Matryoshka has already pushed the meaning to the front, there is no reason to leave that benefit on the table. That is my current conclusion.

Start by comparing 768 and 3072 on your own task with the same data, and see with your own eyes how much the results change. In most cases, you will be a little surprised by how small the difference is. Thank you for reading.