The Day We Went From 30 Categories to 34 — Reclassifying 1,180 Assets Instead of 8,142

A Ledger That Remembers taxonomy_version and Margin

None of this works unless past classifications recorded which taxonomy produced them and how confident the model was. My first version stored only the label string, which made this kind of reasoning impossible and cost me one full rerun to learn.

The ledger now looks like this:

CREATE TABLE asset_labels (
  asset_id         TEXT PRIMARY KEY,
  label            TEXT NOT NULL,      -- chosen label
  runner_up        TEXT,               -- second-place label
  confidence       REAL NOT NULL,      -- top-1 confidence, 0.0-1.0
  runner_up_conf   REAL,
  margin           REAL NOT NULL,      -- confidence - runner_up_conf
  taxonomy_version INTEGER NOT NULL,   -- taxonomy in effect at classification time
  classified_with  TEXT NOT NULL,      -- pinned model version, not an alias
  classified_at    TEXT NOT NULL
);
CREATE INDEX idx_margin ON asset_labels(margin);
CREATE INDEX idx_label_version ON asset_labels(label, taxonomy_version);

With runner_up and margin in place, "which assets were on a boundary" becomes a single query. Asking the model for two labels instead of one costs roughly 20 extra output tokens per image — cheap insurance.

Classification itself goes through structured output:

from pydantic import BaseModel, Field
from google import genai
 
class LabelGuess(BaseModel):
    label: str
    confidence: float = Field(ge=0.0, le=1.0)
 
class Classification(BaseModel):
    top: LabelGuess
    runner_up: LabelGuess
 
client = genai.Client(api_key="YOUR_API_KEY")
MODEL = "gemini-3.5-flash"  # pin the version; record it in the ledger
 
def classify(image_bytes: bytes, categories: list[str]) -> Classification:
    resp = client.models.generate_content(
        model=MODEL,
        contents=[
            {"inline_data": {"mime_type": "image/jpeg", "data": image_bytes}},
            "Pick the single best category and one runner-up from the list "
            "below, with a confidence between 0.0 and 1.0 for each.\n"
            + "\n".join(f"- {c}" for c in categories),
        ],
        config={
            "response_mime_type": "application/json",
            "response_schema": Classification,
            "temperature": 0.0,
        },
    )
    return Classification.model_validate_json(resp.text)

Setting temperature to 0 does not make the output fully deterministic. Sending the same image ten times gave the same top-1 label every time, but confidence wandered between 0.82 and 0.87. Any margin threshold has to sit comfortably above that noise band.

Finding Impacted Labels With Embeddings, Before Sending Any Image

Which of the 30 existing labels does "Night Cityscape" sit next to? You can answer that without a single image call — the category names and their one-line definitions are enough.

import numpy as np
from google import genai
 
client = genai.Client(api_key="YOUR_API_KEY")
EMBED_MODEL = "gemini-embedding-2"
 
def embed(texts: list[str]) -> np.ndarray:
    resp = client.models.embed_content(
        model=EMBED_MODEL,
        contents=texts,
        config={"task_type": "SEMANTIC_SIMILARITY", "output_dimensionality": 768},
    )
    vecs = np.array([e.values for e in resp.embeddings], dtype=np.float32)
    # Truncated dimensions are no longer unit-norm. Renormalize.
    return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
 
def impacted_labels(old_labels: dict[str, str],
                    new_labels: dict[str, str],
                    threshold: float = 0.62) -> set[str]:
    """Both dicts map {label name: one-sentence definition}."""
    old_names = list(old_labels)
    old_vec = embed([f"{k}: {v}" for k, v in old_labels.items()])
    new_vec = embed([f"{k}: {v}" for k, v in new_labels.items()])
 
    sim = new_vec @ old_vec.T          # cosine similarity, shape (new, old)
    impacted: set[str] = set()
    for i in range(sim.shape[0]):
        for j, old in enumerate(old_names):
            if sim[i, j] >= threshold:
                impacted.add(old)
    return impacted

Two details matter here, and both fail silently when you get them wrong. Passing task_type="SEMANTIC_SIMILARITY" changes the embedding geometry; omitting it degrades the comparison without any warning. And truncating output_dimensionality leaves the vectors un-normalized, so every similarity comes out slightly low. I fell into that trap and compensated by lowering the threshold to 0.45, which happily pulled unrelated labels into the impacted set.

Here is what the four new categories actually matched:

After deduplication: 1,003 impacted assets, or 12.3% of the library.

The Selection Rule: Before and After

The old backfill was a straightforward loop over everything.

# Before: unconditionally reclassify every asset
def backfill_all(conn, categories, taxonomy_version):
    rows = conn.execute("SELECT asset_id FROM asset_labels").fetchall()
    for (asset_id,) in rows:                     # 8,142 calls
        result = classify(load_image(asset_id), categories)
        upsert(conn, asset_id, result, taxonomy_version)

The current one narrows the set first.

# After: reclassify impacted-label assets ∪ low-margin assets
MARGIN_THRESHOLD = 0.15   # 5x the observed confidence noise band (+/-0.03)
 
def select_targets(conn, impacted: set[str], current_version: int) -> list[str]:
    placeholders = ",".join("?" for _ in impacted)
    by_label = conn.execute(
        f"SELECT asset_id FROM asset_labels "
        f"WHERE label IN ({placeholders}) AND taxonomy_version < ?",
        (*impacted, current_version),
    ).fetchall()
 
    by_margin = conn.execute(
        "SELECT asset_id FROM asset_labels "
        "WHERE margin < ? AND taxonomy_version < ?",
        (MARGIN_THRESHOLD, current_version),
    ).fetchall()
 
    return sorted({r[0] for r in by_label} | {r[0] for r in by_margin})
 
def backfill_selective(conn, categories, taxonomy_version, impacted):
    targets = select_targets(conn, impacted, taxonomy_version)
    print(f"targets: {len(targets)} / {count_all(conn)}")
    for asset_id in targets:                     # 1,180 calls
        result = classify(load_image(asset_id), categories)
        upsert(conn, asset_id, result, taxonomy_version)
 
    # Assets we deliberately skipped still advance to the new taxonomy version.
    conn.execute(
        "UPDATE asset_labels SET taxonomy_version = ? "
        "WHERE taxonomy_version < ? AND asset_id NOT IN "
        "(SELECT value FROM json_each(?))",
        (taxonomy_version, taxonomy_version, json.dumps(targets)),
    )
    conn.commit()

Skip that final UPDATE and the next taxonomy bump will keep dragging those stale rows into taxonomy_version < ? forever. Choosing not to reclassify is itself a decision, and the ledger should record it.

The taxonomy_version < current_version predicate also buys idempotency. When a backfill dies partway through — and at 8,000 assets it eventually will, on a transient API error if nothing else — rerunning it skips whatever already completed.

Results, and the One Asset We Missed

1,003 impacted assets plus 341 low-margin assets, deduplicated, gave 1,180 targets.

Compared against the full pass, the selective run missed exactly one asset: a photo labeled "Cats" with a sprawling night skyline behind the subject. The full pass moved it to "Night Cityscape." Its confidence was 0.79 and its margin 0.31 — outside the impacted labels, and comfortably above the margin threshold.

Catching it would mean loosening the threshold, and raising the margin cutoff to 0.35 balloons the target set to 2,890. I declined to send 1,710 extra images to rescue one. In a wallpaper app, a single misfiled image out of 8,142 is not the kind of failure that strands a user.

Reasonable people will draw this line differently. Where the cost of an error is asymmetric — medical imaging, credit decisions — the full rerun is the defensible choice. Selective reclassification is an optimization that only holds in domains where mistakes are cheap, and that assumption deserves a comment in the code as much as a paragraph in an article.

Folding Taxonomy Changes Into the Release Checklist

Rather than leaving this as a one-off script, I turned it into three steps that run before every App Store submission that touches categories.

1. Update the category definition file and bump taxonomy_version. Definitions live in one file as "label: one-sentence description," feeding both the embedding step and the classification prompt. Two copies of the definition means the impacted-label math and the model's instructions drift apart.
2. Dry-run impacted_labels() and read the output with your own eyes. Embedding similarity sometimes disagrees with intuition. When "Autumn" pulled in "Food," I rewrote the definition rather than nudging the threshold. Fixing the sentence produced far more stable results than fixing the number.
3. Run the selective backfill, then diff the label distribution against the previous version. If an existing category loses far more assets than expected, the new category is defined too broadly. Adding "Minimal" cut "Geometric" by 41%, which sent me back to tighten the definition to "unornamented compositions built on a single dominant color."

Step three turned out to be the most instructive. Selective reclassification began as a way to spend less money, but narrowing the target set forced me to articulate, in advance, exactly where a new category would encroach on the old ones. The taxonomy itself got sharper as a result. Sometimes a cost constraint is what teaches you the design.

Next time I add categories, I plan to start by plotting a histogram of the margin column. If low-margin assets cluster under one particular label, that may be a signal to split an existing category rather than add a new one.