When Version Numbers Stopped Meaning Generations: Rebuilding Cost Attribution That Parsed Model IDs

One line, three implementations, three answers

Before fixing the rollup, I audited the selection path. "Pick the newest from this list" can be written several ways.

# latest_probe.py — three ways to pick the newest
import re
 
CANDIDATES = ["gemini-3.5-pro", "gemini-3.6-flash",
              "gemini-3.10-flash", "gemini-3.5-flash-lite"]
 
print("[1] lexicographic max()      ->", max(CANDIDATES))
 
def numkey(m):
    x = re.search(r"(\d+)\.(\d+)", m)
    return (int(x.group(1)), int(x.group(2))) if x else (0, 0)
 
print("[2] numeric (major,minor) max ->", max(CANDIDATES, key=numkey))
print("[3] float(version) max        ->",
      max(CANDIDATES, key=lambda m: float(re.search(r"(\d+\.\d+)", m).group(1))))
 
print()
print("float('3.10') =", float("3.10"), " float('3.6') =", float("3.6"),
      "->", float("3.10") > float("3.6"))
print("'3.10' > '3.6' (str) =", "3.10" > "3.6")

[1] lexicographic max()      -> gemini-3.6-flash
[2] numeric (major,minor) max -> gemini-3.10-flash
[3] float(version) max        -> gemini-3.6-flash
 
float('3.10') = 3.1  float('3.6') = 3.6 -> False
'3.10' > '3.6' (str) = False

Three plausible implementations, three different answers.

I included 3.10 because a two-digit minor could appear at any time. Under both string comparison and float conversion, 3.10 sorts below 3.6. The moment a version is treated as a decimal, .10 collapses into .1.

And note that [1] returned gemini-3.6-flash over gemini-3.5-pro. If reasoning-heavy work is routed by the phrase "newest model," it drifts toward Flash on nothing but digit comparison. No error, a valid response, and only the quality moves.

This rhymes with alias resolution drifting underneath you. I wrote about pinning call targets from a different angle in the day I stopped following gemini-flash-latest.

The rollup side hurt more

A wrong selection eventually surfaces as response quality. A wrong rollup lands straight in the ledger and stays there.

I loaded my usage ledger into SQLite and put the regex-derived tier next to a registry join. The ledger below reproduces the shape only — substitute your own export.

# attribution.py — regex derivation vs. registry join
import re, sqlite3
 
con = sqlite3.connect(":memory:")
con.executescript("""
CREATE TABLE usage_fact (day TEXT, model_id TEXT, calls INT, in_tok INT, out_tok INT);
CREATE TABLE model_registry (
  model_id TEXT PRIMARY KEY, family TEXT, generation TEXT, tier TEXT, modality TEXT
);
""")
 
rows = [
    ("2026-07-01", "gemini-3.5-flash",            9800, 4_900_000, 980_000),
    ("2026-07-01", "gemini-3.5-flash-lite",      41000, 8_200_000, 410_000),
    ("2026-07-01", "gemini-3.1-flash-lite-image", 2600,   520_000,  26_000),
    ("2026-07-01", "gemini-3.5-pro",               420, 1_260_000, 336_000),
    ("2026-07-22", "gemini-3.6-flash",            9100, 4_550_000, 755_000),
    ("2026-07-22", "gemini-3.5-flash-lite",      40500, 8_100_000, 405_000),
    ("2026-07-22", "gemini-3.1-flash-lite-image", 2700,   540_000,  27_000),
    ("2026-07-22", "gemini-3.5-pro",               400, 1_200_000, 320_000),
    ("2026-07-22", "gemini-omni-flash",             90,    45_000,  18_000),
]
con.executemany("INSERT INTO usage_fact VALUES (?,?,?,?,?)", rows)
 
reg = [
    ("gemini-3.5-flash",            "flash",      "3.5", "flash",      "text"),
    ("gemini-3.6-flash",            "flash",      "3.6", "flash",      "text"),
    ("gemini-3.5-flash-lite",       "flash-lite", "3.5", "flash-lite", "text"),
    ("gemini-3.5-pro",              "pro",        "3.5", "pro",        "text"),
    ("gemini-3.1-flash-lite-image", "image",      "3.1", "flash-lite", "image"),
    ("gemini-omni-flash",           "omni",       "n/a", "omni",       "video"),
]
con.executemany("INSERT INTO model_registry VALUES (?,?,?,?,?)", reg)
 
PAT = re.compile(r"^gemini-(\d+)\.(\d+)-([a-z]+)")
 
def derive(mid):
    h = PAT.match(mid)
    return (f"{h.group(1)}.{h.group(2)}", h.group(3)) if h else ("unknown", "unknown")
 
print(f"{'tier(regex)':<14}{'out_tok':>12} | {'tier(registry)':<16}{'out_tok':>12}")
a = {}
for mid, out in con.execute("SELECT model_id, SUM(out_tok) FROM usage_fact GROUP BY model_id"):
    k = derive(mid)[1]
    a[k] = a.get(k, 0) + out
b = dict(con.execute("""SELECT r.tier, SUM(f.out_tok) FROM usage_fact f
                        JOIN model_registry r USING(model_id) GROUP BY r.tier"""))
for k in sorted(set(a) | set(b)):
    print(f"{k:<14}{a.get(k,0):>12,} | {k:<16}{b.get(k,0):>12,}")

tier(regex)        out_tok | tier(registry)       out_tok
flash            2,603,000 | flash              1,735,000
flash-lite               0 | flash-lite           868,000
omni                     0 | omni                  18,000
pro                656,000 | pro                  656,000
unknown             18,000 | unknown                    0

All 868,000 output tokens belonging to flash-lite were absorbed into flash, inflating that bucket to 2,603,000 against an actual 1,735,000 — roughly 1.5x.

Breaking the bucket down shows the shape of the contamination:

  gemini-3.1-flash-lite-image       53,000  (  2.0% of regex 'flash')
  gemini-3.5-flash                 980,000  ( 37.6% of regex 'flash')
  gemini-3.5-flash-lite            815,000  ( 31.3% of regex 'flash')
  gemini-3.6-flash                 755,000  ( 29.0% of regex 'flash')

Over 30 percent of what I was reading as "the Flash line" belonged to a different price band entirely. Text generation and image generation shared one container. I had built the ledger to see per-feature economics, and the container itself was mixed. It shares a root with the attribution problem I covered in seeing unit economics per feature.

The measurement of a price improvement gets diluted

This is the part that stung.

Gemini 3.6 Flash is described as consuming fewer output tokens for the same work. If you are deciding whether to switch, comparing output tokens per call before and after is the natural check.

I computed that metric under both groupings.

# effect.py — the same switch, measured two ways
def per_call(day, pred):
    c = o = 0
    for mid, cl, ot in con.execute(
            "SELECT model_id, calls, out_tok FROM usage_fact WHERE day=?", (day,)):
        if pred(mid):
            c += cl
            o += ot
    return o / c if c else 0
 
regex_flash = lambda m: (PAT.match(m).group(3) if PAT.match(m) else None) == "flash"
reg_flash   = lambda m: m in ("gemini-3.5-flash", "gemini-3.6-flash")
 
for label, pred in (("regex tier='flash'", regex_flash),
                    ("registry tier='flash'", reg_flash)):
    b, a = per_call("2026-07-01", pred), per_call("2026-07-22", pred)
    print(f"{label:<24} out_tok/call {b:7.2f} -> {a:7.2f}   ({(a-b)/b*100:+5.1f}%)")

regex tier='flash'       out_tok/call   26.52 ->   22.70   (-14.4%)
registry tier='flash'    out_tok/call  100.00 ->   82.97   (-17.0%)

The reduction reads -17.0% under the registry join and -14.4% under the regex rollup. On its own that gap might pass for noise.

Look at the levels instead: 26.52 versus 100.00, a gap of roughly 3.8x. The high-volume lightweight model sat in the same container and dragged the average down.

Any ceiling or budget drawn from that level would have been drawn far from reality. The regex view survives a directional question — did it go down? — but not a decision that uses the absolute number.

Treating the model ID as an opaque key

The fix reduces to one rule: never derive properties from a model ID.

Keep the ID as a meaningless primary key. Put generation, tier, modality, shutdown date, and price in a declarative registry. Join to that registry when rolling up.

Store the raw ID in the fact table

The key detail is not writing derived values into the fact table. If you persist a generation column, fixing the registry does not fix the rows already written.

Store only the raw ID and every historical rollup corrects itself the moment the registry is corrected. The one thing that saved me after I found the regex bug was that I had done this by accident.

Give the registry a validity window

Model properties change. Prices are revised, shutdown dates get added.

CREATE TABLE model_registry (
  model_id    TEXT NOT NULL,
  generation  TEXT NOT NULL,
  tier        TEXT NOT NULL,
  modality    TEXT NOT NULL,
  shutdown_on TEXT,
  valid_from  TEXT NOT NULL,
  valid_to    TEXT,
  PRIMARY KEY (model_id, valid_from)
);
 
SELECT r.tier, SUM(f.out_tok) AS out_tok
FROM usage_fact f
JOIN model_registry r
  ON r.model_id = f.model_id
 AND f.day >= r.valid_from
 AND (r.valid_to IS NULL OR f.day < r.valid_to)
GROUP BY r.tier;

A date-bounded join lets you rebuild rollups that span a price revision.

Let unknown IDs stay unknown

The resolver now returns "I do not know" rather than a guess.

from dataclasses import dataclass
 
@dataclass(frozen=True)
class ModelFacts:
    generation: str
    tier: str
    modality: str
    known: bool
 
UNKNOWN = ModelFacts("unknown", "unknown", "unknown", known=False)
 
def resolve(model_id: str, registry: dict[str, ModelFacts]) -> ModelFacts:
    facts = registry.get(model_id)
    if facts is None:
        # Do not round to a nearby name. A rounded value wears the face of a correct one.
        return UNKNOWN
    return facts

My first version used a startswith fallback that snapped unrecognized IDs onto the nearest match, because an unknown band on the chart bothered me.

That was the worst call in the whole exercise. A rounded value looks exactly like a correct one on a chart. Missing data is visible; wrong attribution is not. Now the unknown band stands on its own, and when it grows I update the registry.

Since the registry carries shutdown dates, deadline monitoring reads from the same table. The detection logic is unchanged from catching deprecated models in CI; only the source of truth moved.

What ran counter to my expectations

Three things landed the opposite of what I assumed.

Successful parses are more dangerous than failed ones. An exception always reaches somebody. A silent mismatch reaches nobody. The trap lived on the side that raised no error.

Fixing the regex solves only half the problem. It corrects rollups from today forward. If derived values were persisted, the historical bands stay wrong. What actually saved the recovery was keeping raw IDs in the fact table, not the regex fix.

The urge to eliminate unknown was the expensive one. A cosmetic fallback invalidated the entire measurement of a price improvement. A gap on a dashboard is not a defect to paper over; it is a signal arriving.

What I deliberately left alone

I did not rebuild everything.

The call path still passes the model string straight from configuration. The registry is read only by rollups and monitoring. Putting a table lookup on the request path means the app stops when the registry does.

I also limited the historical recomputation to the last three months. Anything older is not feeding a migration decision, so there is no return on touching it.

If you reference only two or three models, this is probably more machinery than the problem deserves. Consolidating your model strings into a single location will pay off faster than a registry will. When I was running a single pipeline for App Store metadata, that was enough.

Where to point your next hour

Start by grepping your live rollup queries for anywhere a model ID is taken apart as a string. substr, split_part, LIKE 'gemini-3.5-%' — they all rest on the same assumption.

Where you find one, expand a single day of that band by model ID. Contamination shows up immediately.

I am still partway through this myself, undecided on how much of the shutdown-date maintenance to automate. If you have solved that part in your own setup, I would be glad to learn from it.