The Memory Wasn't Lost — It Was Written to a Different Profile

Adding Normalization One Layer at a Time

The fix seems obvious: normalize the key. But "normalize" covers several distinct operations, so instead of applying all of them at once I added them one layer at a time and measured each delta. I didn't want to misattribute the improvement.

def naive(s):
    return json.dumps(s)
 
def canon(s):
    # A: the commonly recommended form — stable key order and spacing
    return json.dumps(s, sort_keys=True, separators=(",", ":"))
 
def canon_strip(s):
    # B: A + strip surrounding whitespace from values
    return json.dumps(
        {k: (v.strip() if isinstance(v, str) else v) for k, v in s.items()},
        sort_keys=True, separators=(",", ":"),
    )
 
def canon_type(s):
    # C: B + coerce values to strings
    return json.dumps(
        {k: str(v).strip() for k, v in s.items()},
        sort_keys=True, separators=(",", ":"),
    )
 
ALLOW = ("app_id", "user_id")
 
def canon_allow(s):
    # D: C + fix which keys are permitted in a scope at all
    return json.dumps(
        {k: str(s[k]).strip() for k in ALLOW if k in s},
        sort_keys=True, separators=(",", ":"),
    )

Applied in sequence against identical traffic:

Stage A moved the number by 0.0 points.

I'll admit I suspected my own script first. Canonical dictionary serialization is the standard answer to this class of problem. Printing the keys side by side finally made it click: sort_keys fixes exactly one failure mode, which is identical content in a different order. What was actually happening was different content. {"user_id": "42"} and {"user_id": 42} are not reorderings of each other, and neither is a dict with one extra key. Sorting only aligns things that were already the same.

Ranked by contribution: string coercion (+7.2 points), whitespace (+4.9 points), and the key allowlist (+25.8 points). The dominant lever was not how values are written but which keys are allowed to participate. The first is a coding-style problem; the second is a design decision. I had been attacking only the coding-style side.

The 90.0% figure is also meaningful rather than arbitrary. Across 4,000 turns there are 400 distinct tenants (200 users x 2 apps), and each one must miss on its first touch. So 1 - 400/4000 = 90.0% is the ceiling attainable under these conditions, and stage D lands exactly on it. The remaining 10% isn't a defect; it's cold start.

I measured the cost of normalization too — {"app_id": "wallpaper", "user_id": " 4211", "device_id": "d2", "locale": "ja-JP"}, 200,000 iterations, median of 7 runs.

A difference of 1.741 µs, against the cost of an avoidable network round trip. There's nothing to deliberate here.

The Read-Side Fallback Produced the Best Number

Normalization is a write-path change. It leaves open the question of what to do with the 1,915 profiles already scattered around. Wanting to avoid a migration, I briefly reasoned my way into this: on a read miss, just look for an existing profile whose user_id matches.

def run_with_fallback(calls, norm):
    store = {}
    hits = miss = leaks = 0
    for app, u, kind, s, tier in calls:
        k = (SCHEMA, norm(s))
        p = store.get(k)
        if p is None:
            # On a miss, search for any profile with a matching user_id
            want = str(s.get("user_id", "")).strip()
            for (_sch, kk), v in store.items():
                try:
                    kd = json.loads(kk)
                except Exception:
                    continue
                if str(kd.get("user_id", "")).strip() == want:
                    p = v
                    if v["owner"] != (app, u):
                        leaks += 1        # returned another tenant's record
                    break
        if p is None:
            miss += 1
            store[k] = {"owner": (app, u), "billing_plan": tier}
        else:
            hits += 1
    n = hits + miss
    return {"hit": 100 * hits / n, "profiles": len(store), "leaks": leaks}

Judged on hit rate alone, this was the winner.

In 1,843 of 4,000 turns it returned someone else's profile. User 42 of the wallpaper app was being handed the subscription state of user 42 of the relaxation app.

The numbers explain themselves cleanly. Matching on user_id alone erases the app_id distinction. Profiles halving from 400 to 200 is precisely the footprint of two apps being merged into one namespace. And the hit rate rising from 90.0% to 95.0% matches 1 - 200/4000 = 95.0%.

In other words, what looked like a 5-point improvement was, exactly and entirely, the collapse of the tenant boundary.

This is the finding that unsettled me most. Optimize for hit rate and you will confidently choose the dangerous option. A permissive read undoes a strict write after the fact. A boundary is something you defend at the write path, not something you patch on read. Loosening the read side because a data migration feels tedious is trading safety for convenience — and the receipt arrives much later.

Across five seeds the pattern held (medians):

The ingestion call count deserves a mention as well: 1,911 before, 400 after — a 4.8x difference. Profile generation invokes a model, so that gap lands directly on the invoice. Published pricing for 3.6 Flash is $1.50/1M input and $7.50/1M output, so measuring your own per-call token usage gives you the monthly delta quickly (confirm current figures in the Gemini API changelog). What actually mattered to me, though, wasn't the money — it was realizing that the billing data had noticed the bug before I did. Nobody was watching the hit rate. The call count was in the rollup every single morning.

Fix the Ceiling First, and Don't Let It Look at Itself

After this, I wanted the next occurrence to announce itself. The relationship from the previous section turns directly into a monitor.

attainable hit rate = 1 - (distinct scopes / turns)

If the observed rate exceeds that ceiling, scopes are being merged more coarsely than intended. Straightforward enough. The first version I wrote, however, stayed silent in exactly the case it existed for.

# First attempt — runs fine, misses the anomaly it was built to catch
ceiling = 100 * (1 - len(store) / turns)
if observed_hit > ceiling:
    alert("scopes may be merging")

Once written down the reason is plain. The ceiling was computed from the size of store, so when merging shrinks store, the ceiling drops right along with it. I had derived the monitoring threshold from the state being monitored. The broken system was quietly redrawing its own passing grade.

The fix is to source the ceiling from something the store cannot influence. In my case the set of distinct (app_id, user_id) pairs seen that day is countable independently, on the auth side.

def health(observed_hit_pct, turns, distinct_tenants):
    """Derive the ceiling from an independently counted tenant total,
    never from store state. A store-derived ceiling drops when merging
    occurs, which silences the alarm exactly when it should fire
    (verified by measurement)."""
    ceiling = 100 * (1 - distinct_tenants / turns)
    return {
        "ceiling": ceiling,
        "merged": observed_hit_pct > ceiling + 1e-9,       # boundary collapsed
        "fragmented": observed_hit_pct < ceiling - 5.0,    # scopes split apart
    }

Same data, and now both sides fire correctly:

The two-sided threshold is what makes this worth keeping. Too low means fragmentation; too high means collapse. Hit rate is not a metric where higher is always better. When it pushes past the ceiling, something is usually being trampled.

Linting Call Sites — Flag Inconsistency, Not Incorrectness

A runtime alarm fires after the damage. I wanted to catch it at authoring time, so I wrote an AST lint over scope dict literals. My first attempt was unusable.

Version one encoded "the correct form" as rules: if the value for user_id is a bare variable, str() is missing, and so on. Run against a small repository with 7 scope constructions across 4 files, it reported 12 findings. Five were real. The other seven flagged app_id receiving a bare variable — but app is already a string, so those were noise, not findings. Precision: 41.7%. A linter with that ratio gets muted within a week.

So I changed what the rule is about. Don't judge absolute correctness — detect inconsistency within the corpus. When the same scope key is written differently in different places, flag the minority. Drift, after all, is defined by being different from everything else.

#!/usr/bin/env python3
"""scope drift lint — flags places where the same scope key is written
differently from the rest of the corpus. It makes no claim about what is
"correct"; it only reports the minority spelling."""
import ast
import sys
import pathlib
from collections import defaultdict
 
ALLOWED = {"app_id", "user_id"}
SINKS = {"ingest", "retrieve_profile", "generate_memories"}
 
 
def shape(v):
    """Classify how a value is written: str()-wrapped, bare name, constant, ..."""
    if isinstance(v, ast.Call) and isinstance(v.func, ast.Name) and v.func.id == "str":
        return "str()"
    if isinstance(v, ast.Name):
        return "bare-name"
    if isinstance(v, ast.JoinedStr):
        return "f-string"
    if isinstance(v, ast.Constant):
        return f"const:{type(v.value).__name__}"
    return "other"
 
 
class Collect(ast.NodeVisitor):
    """Gather dict literals passed as scope=, following one level of assignment."""
 
    def __init__(self):
        self.sites = []
        self.assigned = {}
 
    def visit_Assign(self, n):
        if (isinstance(n.value, ast.Dict) and len(n.targets) == 1
                and isinstance(n.targets[0], ast.Name)):
            self.assigned[n.targets[0].id] = n.value
        self.generic_visit(n)
 
    def visit_Call(self, n):
        fname = n.func.attr if isinstance(n.func, ast.Attribute) else getattr(n.func, "id", None)
        if fname in SINKS:
            for kw in n.keywords:
                if kw.arg != "scope":
                    continue
                d = kw.value if isinstance(kw.value, ast.Dict) \
                    else self.assigned.get(getattr(kw.value, "id", None))
                if isinstance(d, ast.Dict):
                    self.sites.append(d)
        self.generic_visit(n)
 
 
def scan(root):
    sites = []
    for p in sorted(pathlib.Path(root).rglob("*.py")):
        try:
            tree = ast.parse(p.read_text(encoding="utf-8"))
        except SyntaxError:
            continue
        c = Collect()
        c.visit(tree)
        sites += [(str(p), d) for d in c.sites]
 
    # Pass 1: count spellings per key and pick the majority
    shapes = defaultdict(lambda: defaultdict(int))
    for _, d in sites:
        for k, v in zip(d.keys, d.values):
            if isinstance(k, ast.Constant) and k.value in ALLOWED:
                shapes[k.value][shape(v)] += 1
    majority = {k: max(s.items(), key=lambda x: x[1])[0] for k, s in shapes.items()}
 
    # Pass 2: report minority spellings, disallowed keys, and missing keys
    out = []
    for path, d in sites:
        keys = []
        for k, v in zip(d.keys, d.values):
            if not (isinstance(k, ast.Constant) and isinstance(k.value, str)):
                out.append((path, d.lineno, "dynamic-key", "key is not a string literal"))
                continue
            keys.append(k.value)
            if k.value not in ALLOWED:
                out.append((path, d.lineno, "extra-key",
                            f"disallowed key {k.value!r} — this fragments the profile"))
                continue
            sh = shape(v)
            if sh != majority[k.value]:
                out.append((path, d.lineno, "shape-drift",
                            f"{k.value} written as {sh} (corpus majority is {majority[k.value]})"))
            if sh.startswith("const:") and sh != "const:str":
                out.append((path, d.lineno, "type-drift", f"{k.value} is not a string"))
        for m in sorted(ALLOWED - set(keys)):
            out.append((path, d.lineno, "missing-key", f"required key {m!r} is absent"))
    return out
 
 
if __name__ == "__main__":
    rows = scan(sys.argv[1] if len(sys.argv) > 1 else ".")
    for f, l, kind, msg in rows:
        print(f"{f}:{l}: [{kind}] {msg}")
    print(f"-- {len(rows)} finding(s)", file=sys.stderr)
    sys.exit(1 if rows else 0)   # fail CI

Against the same repository:

repo/a.py:4: [shape-drift] user_id written as bare-name (corpus majority is str())
repo/b.py:2: [extra-key] disallowed key 'device_id' — this fragments the profile
repo/c.py:2: [shape-drift] user_id written as f-string (corpus majority is str())
repo/c.py:3: [extra-key] disallowed key 'tier' — this fragments the profile
repo/c.py:4: [extra-key] disallowed key 'locale' — this fragments the profile
-- 5 finding(s)

Relaxing the rule eliminated the false positives because the axis of judgment changed. "Add str()" is a claim about the world outside the file; "match the others" is verifiable entirely within the corpus. Detecting drift doesn't require a definition of correct.

One limitation worth stating: this lint follows dict literals and a single level of variable assignment, nothing more. Scopes assembled inside helper functions are invisible to it. That's partly laziness and partly a deliberate choice — as the next section argues, scope construction should converge on one place, and in a codebase that has done that, this lint goes quiet. Going quiet is the correct ending.

Put the Boundary at the Write Path, Not in the Schema

Here's where I landed. Rather than distributing the allowlist across call sites, define exactly one constructor for scopes and close every other route.

from dataclasses import dataclass
import json
 
@dataclass(frozen=True)
class Tenant:
    """Values eligible to become a scope, carried as a type.
    Never accept a raw string dict from a caller."""
    app_id: str
    user_id: str
 
    def __post_init__(self):
        for name in ("app_id", "user_id"):
            v = getattr(self, name)
            if not isinstance(v, str) or v != v.strip() or not v:
                raise ValueError(f"{name} must be a non-empty string with no surrounding whitespace: {v!r}")
 
 
def scope_of(t: Tenant) -> dict:
    """The only place a scope dict is constructed."""
    return {"app_id": t.app_id, "user_id": t.user_id}
 
 
def scope_key(t: Tenant) -> str:
    """Identity of the storage address. Logs and monitors use this same function."""
    return json.dumps(scope_of(t), sort_keys=True, separators=(",", ":"))

Because malformed values are rejected at Tenant construction, nothing downstream needs defensive code. A forgotten str() becomes a ValueError and dies in tests.

For deciding what belongs in the key, I settled on three questions:

1. If this value changes, do I want the memory carried over? If yes, keep it out of the scope
2. If this value matches, is sharing the memory acceptable? If no, put it in the scope
3. When unsure, leave it out. Adding a key later is easy; removing one orphans every existing profile

Applying that, device_id, locale, and tier all fall out. I want memory to survive a new phone. Switching languages doesn't make someone a different person. A plan change is content to be updated inside the profile, not a reason to relocate it. Meanwhile app_id stays, because the same number denotes different people across apps.

I had been treating the third question as a minor caveat. Adding a key "to get more granular later" is in practice an operation that discards the past. It behaves like a schema change without a migration, and because it raises no error, it breaks far more quietly than a schema change would.

Deciding what updates the contents of a profile is a separate axis of judgment. I covered per-field update policies and how quickly values go stale in Choosing a Memory Profiles Update Policy by Measurement. Scope decides the address; the update policy decides the freshness of what's at that address. For the broader question of isolating many apps and tenants on shared infrastructure, Gemini API Multi-Tenant SaaS Architecture covers the surrounding design.

Before Merging the 1,915, I Ran It Dry

The previous section settled the read path: don't patch it there. That still leaves the 1,915 profiles already sitting in the store. Folding the fragments into one record and throwing them away to rebuild are very different amounts of work, and I didn't want to start down one road and turn back halfway. So I ran a pass that writes nothing at all.

The old keys are still there as strings, which means the address can be recovered from them.

def recover(key_str):
    """Recover (app_id, user_id) from an old key string.
    A key missing either one has no determinable address, so return None."""
    d = json.loads(key_str)
    if "app_id" not in d or "user_id" not in d:
        return None
    return (str(d["app_id"]).strip(), str(d["user_id"]).strip())
 
 
def dry_run(store, order):
    """Report whether a merge is possible, without writing anything.
    order is the keys in creation order (used for oldest/newest)."""
    groups = defaultdict(list)
    unresolvable = []
    for k in order:
        t = recover(k[1])
        if t is None:
            unresolvable.append(k)
        else:
            groups[t].append(store[k])
    conflict = sum(1 for v in groups.values()
                   if len({x["billing_plan"] for x in v}) > 1)
    return {
        "profiles": len(store),
        "tenants": len(groups),
        "unresolvable": len(unresolvable),
        "fragmented": sum(1 for v in groups.values() if len(v) >= 2),
        "conflict": conflict,
        "auto_mergeable": len(groups) - conflict,
    }

Run against the store built with naive keys:

That stopped me again. Only 49 of the 400 tenants can be merged mechanically. For the remaining 351, someone has to decide which fragment survives — and that someone is me. Each fragment was written on a different day by a different call site, carrying whatever subscription state was true at the time.

I lined up three survivor rules and scored them against the last state observed in the traffic:

Even "keep the newest," which looked like the obvious choice, gets one in three wrong. And the only thing that knows the truth here is the harness. In production there is nothing to check the answers against — once the merge has run, I have no way to find out which tenants I got wrong.

The merge is not a reversible operation.

So I changed course: fold nothing, and rebuild from whatever holds authority. For subscription state that's the receipt validation result; for preferences it's what the person typed in. The profile is derived from those, not the record of them — an obvious thing I only really absorbed after lining up those agreement rates. What had made the fragments feel too valuable to drop was the size of the number, 1,915, rather than any confidence in what they contained.

One more variation worth measuring. All six drift variants here happened to include app_id, which is why recovery came out at 100%. Had 6% of the calls forgotten app_id, the store grows to 1,998 profiles and 155 of them (7.8%) have no determinable address. That number is why the missing-key rule in the lint above fails CI rather than warning. An extra key costs you fragmentation; a missing key costs you the possibility of recovery.

Counting the same way puts the cross-tenant reads in a different light too. Merging on user_id alone produces 200 groups, and every single one of them mixes more than one tenant — all 400 tenants are affected. Read as a count, 1,843 sounds like an incident confined to a subset. The blast radius was everyone.

Three Things to Check Tomorrow

To close, here's the short version — in the order I actually did it.

1. Divide today's ingestion call count by today's distinct tenant count. Close to 1.0 is healthy. Mine was 4.8. That single ratio is the earliest signal available, and it surfaces long before anyone thinks to look at a hit rate
2. Grep every scope dict literal and read them side by side. grep -rn 'scope=' --include='*.py' is enough. Two distinct spellings means you are already fragmented
3. Compute the hit-rate ceiling from an independent tenant count and compare. Below it means fragmentation; above it means a boundary has collapsed. Treat "above" as the more alarming direction
4. If you've already scattered, run the merge dry once before running it for real. Knowing how many tenants hold disagreeing fragments is enough to decide for yourself between folding and rebuilding

And if your profiles have already scattered, I'd encourage you to resist the read-side patch. In my measurements it came back as 1,843 cross-tenant reads over 4,000 turns. Guard the address at the write path; rebuild the contents from the source of record. It looks like the long way around, and it was still the fastest route back to a state I could explain.

Working solo as an indie developer, this class of silent failure never gets reported to you. From the user's side it's just "it doesn't seem to remember me" — not enough friction to file anything. Which is exactly why the alarm has to be something you place yourself, in the spot that stays quiet. Until I saw the numbers, I trusted my own implementation quite a lot.

One caveat on methodology: the measurements here come from a local harness reproducing the one-profile-per-(schema, scope) contract, not from live API calls. Fragmentation and merging are fully explained by how the key is derived, so I judged the model sufficient — but if you're making decisions that involve billing or latency, please measure against your own configuration. The harness runs as-is if you concatenate the code blocks above.

If this saves someone half a day of chasing the wrong problem, that would make me glad. Thank you for reading.