Measuring Update Policies for Memory Profiles: The Guard That Cost Me 16 Points of Accuracy

Three policies, one set of conditions

Here is the actual experiment. Writes into the profile were governed by three policies:

1. naive — last write always wins
2. guard — never overwrite a confirmed value with an inferred one
3. guard + TTL — same guard, except a confirmed value older than N turns may be overwritten by an inferred one

Scoring is simple: at the end of the run, what fraction of fields matches the true current value? Ground truth mutates over time with a fixed per-turn probability. 25% of utterances carry confirmed values; the rest are inferred, and inferred values are wrong 20% of the time.

import random
 
VALUES = list("abcd")
CONFIRM_P = 0.25   # probability an utterance is an explicit confirmed value
NOISE     = 0.20   # probability an inferred value misses the truth
 
def run(policy, ttl, change_p, seed, n_keys=20, turns=2000):
    r = random.Random(seed)
    keys  = [f"f{i:02d}" for i in range(n_keys)]
    truth = {k: r.choice(VALUES) for k in keys}
    # store[k] = [value, confirmed_flag, last_updated_turn]
    store = {k: [truth[k], 1, 0] for k in keys}
 
    for t in range(1, turns + 1):
        # the world moves on its own
        for k in keys:
            if r.random() < change_p:
                cur = truth[k]
                truth[k] = r.choice([v for v in VALUES if v != cur])
 
        k = r.choice(keys)
        if r.random() < CONFIRM_P:
            v, confirmed = truth[k], 1
        else:
            v = truth[k] if r.random() > NOISE else \
                r.choice([x for x in VALUES if x != truth[k]])
            confirmed = 0
 
        cur = store[k]
        if policy == "naive":
            store[k] = [v, confirmed, t]
        elif policy == "guard":
            if not (cur[1] == 1 and confirmed == 0):
                store[k] = [v, confirmed, t]
        elif policy == "guard_ttl":
            stale = (t - cur[2]) > ttl
            if stale or not (cur[1] == 1 and confirmed == 0):
                store[k] = [v, confirmed, t]
 
    hit = sum(1 for k in keys if store[k][0] == truth[k])
    return hit / len(keys)
 
def avg(policy, ttl, change_p, trials=40):
    return sum(run(policy, ttl, change_p, s) for s in range(trials)) / trials * 100
 
for change_p, label in ((0.001, "slow"), (0.004, "medium"), (0.016, "fast")):
    print(f"== change_p={change_p} ({label}) mean fact lifetime ~{1/change_p:.0f} turns ==")
    print(f"   naive {avg('naive', None, change_p):.1f}%  "
          f"guard {avg('guard', None, change_p):.1f}%", end="")
    for ttl in (50, 100, 200, 400):
        print(f"  TTL{ttl}:{avg('guard_ttl', ttl, change_p):.1f}%", end="")
    print()

change_p is the per-turn probability that a field's underlying fact changes. At 0.001 the mean lifetime is about 1,000 turns; at 0.016 it is about 62. The former models slow attributes like primary language or target platform, the latter volatile ones like billing state or whatever the user is working on right now.

One honest caveat: this harness never calls the Gemini API. What it measures is the behavior of the merge layer I wrote, not the model's recall quality. It is still a far better basis for choosing a policy than intuition was.

The result: the protective move missed the most

The bottom row is where I sat back.

On fast-changing fields, the guard trails naive by 16.4 points — 68.9% down to 52.5%. Refusing to overwrite confirmed values, which reads as unambiguously correct, made accuracy measurably worse.

The mechanism is clear in hindsight. Once a confirmed value lands, it is frozen until the user states something explicitly again. With only 25% of utterances confirmed, that opportunity arrives every few dozen turns on average — while the underlying fact turns over every 62. The value you are protecting expires while you are protecting it.

That billing_plan: pro from the support log was exactly this. The record was correct, the guard was working, and the answer was still wrong.

TTL behavior was counterintuitive too. On fast fields, shorter is better: stretch the TTL to 400 and you land at 53.0%, barely distinguishable from the bare guard's 52.5%. Slow fields invert this — TTL 50 gives 90.4%, worse than the bare guard's 93.0%, because you discard good confirmed values too eagerly.

There is no safe default TTL. It has to track the lifetime of the fact the field holds.

Giving each field its own lifetime

Real profiles mix lifetimes in a single object. I rebuilt the test with 20 fields — 8 slow, 6 medium, 6 fast — and compared a single global TTL against per-field TTLs set to roughly a quarter of each field's mean lifetime (200 / 50 / 15 turns).

Mixing lifetimes erases the guard's advantage entirely. Gains on slow fields and losses on fast ones cancel out, landing on the same 76.0% as naive. As long as one policy governs the whole profile, the average will not improve.

Per-field TTLs reach 81.8%, beating the best global setting by another 2.3 points. So I moved lifetime declarations into the schema itself.

FIELD_POLICY = {
    # slow attributes: trust an explicit statement for a long time
    "primary_language": {"ttl_turns": 200, "guard": True},
    "target_platform":  {"ttl_turns": 200, "guard": True},
    "min_os_version":   {"ttl_turns": 200, "guard": True},
    # medium: revisit every few dozen turns
    "build_tool":       {"ttl_turns": 50,  "guard": True},
    "ci_provider":      {"ttl_turns": 50,  "guard": True},
    # volatile: do not over-protect
    "billing_plan":     {"ttl_turns": 15,  "guard": True},
    "current_task":     {"ttl_turns": 15,  "guard": True},
    "notify_channel":   {"ttl_turns": 15,  "guard": True},
}
DEFAULT_POLICY = {"ttl_turns": 50, "guard": True}
 
def accept_update(field, current, incoming, now_turn):
    """Write to the profile only when this returns True."""
    pol = FIELD_POLICY.get(field, DEFAULT_POLICY)
    if current is None:
        return True
    if not pol["guard"]:
        return True
    if incoming["confidence"] == "confirmed":
        return True                      # explicit statements always pass
    if current["confidence"] == "inferred":
        return True                      # inferred over inferred: newest wins
    # overwrite a confirmed value only once it has expired
    return (now_turn - current["updated_turn"]) > pol["ttl_turns"]

One trap worth flagging. It is tempting to express ttl_turns in wall-clock time — 30 days, say — but conversation frequency varies wildly per user, and wall-clock TTLs produce an inversion where your heaviest users hold the stalest values the longest. Turn counts matched observed behavior much better. In production I now carry both and expire on whichever threshold trips first.

What belongs in the schema, and what does not

Measuring changed my criteria for what to structure at all. Three rules I now apply:

1. Only values that downstream code reads mechanically

If code branches on it — billing state, locale, notification channel — it goes in the schema. Softer signals like "this person cares a lot about visual polish" stayed in free-form memory. If structuring a value does not enable a branch, all you have added is the cost of deciding.

2. Prefer closed value ranges

Anything expressible as an enum structures cleanly; free-text fields invite bad confirmed values. During development a device field once stored the literal string "probably some iPhone" as a confirmed value, and a downstream branch broke quietly for days. Fields that cannot be enumerated now get abstracted until they can be.

3. Allow blanks

This mattered most. Every field you mark required is a field the model dislikes leaving empty, so it fills it by inference — and downstream code cannot distinguish that from a confirmed value. I trimmed required to fields without which processing genuinely cannot proceed. The point of structuring memory is not to eliminate blanks, but to make a blank representable as a blank.

What you can change tomorrow

List the fields in your live profile and write next to each one how many days that fact typically survives. Some fields will resist a number. Those are your riskiest ones right now.

The harness above is about 50 lines. Drop in change_p values that match your own fields and run it — your numbers will differ from my 16.4 points, and that difference is the actual decision input.

Memory profiles gave us a place to keep memory. Deciding what to forget, and when, is still ours to do.

Availability and behavior of Memory profiles will keep shifting, so check the Gemini API changelog before you build against it.

Thank you for staying with a fairly long measurement write-up. I am still finding my footing here, so if your workload produces different results, I would love to hear about it.