The Assistant Switch Is One-Way, So the Baseline Has to Be Captured Now

Keeping the baseline to five fields

The more fields you record, the less appetite you have for reading them later. Since this has to be live before September 4, I cut it to five.

Big numbers like DAU and retention are not the protagonists here. If voice-originated launches are a few percent of the total, halving them disappears into the noise on a DAU chart. This work is about holding a number at a granularity where it cannot hide.

I look at the AdMob side separately for the same reason. When one entry point thins out, total impressions get backfilled by the others, so the revenue chart is the last thing to move.

Clustering the utterances with Gemini to watch the distribution

Reading utterances one at a time tells you nothing. Whether "change my wallpaper", "open the wallpaper app" and "show me an ukiyo-e wallpaper" are one intent or three stops being a judgement you can make by hand once the volume grows.

So I classify weekly with Gemini and keep only the per-cluster counts as a time series. If the granularity of the classification drifts week to week, nothing is comparable, so the categories are fixed and passed in.

# weekly_utterance_profile.py — weekly cluster profile, counts only
import os
import json
from collections import Counter
from google import genai
from pydantic import BaseModel
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
# Fixed granularity. Changing this weekly breaks comparison with last week.
CATEGORIES = [
    "open_app",          # just open the app
    "change_wallpaper",  # set or change a wallpaper
    "search_theme",      # ask for a specific subject or theme
    "control_playback",  # play, pause and similar
    "other",
]
 
class Labeled(BaseModel):
    index: int
    category: str
 
class Batch(BaseModel):
    items: list[Labeled]
 
def classify(utterances: list[str]) -> Counter:
    numbered = "\n".join(f"{i}: {u}" for i, u in enumerate(utterances))
    prompt = (
        "Assign each utterance below to exactly one of the given categories.\n"
        f"Categories: {', '.join(CATEGORIES)}\n"
        "Use other when you cannot decide.\n\n"
        f"{numbered}"
    )
    # temperature / top_p / top_k are deprecated, so nothing is set here
    res = client.models.generate_content(
        model="gemini-3.7-flash",
        contents=prompt,
        config={
            "response_mime_type": "application/json",
            "response_schema": Batch,
        },
    )
    parsed = Batch.model_validate_json(res.text)
    counts = Counter()
    for item in parsed.items:
        # guard against a category name outside the fixed set
        counts[item.category if item.category in CATEGORIES else "other"] += 1
    return counts
 
def profile(week_id: str, utterances: list[str], batch_size: int = 80) -> dict:
    total = Counter()
    for i in range(0, len(utterances), batch_size):
        total += classify(utterances[i : i + batch_size])
    n = sum(total.values()) or 1
    return {
        "week": week_id,
        "n": n,
        "share": {c: round(total[c] / n, 4) for c in CATEGORIES},
    }
 
if __name__ == "__main__":
    with open("utterances_2026w34.json", encoding="utf-8") as f:
        rows = json.load(f)
    print(json.dumps(profile("2026-W34", rows), ensure_ascii=False, indent=2))

Leaving temperature unset is deliberate rather than forgetful. The sampling parameters are deprecated, and I would rather not add new explicit uses of them to code written today. I wrote separately about what actually broke first when they went deprecated in the side of temperature deprecation that hurt first.

The output is stored as shares rather than counts. The denominator moves week to week, and raw counts blend a change in usage with a change in distribution.

Switching exactly one device early

There were three ways to handle my own devices.

I chose the middle one. The shorter your hands-on comparison window, the harder it becomes to tell whether a later report of "I can't open it by voice anymore" belongs to the platform switch or to something I changed in my own app.

In this situation I would recommend making that early device a secondary one. Switch your daily driver first and your own changing habits get mixed into the signal. I moved over a device I only ever use for testing.

The part I got wrong was interpretation, not recording

When I started, I assumed that recording was the hard part and that I could simply draw a line at September 4 and compare either side. That assumption was wrong.

The switch does not happen all at once on September 4. It is progressive and expected to take several weeks. Which means the same day's event log contains both switched and unswitched devices. Split before and after by date and the "after" bucket is full of "before", so any real difference looks diluted. Whether the difference is small or merely mixed is not something a date can tell you.

There is also no way for the app to ask a device which assistant it is running. Since the device will not tell you, splitting by a date imposed from outside cannot be precise, in principle.

So I moved the estimate of the switch from an outside date to an inside signal. Whether the distribution of phrasing has moved should track reality more closely than the calendar does. If only the other share rises during the migration window, that points at a change in phrasing rather than a drop in usage.

The real reason the Gemini clustering step exists is not that I wanted the analysis to look sophisticated. It is that the date could not be used as the split. Had I discovered that in the other order, I would probably have settled for staring at raw counts.

The question of what to start counting before a shutdown came up the same way during the image model retirement. That inventory procedure is written up in the pre-shutdown usage audit.

What to finish before September 4

With the days left, the ordering is fairly clear.

1. Ship the build carrying launch_origin early enough to absorb store review. Review plus a staged rollout eats several days
2. Verify the events actually arrive, by launching your own device both by voice and from the launcher
3. Settle the consent handling and the privacy policy wording for utterance collection first
4. Move exactly one test device to Gemini manually and look at the post-switch behaviour yourself
5. Capture the weekly cluster distribution at least once before the switch. A baseline of one point still beats a baseline of zero

Step 3 is the one worth pulling forward. Ship before the collection design is settled and you will need a second release just to change how consent is handled.

The next move

There is honestly not much you can build ahead of a switch like this. Counting something in a form you can re-read later pays off more than trying to predict the behaviour and code against it.

Start with one line on one chart — the daily count of launches that did not come from your launcher. Whether that line exists changes how you will read every report that arrives in September.

For me, the useful part of this exercise was noticing that I had never measured it at all. If you have the same gap, I hope this helps.