At 1% rollout on day one, my dashboard showed two crashes. That looked no worse than the previous build, so I approved the bump to the next day's percentage without much thought.
By day three, at 5%, reviews describing the same symptom started stacking up. Those two crashes on day one had never meant "this build is fine." They were simply too few to mean anything at all.
Shipping several apps in parallel means each rollout day gets a few minutes of my attention, no more. To keep those few minutes from producing bad calls, I redrew the line between what a deterministic calculation decides, what a language model decides, and what stays on my desk. The short version: the stop-or-go call itself never goes to the model.
A quiet day one is not evidence of quiet
Phased releases on the App Store ramp over seven days: 1%, 2%, 5%, 10%, 20%, 50%, then 100%. The part that is easy to miss is that day one carries one two-hundredth of the sample the final day does.
Take an app seeing 20,000 new sessions per day at full rollout. Holding the true crash rate fixed at 0.8%, here is what you observe each day and what that observation actually licenses you to say (Wilson 95% interval).
| Day | Rollout | Sessions | Observed crashes | 95% interval | Interval width |
|---|---|---|---|---|---|
| Day 1 | 1% | 200 | 2 | 0.27% – 3.57% | 3.30pt |
| Day 2 | 2% | 400 | 3 | 0.26% – 2.18% | 1.93pt |
| Day 3 | 5% | 1,000 | 8 | 0.41% – 1.57% | 1.16pt |
| Day 4 | 10% | 2,000 | 16 | 0.49% – 1.30% | 0.80pt |
| Day 5 | 20% | 4,000 | 32 | 0.57% – 1.13% | 0.56pt |
| Day 6 | 50% | 10,000 | 80 | 0.64% – 0.99% | 0.35pt |
| Day 7 | 100% | 20,000 | 160 | 0.69% – 0.93% | 0.25pt |
Look at the first row. The point estimate reads 1.00%, but the truth could sit anywhere from 0.27% to 3.57%. A build running at four times the normal crash rate produces a day-one number that looks exactly the same.
That interval is precisely what I missed. I had been reading "small number" as "safe."
Replace go/stop with three states
The real cause of the bad call was the shape of the choice. Give someone two options and they will pick one, even when the honest answer is that the data cannot decide yet. So I added a third state.
import math
def wilson(k, n, z=1.96):
"""Range the true rate could plausibly occupy, given k events out of n."""
if n == 0:
return (0.0, 1.0)
p = k / n
d = 1 + z * z / n
center = (p + z * z / (2 * n)) / d
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return (max(0.0, center - half), min(1.0, center + half))
def verdict(n, k, baseline, ceiling):
"""baseline = normal rate, ceiling = the rate at which we halt."""
lo, hi = wilson(k, n)
if lo > ceiling:
return "STOP" # over the ceiling even after discounting the small sample
if hi <= ceiling:
return "GO" # provably not over the ceiling
return "UNDECIDED" # neither claim is supported yet
def required_users(baseline, ceiling, z=1.96, step=100):
"""Sample size at which a normal-looking rate finally earns a GO."""
n = step
while n <= 2_000_000:
if wilson(round(n * baseline), n, z)[1] <= ceiling:
return n
n += step
return NoneWith a 0.8% baseline and a ceiling at double that, 1.6%, the function produces this:
| Sessions | Crashes | Observed | Interval | Verdict |
|---|---|---|---|---|
| 200 | 2 | 1.00% | 0.27% – 3.57% | UNDECIDED |
| 200 | 8 | 4.00% | 2.04% – 7.69% | STOP |
| 1,000 | 8 | 0.80% | 0.41% – 1.57% | GO |
| 1,000 | 30 | 3.00% | 2.11% – 4.25% | STOP |
| 4,000 | 90 | 2.25% | 1.83% – 2.76% | STOP |
required_users(0.008, 0.016) returns 900. Until 900 sessions have accumulated, a perfectly normal-looking number gives you no grounds to say the build is healthy. Day one gave me 200, less than a quarter of that.
Notice the asymmetry, though: STOP still fires on small samples. Eight crashes in 200 sessions clears the ceiling even after the uncertainty is priced in. Breakage shows up early; health shows up late. I have made peace with that trade rather than trying to engineer it away.
Counting reviews is not enough
Crash rates are a deterministic problem. Reviews are not. "Can't save photos since the update" and "too many ads" both land in the same one-star bucket. One of them is about the build now rolling out; the other has been true for months.
Deciding which is which is exactly the kind of judgment a language model is good at. I pin the output shape with responseSchema and classify one review at a time.
from google import genai
from google.genai import types
client = genai.Client(api_key="YOUR_API_KEY")
SCHEMA = {
"type": "object",
"properties": {
"attributable_to_update": {"type": "boolean"},
"symptom": {"type": "string"},
"severity": {"type": "string", "enum": ["blocking", "major", "minor", "none"]},
},
"required": ["attributable_to_update", "symptom", "severity"],
}
PROMPT = """Classify the app review below.
Decide only one thing: does it describe a problem that started after this update?
Do not judge the quality of the app, and do not judge whether the rollout should continue.
Complaints that predate the update, such as pricing or ad volume, get attributable_to_update = false.
Review:
{body}"""
def classify(body: str) -> dict:
res = client.models.generate_content(
model="gemini-3.7-flash",
contents=PROMPT.format(body=body),
config=types.GenerateContentConfig(
response_mime_type="application/json",
response_schema=SCHEMA,
),
)
return res.parsedLines two and three of that prompt were carved out rather than added. My first version also asked whether the review warranted halting the rollout. The answers read well, and on days with the smallest samples they leaned reliably toward "it seems reasonable to keep watching."
A measured answer with a paragraph of reasoning behind it is hard to argue with, so I approved it. The moment I asked the model to decide, my three states collapsed back into two. I stopped asking.
If you are generating the review replies themselves, the latency trap I ran into is written up separately in Automating App Store and Google Play Review Replies with Gemini API — The 8-Second Rule I Discovered the Hard Way.
Keep model output out of the numerator
Once classifications are in hand, folding them into the crash-rate math is tempting. I tried it. The result was a gate that wobbled every time the model missed a single review.
The two signals now stay separate and meet only at the end, where the merge rule is deliberately dull: if either side is red, halt; if they disagree, escalate to a human.
def route(judgements, crash_state):
"""judgements: model classifications, crash_state: the deterministic verdict."""
regressions = [
j for j in judgements
if j["attributable_to_update"] and j["severity"] in ("blocking", "major")
]
if crash_state == "STOP":
return ("STOP", "crash-rate lower bound cleared the ceiling")
if len(regressions) >= 3:
return ("HUMAN_REVIEW", f"{len(regressions)} serious update-caused reviews")
if crash_state == "UNDECIDED":
return ("HOLD", "sample too small; hold the current percentage another day")
return ("GO", "within threshold; advance to the next percentage")Making HUMAN_REVIEW a first-class return value did more for this pipeline than anything else. When the model flags three reviews, the system neither halts nor advances. It hands me three pieces of text to read, which fits comfortably into a morning.
HOLD and GO are also deliberately distinct. HOLD means "stay at today's percentage for one more day," which is an action, not an absence of one. Choosing to wait and failing to decide are different things.
Four numbers I fix before shipping
Thresholds adjusted mid-rollout get adjusted in the convenient direction. I set these before release and leave them alone until it ends.
| Setting | My value | Why |
|---|---|---|
| baseline | Median of the last 4 weeks | A mean drags in whatever bad build came before |
| ceiling | 2x baseline | At 1.5x, UNDECIDED never resolved before the final day |
| Minimum sample | Whatever required_users returns | A GO below that number is a GO without grounds |
| Model output | Escalation only | In the numerator, a missed classification moves the verdict |
That ceiling deserves a note. When I tried 1.5x, required_users climbed to 2,800, which meant a completely healthy build stayed UNDECIDED past day four. Tightening the threshold did not make the rollout safer; it made the whole week uninformative. Strictness and time-to-decision trade against each other, and the trade is worth pricing explicitly.
Before your next rollout, run required_users against your own baseline. Divide that number by your daily session count at each percentage and you will know which day your dashboard starts saying anything at all. For me it turned out to be day three, which means everything I stared at before then was reassurance rather than evidence.
Checking generated strings before they ship — release notes, push copy, store text — is a neighbouring problem I wrote about in Why My Length Limit Only Failed on the English Notifications, and the Width-Based Fit That Replaced It. Read together, the release-day sequence hangs together a little better.
Misreading those numbers is a mistake I made very recently, not a hypothetical. If it saves one person the same three days, it was worth writing down.