Letting Gemini Flash Decide continue / pause / rollback for Staged Rollouts: An Indie Developer's Three-Signal Engine

The decision schema, after two months of pruning

The schema started at around 200 lines and shrank as I cut things I never actually used.

from pydantic import BaseModel
from typing import Literal
 
class StagedRolloutDecision(BaseModel):
    verdict: Literal["continue", "pause", "rollback"]
    confidence: Literal["high", "medium", "low"]
    primary_signal: Literal["crash", "review", "revenue", "composite"]
    reason: str  # 1-2 sentences
    next_action: str  # e.g. "Re-evaluate at 12:00 on 2026-05-28"
    flagged_devices: list[str]
    flagged_review_keywords: list[str]

Three verdicts only. I tried adding a fourth "watch more closely" value at one point and immediately noticed I started using it as a way to procrastinate harder decisions. Killing it forced me to either pause or commit, which is the whole point of having a rollout policy in the first place.

Confidence is a three-value enum, not a float. Gemini Flash is not well-calibrated on continuous probability, and a three-bucket enum matched my own intuition much better than a number that pretends to be precise.

The primary_signal field is metadata for me. It lets me look back across a month and ask "what fraction of pauses were driven by crashes versus reviews versus revenue?" — which is useful for tuning the prompt over time.

Building the prompt

The prompt is four blocks: role, raw three-signal snapshot, decision policy, output schema. The whole input ends up around 1,200 tokens. The structured output uses 300–500 tokens.

import google.generativeai as genai
from pydantic import BaseModel
 
SYSTEM_PROMPT = """\
You are an SRE responsible for staged rollouts of indie iOS / Android apps.
For each app, 24–72 hours after release, look at three signals and decide whether
to continue, pause, or rollback the rollout:
 
Policy:
- Crash-free rate down 0.5% vs baseline -> pause; down 1.0% -> rollback.
- If the regression is concentrated on a small set of devices/OS combos,
  list them in flagged_devices but do not over-trigger on the overall number.
- If new-version reviews mention "won't open / crash / consent loop / can't
  purchase" above 10% of all new-version reviews -> pause.
- If AdMob ARPU is below 70% of the previous version while DAU is above 90%
  of the previous version -> pause (likely a monetization regression).
- Even if no single threshold is hit, if multiple signals concentrate on the
  same device family, treat as composite -> pause.
- If unsure, return confidence=low and write "human review recommended" in reason.
"""
 
def build_user_prompt(app_id: str, snapshot: dict) -> str:
    return f"""\
App: {app_id}
New version: {snapshot['new_version']} (released {snapshot['hours_since_release']}h ago)
Rollout percent: {snapshot['rollout_percent']}%
 
[Crashlytics]
- Crash-free rate (new): {snapshot['crash_free_new']}%
- Crash-free rate (baseline): {snapshot['crash_free_prev']}%
- New issues: {snapshot['new_issues']}
- Crash distribution by device: {snapshot['crash_by_device']}
 
[Reviews]
- Reviews on new version: {snapshot['review_count']}
- 1-2 star ratio: {snapshot['low_rating_ratio']}%
- Negative keywords detected: {snapshot['negative_keywords']}
- Sample reviews (5 excerpts): {snapshot['review_excerpts']}
 
[AdMob]
- 24h ARPU (new): {snapshot['arpu_new']}
- 24h ARPU (baseline): {snapshot['arpu_prev']}
- DAU (new): {snapshot['dau_new']} ({snapshot['dau_ratio']}% of baseline)
- Fill rate (new): {snapshot['fill_rate_new']}%
 
Return verdict / confidence / primary_signal / reason / next_action /
flagged_devices / flagged_review_keywords as JSON conforming to the schema.
"""

Calling Gemini Flash 3.0 with structured output is the now-standard pattern:

def evaluate(app_id: str, snapshot: dict) -> StagedRolloutDecision:
    model = genai.GenerativeModel(
        "gemini-3.0-flash",
        system_instruction=SYSTEM_PROMPT,
        generation_config={
            "response_mime_type": "application/json",
            "response_schema": StagedRolloutDecision,
            "temperature": 0.2,
        },
    )
    response = model.generate_content(build_user_prompt(app_id, snapshot))
    return StagedRolloutDecision.model_validate_json(response.text)

I run at temperature=0.2. Lower temperatures occasionally returned empty responses or finish_reason=RECITATION on this kind of structured task, so 0.2 became my stable floor.

Three potholes worth knowing about

1. "Previous version" needs to be a manual baseline

My first version computed "previous version" as whatever release came before the current one. That broke during hotfix streaks: each hotfix lowered the baseline a little, and after three or four hotfixes the engine stopped catching real regressions because the baseline had quietly drifted.

The fix was to store a stable_baseline field in Firestore, manually advanced only when a rollout reaches 100% and stays healthy for seven days.

def get_baseline_version(app_id: str) -> str:
    doc = firestore.collection("apps").document(app_id).get()
    return doc.to_dict()["stable_baseline"]

This single change cut bogus rollback flags by roughly 40% in my logs.

2. Version strings drift, especially on Google Play

The App Store Connect review API takes a clean version filter. Google Play Console exports CSV with version strings that wander — "2.4.1", "2.4.1 (build 89)", and even "2.4.1 (beta)" mixed in. The beta variant was the worst because it pulled review feedback for testers into the production decision.

import re
 
def normalize_version(s: str) -> str:
    m = re.match(r"^v?(\d+\.\d+\.\d+)", s)
    return m.group(1) if m else s

Normalize everything to MAJOR.MINOR.PATCH before grouping. This is the kind of edge case you would never anticipate until you ship the first version of the pipeline and watch it misfire.

3. AdMob numbers are unreliable for the first ~8 hours

Right after release the population on the new version is small, and AdMob aggregation has a 2–4 hour reporting lag. Early in development I was getting pause verdicts in the first few hours that were almost always false alarms.

The fix was telling the model to ignore AdMob signals when hours_since_release < 8. I added a single line to the system prompt: "If less than 8 hours since release, ignore AdMob signals and decide on Crashlytics and reviews only."

After that change, early-stage false pauses essentially went to zero in my logs.

Running it across six apps

The trigger is a Cloud Scheduler firing every four hours into a Cloud Run Job, with Pub/Sub fan-out so each app's evaluation runs in parallel.

Per-evaluation token usage runs roughly 1,200–1,800 input plus 300–500 output. On Gemini Flash 3.0 pricing that comes out to about 0.3–0.6 yen per call. Six apps × six calls per day × 30 days = 1,080 calls per month. Monthly spend has landed in the 320–640 yen range. For context, this is less than the AdMob revenue from one healthy app on one healthy day.

import os
from firebase_admin import firestore
 
def main():
    app_id = os.environ["APP_ID"]
    snapshot = collect_snapshot(app_id)
    decision = evaluate(app_id, snapshot)
    firestore.collection("rollout_decisions").add({
        "app_id": app_id,
        "ts": firestore.SERVER_TIMESTAMP,
        "snapshot": snapshot,
        "decision": decision.model_dump(),
    })
    if decision.verdict == "rollback":
        send_urgent_notification(app_id, decision)
    elif decision.verdict == "pause":
        send_review_request(app_id, decision)

Only rollback verdicts page me; pause verdicts queue for my morning review. I was paged three times in two months, and all three were correct calls.

What two months of production looked like

Numbers from six apps over 60 days (12 app-months total):

• Total evaluations: 2,160 (6 calls/day × 60 days × 6 apps)
• continue: 1,924 (89.1%)
• pause: 218 (10.1%)
• rollback: 18 (0.8%)
• Sent to manual review (confidence=low): 73 (3.4%)

Agreement between the model's pause/rollback decisions and my final human call was 91%. The remaining 9% split roughly evenly: half I downgraded back to continue ("let me watch one more cycle"), half I escalated to rollback faster than the model wanted.

The number that matters most: zero rollouts shipped customer-visible damage during the test period. The reason is that the engine never executes the rollout call — it only recommends. I still tap the button in App Store Connect or Google Play Console myself.

Where the automation boundary stops

I deliberately stopped short of automating the rollout API itself. App Store Connect and Google Play Console each have their own rate limits, token expiries, and per-locale rollout quirks. Automating the recommendation paid off immediately. Automating the action would mean owning that whole surface, and the debugging cost would dwarf the time savings.

Decision is AI, action is human. For a one-person operation it is the line that scales best. I think about what I am leaving behind for my kids when I work like this — and what holds up best across years is not the cleverest automation, but the one whose decision history I can still read clearly years later.

Three extensions I want to try next

Three follow-ups I haven't shipped yet:

1. Send 30 days of past decisions as long context — let Gemini Flash spot "we've seen this regression pattern before" across history. This is the long-context use case I'd most like to validate next.
2. Replace regex keyword detection with Gemini Embedding — "won't launch" and "won't open" and "won't start" should collapse into one semantic cluster instead of three regex hits.
3. Daily Batch Prediction summary — run a 0:00 Vertex AI batch over the day's decisions and emit a single long-form report. The cost math gets even friendlier at batch pricing.

I'll write those up separately as I get the numbers in.

Hope the implementation is useful. If you're running staged rollouts on your own apps, I'd love to hear your decision rules — feel free to send a listener letter on my stand.fm.