Spend Deep Reasoning Only Where It's Needed: Per-Request thinking_level Routing in Gemini

Implementing the router

Here is a minimal router in Python that unites the heuristic and the model judgment. The trick is to keep the part that maps a difficulty decision to a tier separate from the part that injects thinking_level into the actual request, so you can tune thresholds later.

# router.py — pick thinking_level from a request's difficulty
from google import genai
from google.genai import types
 
client = genai.Client()
 
# minimal hints that suggest multi-step reasoning (to reduce misses)
_HARD_HINTS = ("compare", "which", "why", "reason", "steps", "design",
               "trade-off", "vs", "recommend", "choose")
 
def heuristic_tier(text: str, sensitive: bool) -> str | None:
    """Zero-cost first pass. Returns None to defer to the model."""
    if sensitive:
        return "medium"                     # raise the floor where misses hurt
    n = len(text)
    hard = any(h in text.lower() for h in _HARD_HINTS)
    if n <= 40 and not hard:
        return "low"                        # short and simple -> shallow, fast
    if hard or n >= 200:
        return "high"                       # multi-step or long -> deep
    return None                             # gray zone -> defer to model
 
def model_tier(text: str) -> str:
    """Let Flash judge difficulty itself, but only for the gray zone."""
    r = client.models.generate_content(
        model="gemini-flash-latest",
        contents=f"Answer with one word (low/medium/high): how much "
                 f"reasoning depth does this question need? Question: {text}",
        config=types.GenerateContentConfig(
            thinking_config=types.ThinkingConfig(thinking_level="low"),
            max_output_tokens=4,
        ),
    )
    out = (r.text or "").strip().lower()
    return out if out in ("low", "medium", "high") else "medium"
 
def route(text: str, sensitive: bool = False) -> str:
    return heuristic_tier(text, sensitive) or model_tier(text)
 
def answer(text: str, sensitive: bool = False) -> str:
    level = route(text, sensitive)
    r = client.models.generate_content(
        model="gemini-flash-latest",
        contents=text,
        config=types.GenerateContentConfig(
            thinking_config=types.ThinkingConfig(thinking_level=level),
        ),
    )
    return r.text

model_tier runs only when heuristic_tier returns None. Because that pre-pass itself runs at thinking_level="low" with max_output_tokens=4, its cost stays small next to a real answer. Constraining the judgment to a single word matters: if you let the gray-zone estimate deliberate, you undo the very latency you set out to cut.

Backing the ceiling out of a mobile wait-time budget

Once routing works, you set the ceiling on how deep a hard question may go by working backward from perceived speed. Skip this and a rare hard question pushes up p95 until the whole feature reads as "the app that sometimes freezes."

For interactive features I treat a p95 of five seconds to completion as a working target. Against the table above, high's p95 of ~9.1s blows the budget. So I capped ordinary hard questions at medium (p95 ~5.2s) and opened high by hand only to requests that are both high-stakes and rare. Put plainly: allow each level only in the proportion where its p95 doesn't break the overall budget.

There is a cost ceiling too. Raising thinking_level loads thinking tokens onto the output side, so leaning on high spikes the per-request unit cost. If you expose the AI feature to free users, the test is whether AdMob revenue covers the per-request cost. In my case I pinned free-tier requests to low by default and reserved medium-and-up for actions initiated by premium members. Both speed and unit cost land on the same one-page table of who gets which level, and when.

Catching mis-routes

This router's weak spot is dropping a hard question to low. It's faster, but the answer goes shallow and reads as off-target to the user. The nasty part is that a mis-route throws no exception: the call completes normally and simply returns a lower-quality answer, so without monitoring you never notice.

The detection I run in production is two cheap signals together. One records requests that were handled at low yet produced output longer than expected — if a supposedly shallow question needed a long answer, the difficulty read was probably wrong. The other ties a lightweight "was this helpful?" tap right after the answer to the level it was routed at. If low-routed requests skew toward "not helpful," that's the cue to revisit the heuristic threshold or _HARD_HINTS.

I also fix the order of response. First raise the share of gray-zone requests sent to the model judge; if a pattern still slips through, add exactly one word for it to _HARD_HINTS. Making the heuristic too clever makes the pre-pass heavy, so I only add words I have evidence actually missed.

Whether the Flash pre-pass pays off

The core of the two-stage design is deferring only the gray zone to a model. The obvious worry is whether the extra judgment call costs more than it saves. Measuring it settled the question.

In my feature the heuristic alone settled about 70% of requests, leaving roughly 30% for the model judge. The judgment cost on that 30% is a tiny max_output_tokens=4 call — a fraction of a real answer's cost. Meanwhile, having that stage cut the mistake of running easy questions at high, shrinking the total volume of deliberation. The small cost of judging came out smaller than the wasted deep-thinking it avoided.

The pre-pass doesn't always pay, though. In a feature where nearly every request sits at the same difficulty, there's little to route, and only the router's complexity remains. There, drop the extra stage and run one fixed thinking_level. Routing earns its keep where difficulty is bimodal. The thinking_level API itself is documented in the Gemini API docs; check the defaults on your own model before you set the ceiling.

Before you ship it

If an interactive AI feature is suffering from uneven wait times, the first step is to run a few dozen representative requests at low, medium, and high, and measure response time and accuracy on your own feature once. My numbers are only true for my inputs; change the feature and the break points move. Once you have your own figures, where to draw the ceiling stops being a guess.

Boiled down, designing the routing was the work of putting "which requests, and which users, pay for speed versus depth" onto one page. I'm still tuning it, but just dropping the fixed value and holding that one page made the feature feel visibly steadier. Thanks for reading.