What Decided Our Cascade's Economics Wasn't the Escalation Rate

Solve for the break-even

At what escalation rate does the cascade cost the same as the strong model alone?

def breakeven_e(t_in, t_out_lite, t_out_flash, k=1.0) -> float:
    """Escalation rate at which the cascade matches the strong model alone."""
    strong = unit_cost(FLASH, t_in, t_out_flash)
    first  = unit_cost(LITE,  t_in, t_out_lite)
    second = unit_cost(FLASH, t_in, t_out_flash * k)
    if second <= 0:
        return float("inf")
    return (strong - first) / second

Solved for three workload shapes. Input-dominant is one image plus a short instruction (1,600 in / 40 out), balanced is summarization or extraction (2,000 / 800), output-dominant is generation with reasoning (600 / 2,400).

For an output-dominant workload, moving k from 1.0 to 2.0 cuts the tolerable escalation rate from 67.3% to 34.5% — roughly half. My confidence that "25% leaves plenty of headroom" came from reading only the k=1.0 column.

Holding the escalation rate at 25%, here is what savings look like per million items.

Same escalation rate, same price table. Only k differs — and for the output-dominant workload it takes savings from 42.3% down to 18.5%.

The part that ran opposite to my prediction

My hypothesis going in: escalated items pay input twice, so the heavier the input, the worse a cascade performs.

The arithmetic said the opposite.

Where input is 90% of spend, doubling k moves the break-even by only 10.0%. Where output dominates, it moves 48.8%.

The reason is the direction of the price gap. Input is 5× cheaper on the small model, so the first stage's input costs 20% of the strong model's. Paying twice gives 0.30 + 1.50 = 1.80 against 1.50 for the strong model alone — an increment of 0.30. Small. But k applies only to the output side. When output dominates spend, k passes through almost undiminished.

Double-billed input and biased output length are risks for different workloads. Input-heavy pipelines survive the double billing. Output-heavy pipelines break on the bias, not the double billing.

I could not see that inversion until I wrote the equation and pushed numbers through it. Staring at a price table gives you no feel for which term is doing the work.

How much a wrong k costs

k is an estimate, so here is the sensitivity (output-dominant, e fixed at 0.25).

At k=3.0 savings go negative: the cascade costs more than the strong model alone. Missing k by the gap between 2.0 and 2.5 swings savings by 12 points.

Paying for the gate turned out to be second-order

Should the escalation decision be its own call, or should the cheap stage report a confidence score inline? I went back and forth on this for a while, because sending the input again just to decide felt wasteful.

Here are the savings with a Flash-Lite gate call on every item (output-dominant, k=2.0).

No gate versus a 30-token gate is a 1.4-point difference. Even a 300-token gate costs 4.9 points. Moving the escalation rate from 0.10 to 0.40 costs 58 points. A separate gate call resends the input, so even at zero output tokens it runs $180 per million items — still a different order of magnitude.

Saving on gate cost at the expense of gate accuracy was optimizing the wrong term. I changed the design here: the inline confidence score is gone, and the decision lives in its own small call.

When production escalation runs past your estimate

Calculating a break-even does not stop the escalation rate from spiking in production. On a day when the input distribution shifted — a run of images with fine-grained composition — escalation came in at roughly double my estimate. That day's spend beat the strong model alone.

My fix is a per-window escalation cap. Once the cap is hit, remaining items keep the cheap-stage result and get a needs_review flag so the next window can pick them up.

The gotcha is implementing that cap as a silent drop. Without the flag, the quality regression shows up in neither the invoice nor the logs.

class EscalationBudget:
    """Per-window escalation cap; overflow defers to the next window."""
 
    def __init__(self, window_items: int, max_rate: float):
        if not 0.0 < max_rate <= 1.0:
            raise ValueError(f"max_rate out of range: {max_rate}")
        self.cap = max(1, int(window_items * max_rate))
        self.used = 0
        self.deferred: list[str] = []
 
    def allow(self, item_id: str) -> bool:
        if self.used < self.cap:
            self.used += 1
            return True
        self.deferred.append(item_id)   # never drop silently
        return False

I recommend setting the cap somewhat inside the computed break-even. Mine sits at 70% of it. Savings may land below the estimate, but the cascade never ends up costing more than the single-model setup.

Only two numbers need measuring before you switch

e and k. Neither is observable from the cheap stage alone, because you cannot measure the strong stage's output length without running the strong stage.

So run a small paired sample first. Send the same inputs to both models and record per-item output tokens alongside the gate decision.

import random
import statistics
from typing import Iterable, NamedTuple
 
class Paired(NamedTuple):
    item_id: str
    escalated: bool       # did the gate mark this for escalation
    out_tokens_flash: int # strong-stage output tokens, from usage_metadata
 
def estimate_e_and_k(rows: Iterable[Paired]) -> dict:
    rows = list(rows)
    if not rows:
        raise ValueError("no paired rows collected")
    esc = [r.out_tokens_flash for r in rows if r.escalated]
    mean_all = statistics.fmean(r.out_tokens_flash for r in rows)
    if mean_all <= 0:
        raise ValueError("mean output tokens is zero; check usage_metadata parsing")
    e = len(esc) / len(rows)
    k = (statistics.fmean(esc) / mean_all) if esc else 1.0
    return {"n": len(rows), "e": e, "k": k, "mean_all": mean_all}
 
def bootstrap_k(rows: list[Paired], iters: int = 2000, seed: int = 17) -> tuple:
    """95% interval for k. A small escalated subset widens it honestly."""
    rng = random.Random(seed)
    ks = []
    for _ in range(iters):
        sample = [rows[rng.randrange(len(rows))] for _ in range(len(rows))]
        try:
            ks.append(estimate_e_and_k(sample)["k"])
        except ValueError:
            continue
    if len(ks) < iters * 0.5:
        raise RuntimeError("bootstrap unstable; collect more paired rows")
    ks.sort()
    return ks[int(0.025 * len(ks))], ks[int(0.975 * len(ks))]

For sample sizing, here are Wilson intervals assuming an escalation rate near 0.25.

At 50 samples the interval spans 23.4 points, which says nothing useful about headroom against a 34.5% break-even. At 400 you get roughly 4 points per side. k is worse off, since only escalated items contribute — at 400 paired rows the effective sample is around 100. I always report the bootstrap interval alongside the point estimate.

Why I stopped using self-reported confidence as the gate

Asking the cheap stage for a 0-to-1 confidence score was my first attempt, and I dropped it. Most of the misclassifications came back with high confidence. The premise that uncertain items score low simply did not hold for my subject matter.

What I run instead: sample the cheap stage twice and escalate when the two results disagree. No separate judge model, just double the cheap-stage cost.

Whether that doubling is affordable comes out of the same equation (e=0.25, k=2.0).

An input-dominant workload keeps 29.3% savings even with the cheap stage run twice. Balanced drops to 0.6%, which makes the whole arrangement pointless. My wallpaper classification sits on the input-dominant side, so double sampling was available to me. I have not carried the same gate over to the summarization pipeline.

Applying this to your own workload

Three steps.

1. Measure input and output tokens on even a single real item, and compute what share of spend is input
2. Run a small paired sample to measure e and k. k is the escalated subset's mean output length divided by the population mean
3. Feed k into breakeven_e, then compare the result against your measured e

If input is more than 80% of spend, a cascade is structurally forgiving about escalation-rate drift. If output dominates, switching without measuring k first is a gamble.

A 5× gap in unit price is not a 5× gap in total spend. The multipliers apply to different terms and to different subsets. I mixed those two together, built an estimate on it, and rebuilt it. If this saves you one round of that, it has done its job. Thank you for reading.

Prices quoted here were verified on July 30, 2026. They do get revised, so confirm against primary sources before running your own numbers.