Your Night Batch Is Causing the Morning 429s — Priority Admission Control for a Shared Gemini Quota

The Priority Token Bucket

Internally it's a classic token bucket split into two lanes. The only LLM-specific part is that it meters both RPM and TPM.

import asyncio
import time
from contextlib import asynccontextmanager
 
class PriorityAdmissionGate:
    def __init__(self, rpm_limit: int, tpm_limit: int,
                 reserved_interactive_ratio: float = 0.3,
                 safety_margin: float = 0.85):
        # stop short of the API's real ceiling (headroom for other processes)
        self._rpm = rpm_limit * safety_margin
        self._tpm = tpm_limit * safety_margin
        self._reserved = reserved_interactive_ratio
        self._req_tokens = self._rpm   # request-count bucket
        self._tok_tokens = self._tpm   # token-count bucket
        self._last = time.monotonic()
        self._cond = asyncio.Condition()
 
    def _refill(self):
        now = time.monotonic()
        elapsed = now - self._last
        self._last = now
        self._req_tokens = min(self._rpm, self._req_tokens + self._rpm * elapsed / 60)
        self._tok_tokens = min(self._tpm, self._tok_tokens + self._tpm * elapsed / 60)
 
    def _floor(self, priority: str) -> tuple[float, float]:
        """Bulk must stop above the reserved line; interactive can drain to zero."""
        if priority == "interactive":
            return (0.0, 0.0)
        return (self._rpm * self._reserved, self._tpm * self._reserved)
 
    @asynccontextmanager
    async def acquire(self, *, feature: str, priority: str, est_tokens: int):
        async with self._cond:
            req_floor, tok_floor = self._floor(priority)
            while True:
                self._refill()
                if (self._req_tokens - 1 >= req_floor
                        and self._tok_tokens - est_tokens >= tok_floor):
                    self._req_tokens -= 1
                    self._tok_tokens -= est_tokens
                    break
                # not enough headroom: wait for refill (bulk decelerates naturally)
                await asyncio.wait_for(self._cond.wait(), timeout=1.0)
        try:
            yield
        finally:
            async with self._cond:
                self._cond.notify_all()

The heart of it is _floor(). The bulk lane is forced to wait before the bucket drops below the 30% reserve line, so an arriving interactive request always finds capacity immediately. At night, when interactive traffic is dead, bulk can run right down to the reserve line — so total throughput barely suffers. In my nighttime measurements, this borrowing behavior gave bulk 22% more effective throughput than a naive static split that permanently carves out 30%.

est_tokens is a pre-call estimate. You could call count_tokens, but that costs a request of its own, so for this purpose I use a rough heuristic (characters ÷ 2 plus the output cap) and correct afterward with the real usage_metadata numbers. The estimate ran about +11% high on average over two weeks — plenty accurate for preventing quota exhaustion, which is all it needs to do.

The Week RPM-Only Control Failed — TPM Is a Separate Cliff

My first implementation only metered RPM. It was fine for a week, then 429s came back the day I added long-document summarization to the bulk queue — with RPM nowhere near the limit. The culprit was TPM. A task with ~9,000 prompt tokens per item is trivial in request count and voracious in token count.

Per-request cost varying by an order of magnitude between features is characteristic of LLM workloads. Meter only one dimension and the "few but heavy" feature becomes a bypass. That failure is why the gate above carries two buckets from the start.

Bulk Etiquette — An External Slowdown Signal

Admission control is in-process traffic sorting; it can't see congestion the API sees (other machines, other processes, shifting limits). So my bulk workers carry one more brake: a simple circuit that watches the interactive lane's 429 rate and p95 latency at one-minute granularity and pauses bulk when either degrades.

async def bulk_worker(items, gate, health):
    for item in items:
        # if the interactive lane looks unhealthy, bulk quietly waits
        while not health.interactive_healthy():
            await asyncio.sleep(30)
        async with gate.acquire(feature="bulk_regen", priority="bulk",
                                est_tokens=estimate(item)):
            await process(item)
 
class InteractiveHealth:
    def __init__(self, max_429_rate=0.005, max_p95_sec=3.5):
        self._max_429 = max_429_rate
        self._max_p95 = max_p95_sec
        self.window = deque(maxlen=300)  # (ts, status, latency)
 
    def interactive_healthy(self) -> bool:
        recent = [w for w in self.window if w[0] > time.time() - 60]
        if not recent:
            return True
        rate_429 = sum(1 for _, s, _ in recent if s == 429) / len(recent)
        lats = sorted(l for _, _, l in recent)
        p95 = lats[int(len(lats) * 0.95) - 1] if len(lats) >= 20 else 0
        return rate_429 <= self._max_429 and p95 <= self._max_p95

A batch job's greatest asset is that pausing it hurts nobody. Automate the decision to pause the side that doesn't hurt, and on mornings when the batch tail collides with the interactive peak, bulk yields the road before damage is done. This is the defensive twin of the throughput-maximizing approach in my article on adaptive concurrency for bulk workloads — designing the offense and the defense as a pair works well.

Why "Just Use a Separate API Key" Doesn't Work

My very first idea was to issue a second API key for the bulk job. It does nothing here. Gemini API rate limits are enforced per project, not per key — split one project's traffic across ten keys and they still drain the same bucket. Key separation is meaningful for credential hygiene and permission scoping, but it is not quota isolation.

Real isolation means a separate project. Project-level separation is genuinely useful for containing blast radius, and I covered the cost-side version of that argument in my piece on spend caps and blast radius. But every extra project multiplies billing, monitoring, and key-management overhead, so my decision table looks like this:

Two Weeks of Numbers

Two weeks before versus two weeks after, measured application-side on first attempts (before any retry), interactive feature only:

Yes, the bulk job takes 16 minutes longer — that is precisely the time it spent yielding to the interactive peak, which is the design working as intended. Sixteen minutes on a night batch and a 429 in front of a waiting user are not equivalent costs. Encoding that asymmetry into the call path is what admission control is really for.

One honest caveat: if you effectively have one feature, or every call shares the same delay tolerance, this gate is pure complexity. A plain concurrency cap and a sane retry policy will serve you better. Add the gate only after measurement shows actual contention — that ordering keeps the system honest.

Where to Start — One Feature Tag

If you can't yet name the feature causing your 429s, don't start with the gate. Start with a week of feature-tagged measurement like the TaggedGeminiClient at the top. Once the culprit is visible, the reserve ratio and thresholds practically choose themselves from the table. In my case, attribution took three days of data, and the gate itself was one weekend of implementation and tuning. If your project shares one quota across features that don't share a deadline, I hope these notes save you the morning I lost.