Coalescing Gemini API Requests with SSE Fan-out: Collapsing 100 Simultaneous Hits into a Single Call

Decision 2: tuning the TTL window

Coalescing has a time window. A long window catches more requests but adds latency to the very first one. A short window keeps latency tight but reduces the herd-collapsing effect.

Fourteen days of measurement against AdMob reports on the wallpaper app pointed me at a hybrid: "TTL = 5 seconds during the 5-second window immediately after a push notification, TTL = 200ms otherwise."

The "notification active" flag lives in the Worker Cache API with a 6-second TTL, written by the notification dispatcher. The Durable Object reads that flag to pick a TTL. Compared to the previous fixed TTL=1s, costs dropped another 38% and p99 didn't degrade.

Decision 3: Durable Object granularity

Where you draw the Durable Object instance boundary directly determines coalescing performance. "One global instance" is obviously a single point of failure. "One instance per coalesceKey" sounds clean but pays cold-start cost on every key.

I settled on "one Durable Object per template ID." For an indie operation, there are typically 30–40 templates, of which only about 10 stay warm given Cloudflare's geographical distribution. Each DO holds an in-memory Map<coalesceKey, Promise<Response>> and resolves coalescing entirely within itself.

export class GeminiCoalescer {
  state: DurableObjectState;
  inflight: Map<string, Promise<GeminiResponse>>;
  cache: Map<string, { value: GeminiResponse; expiresAt: number }>;
 
  constructor(state: DurableObjectState) {
    this.state = state;
    this.inflight = new Map();
    this.cache = new Map();
  }
 
  async coalesce(
    key: string,
    fetchFn: () => Promise<GeminiResponse>,
    ttlMs: number,
  ) {
    const cached = this.cache.get(key);
    if (cached && cached.expiresAt > Date.now()) return cached.value;
 
    const inflight = this.inflight.get(key);
    if (inflight) return inflight;
 
    const promise = fetchFn().then(value => {
      this.cache.set(key, { value, expiresAt: Date.now() + ttlMs });
      this.inflight.delete(key);
      return value;
    }).catch(err => {
      this.inflight.delete(key);
      throw err;
    });
 
    this.inflight.set(key, promise);
    return promise;
  }
}

Storing the Promise itself in inflight is the key — late arrivals just await it. The catch branch must always call inflight.delete; without it, an error leaves a zombie Promise occupying the slot forever. I learned that the hard way one evening when Crashlytics lit up with 200 "Gemini timeout 30s" reports in a row.

Decision 4: SSE fan-out and the "primary departure" problem

Fan-out is easy when the response is a single JSON blob. With Gemini streaming, it gets noticeably harder. While one request is receiving streamed bytes from Gemini, the other 99 need to receive the same bytes concurrently.

The nastiest failure mode here is "primary departure." If the request that opened the connection to Gemini drops first — the user closes the screen, the network flips — the fan-out to the other 99 dies with it.

My fix was to give the Durable Object a background task that owns the Gemini connection independently of any single client.

async streamCoalesce(key: string, prompt: PromptSpec): Promise<ReadableStream> {
  let producer = this.streams.get(key);
  if (!producer) {
    producer = this.startGeminiStream(prompt);
    this.streams.set(key, producer);
  }
  return producer.subscribe();
}
 
private startGeminiStream(prompt: PromptSpec): StreamProducer {
  const buffer: string[] = [];
  const subscribers = new Set<WritableStreamDefaultWriter>();
  let complete = false;
 
  // Gemini connection is owned by the DO, not by any subscriber
  (async () => {
    const stream = await callGeminiStreaming(prompt);
    for await (const chunk of stream) {
      buffer.push(chunk);
      for (const sub of subscribers) {
        try { await sub.write(chunk); } catch { subscribers.delete(sub); }
      }
    }
    complete = true;
  })();
 
  return {
    subscribe(): ReadableStream {
      const { readable, writable } = new TransformStream();
      const writer = writable.getWriter();
      // backfill prior chunks
      for (const c of buffer) writer.write(c);
      if (complete) writer.close();
      else subscribers.add(writer);
      return readable;
    }
  };
}

The buffer collects all chunks seen so far so that a late subscriber gets backfilled before catching up to the live stream. The very concept of "primary departure" disappears.

The trap is Durable Object memory and CPU limits. Per-stream buffers above ~50 KB get problematic, so I added a scheduler via state.storage.setAlarm that drops the buffer 30 seconds after the last subscriber closes.

Decision 5: backfill for reconnecting clients

In a notification-driven environment, reconnects are routine. AdMob ad-load blocks the main thread, iOS URLSession breaks on background resume — the causes are varied. Crashlytics shows reconnects account for 17% of all stream lifetimes.

Reconnecting clients send a Last-Event-ID header indicating how far they got, and the Durable Object resumes from its buffer. That's vanilla SSE behavior, but combining it with the fan-out design has a nice side effect: most reconnecting clients can still board the same in-flight stream rather than triggering a fresh Gemini call.

async handleReconnect(req: Request): Promise<Response> {
  const key = req.headers.get("X-Coalesce-Key")!;
  const lastEventId = parseInt(req.headers.get("Last-Event-ID") ?? "0", 10);
 
  const producer = this.streams.get(key);
  if (!producer) {
    // stream already finished — return KV snapshot
    const snapshot = await this.env.SNAPSHOT_KV.get(`stream:${key}`);
    return snapshot
      ? new Response(snapshot)
      : new Response(null, { status: 410 });
  }
 
  return new Response(producer.subscribeFrom(lastEventId), {
    headers: { "Content-Type": "text/event-stream" }
  });
}

Over 14 days of telemetry, 84% of reconnects landed on an in-flight stream, with only 16% falling back to the KV snapshot. Even that fallback doesn't call Gemini again.

Measured: 14 days of cost and latency

The table below covers 14 evening (21:00 JST) push notifications during late April 2026. The baseline is the same traffic pattern under the previous KV-only setup running through March 2026.

Past the 30-second mark, the coalesce rate drops sharply, but latency requirements relax too, so I let the system fall back to plain KV caching.

My recommended decision order for indie developers

I've applied this design to the wallpaper app and several other indie projects. For a personal scale, deciding whether to adopt this architecture comes down to a quick four-step gate.

1. Do you see 50 or more identical-prompt requests within 10 seconds of a push notification or email blast? Below 50, a plain KV cache is fine and the Durable Object maintenance cost isn't worth it.
2. Can your prompts be split into a template plus per-user variables? Without that split, the coalescing key collides too often and the win evaporates.
3. Are you calling Gemini in streaming mode? Single-shot JSON makes coalescing trivial; streaming requires subscribe/backfill scaffolding.
4. Does your monthly Gemini API spend exceed ¥10,000? Below that, the engineering effort rarely pays back.

If any answer is "no," start with a simple KV cache and in-process dedup (a plain Map). Since teaching myself programming in 1997, I've repeatedly thrown out simple solutions in favor of clever architectures and regretted it. Less code usually wins.

A back-of-the-envelope cost estimator

Here's the function I use at the final go/no-go gate. It's deliberately rough, assumes Cloudflare Workers + Durable Objects pricing, and produces savings estimates accurate enough to decide.

def estimate_savings(
    notifications_per_month: int,
    requests_per_notification: int,
    coalesce_rate: float,       # 0.0 - 1.0
    avg_gemini_cost_yen: float, # per request
    do_request_cost_yen: float = 0.00015,  # rough Cloudflare DO unit cost
):
    naive_cost = notifications_per_month * requests_per_notification * avg_gemini_cost_yen
    coalesced_requests = requests_per_notification * (1 - coalesce_rate)
    new_cost = notifications_per_month * (
        coalesced_requests * avg_gemini_cost_yen
        + requests_per_notification * do_request_cost_yen
    )
    return {
        "naive": naive_cost,
        "coalesced": new_cost,
        "savings": naive_cost - new_cost,
        "ratio": new_cost / naive_cost if naive_cost else 1.0,
    }
 
# My case: 14 notifications/month, 800 requests each, 71% coalesce rate
print(estimate_savings(14, 800, 0.71, 4.2))
# {'naive': 47040.0, 'coalesced': 13641.6, 'savings': 33398.4, 'ratio': 0.29}

Actual production savings exceed the estimate slightly. The reasons are AdMob-triggered re-entries and reconnecting clients that opportunistically board later coalesced streams. As a rule of thumb, if the estimator predicts a 70%+ reduction, the architecture is worth shipping.

Production gotchas

Three things that bit me during the first week:

1. Durable Object hibernation wipes the in-memory Maps. After 30 seconds of inactivity the DO hibernates and both inflight and cache Maps vanish. This is by design, but the moment the DO re-warms, coalescing collapses and a fresh burst hits Gemini. The fix is a state.storage.setAlarm set on a 25-second cadence to keep the DO active. Monthly DO request billing goes up a little, but the saved Gemini spend dwarfs the difference.

2. Structured-output schema violations fan out to every caller. If Gemini returns a payload that fails JSON-schema validation, all 100 coalesced clients receive the same error. The mitigation: validate inside the DO with a Pydantic-style guard, and on failure demote the request to individual retry — abandoning coalescing rather than amplifying the bug.

3. AdMob ad-load contention. When a fresh app launch fires AdMob and the Gemini recommendation request in parallel, iOS URLSession ends up arbitrating bandwidth. Delaying the Gemini subscribe by 200ms until AdMob finishes loading dropped Crashlytics ANR counts by 38%.

That's the full picture of the coalescing + fan-out design I've been running alongside the 50M-download wallpaper app as an indie developer. The architecture looks ornate on a whiteboard, but in operation it feels like quiet, dependable infrastructure — the kind of optimization that comes from staring at one specific cost curve for too long. If you're feeling the same 10-second cliff after every push, hopefully this maps cleanly onto your setup.

Looking back at running the business since 2014, the most useful insight from 50M cumulative downloads is that "cost optimization is like sculpting — the more you strip away, the clearer the shape becomes." Next on my list is wiring a model router between Gemini Flash and Pro per coalescing key, which I'll write up separately.