Building a Production-Grade Gemini API Backend with NestJS — DI, Filters, and Guards

Streaming responses with @Sse()

NestJS exposes Server-Sent Events through the @Sse() decorator. By returning an Observable, you get automatic teardown when the client disconnects — no manual AbortController plumbing needed.

// src/gemini/gemini.controller.ts
import { Body, Controller, Post, Sse, MessageEvent } from '@nestjs/common';
import { Observable } from 'rxjs';
import { GeminiService } from './gemini.service';
 
@Controller('gemini')
export class GeminiController {
  constructor(private readonly gemini: GeminiService) {}
 
  @Sse('stream')
  stream(@Body() body: { prompt: string }): Observable<MessageEvent> {
    return new Observable((subscriber) => {
      (async () => {
        try {
          for await (const text of this.gemini.stream(body.prompt)) {
            subscriber.next({ data: { text } });
          }
          subscriber.complete();
        } catch (err) {
          subscriber.error(err);
        }
      })();
    });
  }
}

One thing worth knowing: calling subscriber.error() does not route through your exception filter. SSE treats errors as a stream-end event. If you need the client to react to mid-stream Gemini failures, send a typed payload like subscriber.next({ data: { error: 'rate_limited' } }) and let the front-end decide what to do.

A single exception filter for Gemini errors

Gemini's failure modes — 429 quota, 503 overloaded, 400 safety — all want different client-side handling. Doing this with try/catch in every controller adds noise. Centralize it in a @Catch() filter.

// src/gemini/gemini-exception.filter.ts
import { ArgumentsHost, Catch, ExceptionFilter, HttpStatus } from '@nestjs/common';
import { Response } from 'express';
 
@Catch()
export class GeminiExceptionFilter implements ExceptionFilter {
  catch(exception: any, host: ArgumentsHost) {
    const res = host.switchToHttp().getResponse<Response>();
    const msg = String(exception?.message ?? exception);
 
    if (msg.includes('RESOURCE_EXHAUSTED') || msg.includes('429')) {
      return res.status(HttpStatus.TOO_MANY_REQUESTS).json({
        error: 'rate_limited',
        retryAfter: 30,
      });
    }
    if (msg.includes('SAFETY') || msg.includes('blocked')) {
      return res.status(HttpStatus.BAD_REQUEST).json({
        error: 'content_blocked',
      });
    }
    return res.status(HttpStatus.INTERNAL_SERVER_ERROR).json({
      error: 'gemini_failure',
      detail: process.env.NODE_ENV === 'production' ? undefined : msg,
    });
  }
}

Wire it up globally with app.useGlobalFilters(new GeminiExceptionFilter()) in bootstrap(). Normalizing Gemini's errors into a stable shape makes both retry logic and front-end UX dramatically easier — and hiding raw detail outside development prevents leaking internal traces.

Guards keep auth and rate limits out of controllers

For a public-facing wrapper, you don't want auth code mixed into business logic. A Guard does the trick:

// src/gemini/api-key.guard.ts
import { CanActivate, ExecutionContext, Injectable, UnauthorizedException } from '@nestjs/common';
 
@Injectable()
export class ApiKeyGuard implements CanActivate {
  canActivate(ctx: ExecutionContext): boolean {
    const req = ctx.switchToHttp().getRequest();
    const key = req.headers['x-api-key'];
    if (!key || key !== process.env.PUBLIC_API_KEY) {
      throw new UnauthorizedException('invalid api key');
    }
    return true;
  }
}

For rate limits, layer on @nestjs/throttler. If you need per-user (rather than per-IP) throttling, subclass ThrottlerGuard and override getTracker() to return req.headers['x-api-key']. Putting a soft limit one layer in front of Gemini's own quota stops a single noisy client from torching your billing or cascading a 429 storm into your other endpoints.

Keep multi-model fallback inside the Service

The single change that helped most in production was falling back from gemini-2.5-flash to gemini-2.5-pro when the former returned a 503 (overloaded). The important part is keeping that decision inside GeminiService rather than in the controller. The moment a controller knows which model to pick, every model swap turns into a hunt across call sites.

// src/gemini/gemini.service.ts (excerpt)
private readonly fallbackOrder = ['gemini-2.5-flash', 'gemini-2.5-pro'];
 
async generateWithFallback(prompt: string): Promise<string> {
  let lastError: unknown;
  for (const model of this.fallbackOrder) {
    try {
      const m = this.client.getGenerativeModel({ model });
      const res = await m.generateContent(prompt);
      return res.response.text();
    } catch (err) {
      const msg = String((err as Error)?.message ?? err);
      // Only overload / quota moves to the next model; safety blocks re-throw.
      if (msg.includes('503') || msg.includes('RESOURCE_EXHAUSTED')) {
        lastError = err;
        continue;
      }
      throw err;
    }
  }
  throw lastError;
}

The rule I settled on is "don't fall back on everything." A safety block (400) is caused by the input itself, so retrying it on another model produces the same result and just doubles your cost and latency. Only overload and quota errors advance to the next candidate; everything else flows straight to the exception filter. The side effect — higher latency and unit cost whenever a request lands on Pro — is something you want the metrics below to make visible.

Centralize exponential-backoff retries in an Interceptor

A 503 usually clears within a few hundred milliseconds, so a single immediate retry noticeably lifts your perceived success rate. But writing retry loops into each method tangles them with the fallback code. Pulling it into a NestJS Interceptor lets you wrap retries over everything without touching the controller or the service.

// src/gemini/retry.interceptor.ts
import { CallHandler, ExecutionContext, Injectable, NestInterceptor } from '@nestjs/common';
import { Observable, throwError } from 'rxjs';
import { retry, timer } from 'rxjs';
 
@Injectable()
export class RetryInterceptor implements NestInterceptor {
  intercept(_ctx: ExecutionContext, next: CallHandler): Observable<unknown> {
    return next.handle().pipe(
      retry({
        count: 3,
        delay: (error, retryCount) => {
          const msg = String((error as Error)?.message ?? error);
          const retriable = msg.includes('503') || msg.includes('RESOURCE_EXHAUSTED');
          if (!retriable) return throwError(() => error);
          // Exponential backoff + jitter (around 250ms, 500ms, 1000ms).
          const base = 250 * 2 ** (retryCount - 1);
          const jitter = Math.random() * 100;
          return timer(base + jitter);
        },
      }),
    );
  }
}

Jitter is non-negotiable: without it, many requests that hit a 503 at the same instant all retry on the same schedule and hammer the API again in unison — a retry "thundering herd." The numbers can stay modest. In my own operation, starting at 250ms and capping at 3 attempts absorbed most transient overload without meaningfully growing perceived wait time. Non-retriable errors are re-thrown via throwError and left to the exception filter to normalize.

Minimal observability: measure latency and token usage

The line between "it works" and "it's operable" is almost always the moment you add measurement. Before reaching for a heavy monitoring stack, an Interceptor that records elapsed time and pulls token counts from Gemini's usageMetadata already gives you something to reason about.

// src/gemini/metrics.interceptor.ts
import { CallHandler, ExecutionContext, Injectable, Logger, NestInterceptor } from '@nestjs/common';
import { Observable, tap } from 'rxjs';
 
@Injectable()
export class MetricsInterceptor implements NestInterceptor {
  private readonly logger = new Logger('GeminiMetrics');
 
  intercept(ctx: ExecutionContext, next: CallHandler): Observable<unknown> {
    const started = Date.now();
    const route = ctx.switchToHttp().getRequest().url;
    return next.handle().pipe(
      tap((payload: any) => {
        const ms = Date.now() - started;
        const usage = payload?.usageMetadata;
        this.logger.log(
          `route=${route} latency_ms=${ms} ` +
          `in=${usage?.promptTokenCount ?? '-'} out=${usage?.candidatesTokenCount ?? '-'}`,
        );
      }),
    );
  }
}

Watching this log for about a week tells you which endpoints eat tokens and how often you fall back to Pro. Here are the numbers I measured on my own wrapper API over one week (about 12,000 requests). They reflect my configuration and prompt lengths, but they gave me a concrete basis for model selection and fallback policy.

What this told me is that fallback fired on under 1% of traffic, so the volume drifting onto the pricier Pro stayed small. The p95 spikes, meanwhile, were driven mostly by prompt length rather than retries — trimming input tokens did more for perceived speed than anything else. Without measurement, I'd almost certainly have optimized the wrong thing. When you're ready to grow this into real observability, Production-Grade Observability for the Gemini API with Langfuse drops in as a direct replacement for this Interceptor.

Three things that bite people in production

The setup above is functionally complete, but here are three traps I keep seeing right before launch.

The first is buffered streaming. On Cloud Run, Fargate, or anything fronted by Nginx/ALB, SSE often appears to "freeze" for several seconds. The fix is usually X-Accel-Buffering: no on the response, or an LB-level setting that allows streaming. If your first chunk takes 3+ seconds to appear, suspect this before suspecting Gemini.

The second is forgetting ConfigModule.forRoot({ isGlobal: true }) in AppModule, which leaves process.env.* references scattered everywhere. Going through ConfigService from day one means swapping in Vault or Secret Manager later is a one-file diff instead of a hunt.

The third is unit tests that talk to the real API because GoogleGenerativeAI was never mocked. CI runs burn quota and money. The whole reason the SDK client is a Provider in the first place is so your test module can do { provide: 'GENAI_CLIENT', useValue: mock } and stay offline.

Where to go next

Moving Gemini onto NestJS feels heavy for the first day or two, but somewhere around the third feature addition it pays off in obvious ways. Start with a single /gemini/stream endpoint, then layer the exception filter and guards on top. Once you are handling function calls in production, Production Patterns for Function Calling with the Gemini 2.5 Pro API is a natural next step; when 503s start piling up, Causes and Fixes for the Gemini API 503 Error covers the retry side in depth; and to keep costs down, Operating Notes on Gemini API Caching Strategy picks up where this layout leaves off. Thanks for reading.