◈ API / SDK/2026-06-22Advanced

Gemini API on Google Cloud: Diagnosing Production Errors Layer by Layer

Systematically diagnose Gemini API errors in Google Cloud production environments. Covers IAM permissions, Vertex AI vs AI Studio, VPC Service Controls, quota management, service accounts, and multi-region failover with full code examples.

Gemini API¹⁹² Google Cloud⁵ Vertex AI¹¹ IAM production¹⁴⁰ troubleshooting⁸²

✦ Premium Article

Running several Dolice Labs sites on Google Cloud as an indie developer, I have lost more afternoons than I would like to a Gemini API call that ran fine locally and then returned a 403 or 429 the moment it reached production. One IAM role, one service-account key, and half a day is gone. Out of those afternoons came a habit: read the error string first, isolate which layer it belongs to, and only then touch anything. This guide walks through each production-specific failure category — auth, network, quota, and model deprecation — with diagnosis steps and working Python code to resolve them.

Two Entry Points: AI Studio vs Vertex AI in Production

Understanding which API endpoint you're targeting is the first thing to get right. Mismatches between development and production configurations are a common source of errors.

Google AI Studio API (generativelanguage.googleapis.com)

Authenticates with an API key
Designed for rapid prototyping and personal projects
Not subject to VPC Service Controls or organization policies
No enterprise SLA

Vertex AI Gemini API (aiplatform.googleapis.com)

Authenticates with service accounts or OAuth2
Enterprise-grade: SLA, audit logs, VPC support, CMEK support
Fine-grained access control via IAM
Region selection for data residency compliance

Recommendation: use Vertex AI for production. It supports your organization's security policies, enables least-privilege IAM implementation, allows private network access within your VPC, and integrates with Cloud Monitoring and Cloud Logging.

IAM and Service Account Errors

PERMISSION_DENIED

google.api_core.exceptions.PermissionDenied: 403 Permission 'aiplatform.endpoints.predict' denied

This means the service account running your application doesn't have the required IAM role.

Check what roles are currently assigned:

gcloud projects get-iam-policy YOUR_PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:YOUR_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --format="table(bindings.role)"

Minimum roles required for Vertex AI Gemini API:

roles/aiplatform.user: Send prediction requests to Vertex AI endpoints
roles/ml.viewer: List and view models (optional)

Grant the required role:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:YOUR_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

Using Application Default Credentials (Recommended)

In production, avoid bundling service account key JSON files in your application. Use Application Default Credentials (ADC) instead — on Cloud Run, GKE, and Compute Engine, the attached service account is used automatically.

import vertexai
from vertexai.generative_models import GenerativeModel
 
# On managed Google Cloud infrastructure, no credential setup needed
vertexai.init(project="YOUR_PROJECT_ID", location="us-central1")
model = GenerativeModel("gemini-2.0-flash-001")
 
response = model.generate_content("Hello from production!")
print(response.text)

For local development:

# Set up ADC using your personal credentials
gcloud auth application-default login
 
# Impersonate a service account to test production permissions locally
gcloud auth application-default login \
  --impersonate-service-account=YOUR_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com

Service Account Impersonation Errors

google.auth.exceptions.TransportError: Unable to fetch token

When impersonating a service account, the calling identity needs the roles/iam.serviceAccountTokenCreator role on the target service account.

gcloud iam service-accounts add-iam-policy-binding \
  TARGET_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --member="user:YOUR_EMAIL@example.com" \
  --role="roles/iam.serviceAccountTokenCreator"

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦A diagnosis flow that splits production errors into four layers: auth, network, quota, and model deprecation

✦A startup preflight that detects deprecated models and auto-falls-back to a successor across shutdown dates

✦Step-by-step fixes for IAM least-privilege, VPC-SC, and quota errors with working code

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

API Enablement Errors

Vertex AI Gemini requires several APIs to be enabled in your project.

Check and enable required APIs:

# Check which APIs are enabled
gcloud services list --enabled --filter="name:(aiplatform OR ml OR storage)"
 
# Enable missing APIs
gcloud services enable \
  aiplatform.googleapis.com \
  ml.googleapis.com \
  storage.googleapis.com \
  --project=YOUR_PROJECT_ID

Note: API enablement takes a few minutes to propagate. Requests immediately after enabling may return SERVICE_DISABLED. Verify the state programmatically:

from googleapiclient.discovery import build
from google.auth import default
 
credentials, project = default()
service = build('serviceusage', 'v1', credentials=credentials)
 
response = service.services().get(
    name=f"projects/{project}/services/aiplatform.googleapis.com"
).execute()
 
print(f"API state: {response.get('state')}")  # Should be "ENABLED"

Quota and Rate Limit Errors

429 Resource Exhausted

google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric
'generate_requests_per_minute_per_project_per_base_model'

Check your quota usage:

# View quotas in Cloud Console
# IAM & Admin → Quotas → Filter: "aiplatform"

Retry with exponential backoff:

import time
import random
import google.api_core.exceptions
import vertexai
from vertexai.generative_models import GenerativeModel
 
vertexai.init(project="YOUR_PROJECT_ID", location="us-central1")
model = GenerativeModel("gemini-2.0-flash-001")
 
def generate_with_retry(prompt: str, max_retries: int = 5) -> str:
    for attempt in range(max_retries):
        try:
            response = model.generate_content(prompt)
            return response.text
 
        except google.api_core.exceptions.ResourceExhausted as e:
            if attempt >= max_retries - 1:
                raise
 
            retry_delay = None
            if hasattr(e, 'metadata'):
                for key, value in e.metadata:
                    if key == 'retry-info':
                        retry_delay = float(value)
                        break
 
            if retry_delay is None:
                retry_delay = min(2 ** attempt + random.uniform(0, 1), 60)
 
            print(f"Quota exceeded: retrying in {retry_delay:.1f}s (attempt {attempt + 1}/{max_retries})")
            time.sleep(retry_delay)
 
        except google.api_core.exceptions.DeadlineExceeded:
            if attempt >= max_retries - 1:
                raise
            time.sleep(2 ** attempt)
 
    return ""

Requesting quota increases:

If you consistently hit quota limits, request an increase through Cloud Console → IAM & Admin → Quotas. Approval typically takes a few business days.

Tokens Per Minute (TPM) Exhaustion

When processing large volumes of text, you may hit TPM limits before RPM limits.

import time
 
def batch_process_with_tpm_control(
    texts: list[str],
    tpm_limit: int = 100000
) -> list[str]:
    results = []
    current_minute_tokens = 0
    minute_start = time.time()
 
    for text in texts:
        estimated_tokens = len(text) // 4  # rough estimate: ~4 chars per token
 
        if current_minute_tokens + estimated_tokens > tpm_limit * 0.9:
            elapsed = time.time() - minute_start
            if elapsed < 60:
                wait_time = 60 - elapsed + 1
                print(f"Approaching TPM limit: waiting {wait_time:.1f}s")
                time.sleep(wait_time)
            current_minute_tokens = 0
            minute_start = time.time()
 
        response = generate_with_retry(text)
        results.append(response)
        current_minute_tokens += estimated_tokens
 
    return results

VPC Service Controls and Private Service Connect

In enterprise environments, VPC Service Controls (VPC-SC) can block Gemini API access even when IAM permissions are correct.

Identifying VPC-SC Errors

google.api_core.exceptions.PermissionDenied: 403 Request is prohibited by organization's policy.
vpcServiceControlsUniqueIdentifier: XXXXXXXX

This is a VPC-SC policy block, not an IAM issue.

Diagnose with audit logs:

gcloud logging read \
  'protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"
   AND protoPayload.status.code=7
   AND resource.type="audited_resource"' \
  --freshness=1h \
  --format="json" | head -100

Fix: add Vertex AI to the VPC-SC perimeter

Your organization admin needs to add aiplatform.googleapis.com to the accessible services in your VPC-SC perimeter:

gcloud access-context-manager perimeters update PERIMETER_NAME \
  --add-restricted-services=aiplatform.googleapis.com \
  --policy=POLICY_NUMBER

Private Service Connect

For production traffic that should never leave your private network:

import vertexai
 
vertexai.init(
    project="YOUR_PROJECT_ID",
    location="us-central1",
    api_endpoint="YOUR_PSC_ENDPOINT.p.googleapis.com"
)

Find your PSC endpoint in Cloud Console → Vertex AI → Private Service Connect.

Multi-Region Configuration and Failover

Production workloads shouldn't depend on a single region. Gemini API is available in multiple regions.

import vertexai
from vertexai.generative_models import GenerativeModel
import google.api_core.exceptions
import logging
 
logger = logging.getLogger(__name__)
 
REGION_PRIORITY = ["us-central1", "us-east4", "europe-west4", "asia-northeast1"]
 
def generate_with_region_failover(
    prompt: str,
    project_id: str
) -> tuple[str, str]:
    for region in REGION_PRIORITY:
        try:
            vertexai.init(project=project_id, location=region)
            model = GenerativeModel("gemini-2.0-flash-001")
            response = model.generate_content(prompt)
            return response.text, region
 
        except google.api_core.exceptions.ServiceUnavailable as e:
            logger.warning(f"Region {region} unavailable: {e}")
            continue
 
        except google.api_core.exceptions.DeadlineExceeded as e:
            logger.warning(f"Region {region} timed out: {e}")
            continue
 
    raise RuntimeError("All regions failed")
 
result, used_region = generate_with_region_failover("Failover test", "YOUR_PROJECT_ID")
print(f"Generated successfully in region: {used_region}")

Organization Policy Violations

google.api_core.exceptions.PermissionDenied: 403 RESOURCE_LOCATION_POLICY_VIOLATION

This means the region you're targeting isn't allowed by your organization's resource location policy.

Check the constraint:

gcloud resource-manager org-policies describe \
  constraints/gcp.resourceLocations \
  --project=YOUR_PROJECT_ID

Resolution requires your administrator to add the target region to the allowed values list, or switching to a permitted region in your code.

Cloud Monitoring Setup

from google.cloud import monitoring_v3
 
def create_gemini_api_alerts(project_id: str):
    client = monitoring_v3.AlertPolicyServiceClient()
    project_name = f"projects/{project_id}"
 
    # Alert when quota usage exceeds 80%
    quota_alert = monitoring_v3.AlertPolicy(
        display_name="Gemini API Quota Usage > 80%",
        conditions=[
            monitoring_v3.AlertPolicy.Condition(
                display_name="High quota usage",
                condition_threshold=monitoring_v3.AlertPolicy.Condition.MetricThreshold(
                    filter='resource.type="aiplatform.googleapis.com/Location"'
                           ' AND metric.type="aiplatform.googleapis.com/quota/online_prediction_requests_per_base_model/usage"',
                    comparison=monitoring_v3.ComparisonType.COMPARISON_GT,
                    threshold_value=0.8,
                    duration={"seconds": 300},
                )
            )
        ],
        notification_channels=["projects/YOUR_PROJECT_ID/notificationChannels/YOUR_CHANNEL_ID"],
        combiner=monitoring_v3.AlertPolicy.ConditionCombinerType.OR
    )
 
    policy = client.create_alert_policy(name=project_name, alert_policy=quota_alert)
    print(f"Alert policy created: {policy.name}")

Key metrics to monitor: request count and error rate via aiplatform.googleapis.com/prediction/online/request_count, latency distribution via aiplatform.googleapis.com/prediction/online/response_latencies, and quota usage with alerts at 80% threshold.

Catching Deprecation and GA-Swap Errors Before They Hit

The nastiest production failures are the ones where you changed nothing and it broke anyway. The classic cause is a model you reference being deprecated, or a default being quietly swapped as it reaches GA. On 2026-06-25, for example, gemini-3.1-flash-image-preview and gemini-3-pro-image-preview shut down — any pipeline calling them by name returns NOT_FOUND that day. Even the 3.5 Flash GA can shift the default behind an alias, changing output granularity and cost.

The common mistake is hardcoding a model name and swallowing exceptions:

import google.generativeai as genai
 
genai.configure(api_key="YOUR_API_KEY")
 
# Hardcoded model name -- fragile to shutdowns and renames
model = genai.GenerativeModel("gemini-3.1-flash-image-preview")
 
def generate(prompt: str):
    try:
        return model.generate_content(prompt).text
    except Exception:
        # Swallows the error without knowing which layer failed
        return None

You cannot tell from the logs why it failed. In production, run a startup preflight that confirms the model exists, detects deprecation, and explicitly falls back to a successor — so a shutdown date never turns into a silent, total outage:

import logging
import google.generativeai as genai
from google.api_core.exceptions import NotFound
 
genai.configure(api_key="YOUR_API_KEY")
log = logging.getLogger("gemini.preflight")
 
# Deprecated model -> successor, with shutdown date
DEPRECATIONS = {
    "gemini-3.1-flash-image-preview": ("gemini-3.5-flash", "2026-06-25"),
    "gemini-3-pro-image-preview": ("gemini-3.5-flash", "2026-06-25"),
}
 
def resolve_model(requested: str) -> str:
    """Run once at startup. Detect shutdowns/renames and route to a successor."""
    target = requested
    if requested in DEPRECATIONS:
        successor, shutdown = DEPRECATIONS[requested]
        log.warning("model %s shuts down on %s; falling back to %s",
                    requested, shutdown, successor)
        target = successor
    try:
        genai.get_model(f"models/{target}")  # raises NotFound if missing
    except NotFound:
        log.error("successor %s not found either; check your config", target)
        raise
    return target
 
MODEL_NAME = resolve_model("gemini-3.1-flash-image-preview")
model = genai.GenerativeModel(MODEL_NAME)
 
def generate(prompt: str) -> str | None:
    try:
        resp = model.generate_content(prompt)
    except NotFound:
        log.error("model vanished at runtime; re-run preflight")
        raise
    used = getattr(resp, "model_version", MODEL_NAME)
    if used != MODEL_NAME:
        log.info("response model switched to %s (requested: %s)", used, MODEL_NAME)
    return resp.text

Three things matter. First, catch NOT_FOUND in the startup preflight and route to a successor via a fallback table. Second, log the actual model identifier the response reports and reconcile it against what you requested. Third, keep deprecated models in config with their shutdown dates so CI can warn you before the deadline arrives. I have been caught out on a shutdown date by a pipeline that leaned on a preview feature; since adding this preflight, missing a deprecation notice no longer means production silently goes dark.

Production Diagnosis Flow

When a Gemini API error occurs in production, follow this sequence.

Step 1: Identify the error type. PERMISSION_DENIED → check IAM roles, service accounts, and VPC-SC. RESOURCE_EXHAUSTED → check quota usage and add retry with backoff. SERVICE_UNAVAILABLE → check API enablement status and regional availability. DEADLINE_EXCEEDED → adjust timeout settings and implement multi-region failover.

Step 2: Check Cloud Audit Logs. Detailed error context is always logged there. VPC-SC errors in particular can be traced precisely using the vpcServiceControlsUniqueIdentifier.

Step 3: Apply least-privilege principle. Granting the Owner role to debug is tempting but dangerous. Start with roles/aiplatform.user and add only what's actually needed.

Once production Gemini API is properly configured, it operates reliably at scale. We hope this guide helps you build secure and trustworthy AI features with confidence.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.