Parallel Function Calling in Gemini API: Production Patterns, Pitfalls, and Monitoring

Handling Tool Dependencies: Think in DAGs

The hardest part of Parallel Function Calling is not on the model side, it's on the application side. As soon as tools depend on one another, you need to think in terms of a DAG, with some steps parallel and others sequential. Consider a travel booking agent:

• search_flights(dep, arr, date) — find candidate flights
• search_hotels(city, date) — find candidate hotels
• book_flight(flight_id) — depends on search_flights
• book_hotel(hotel_id) — depends on search_hotels
• send_itinerary(booking_ids) — depends on both bookings

The ideal flow is: turn 1 runs search_flights and search_hotels in parallel, turn 2 runs book_flight and book_hotel in parallel, turn 3 runs send_itinerary. If you leave ordering entirely up to the model it will sometimes deviate and cause failures. I prefer to bake the ordering into the tool descriptions themselves:

types.FunctionDeclaration(
    name="book_flight",
    description=(
        "Reserve a flight using a flight_id. "
        "Only call this after search_flights has produced a chosen flight_id. "
        "Safe to call in parallel with book_hotel."
    ),
    ...
)

Gemini 2.5 Pro is surprisingly reliable at following these rules when you spell them out. Before you add a new tool, try writing one sentence that answers "which other tools can safely run alongside this one?" If you can't, your tool design isn't finished yet.

Argument design that keeps parallelism safe

Beyond descriptions, the shape of arguments and side effects matters. These are the three rules I enforce:

• Do not allow concurrent writes to the same resource: running book_flight(flight_id) and cancel_flight(flight_id) for the same flight in parallel is a race waiting to happen. I sometimes use toolConfig.function_calling_config.allowed_function_names to temporarily forbid one of them while the other is active.
• Keep read-only tools parallel-ready: state explicitly in the description that the tool is safe to call concurrently.
• Collapse dependent calls into one tool: if tool B always needs tool A's result, expose a merged get_user_and_orders(user_id) instead. The model now sees it as a single parallelizable unit.

Designing for Partial Failure

Concurrency makes "two out of three succeeded" a first-class case. A naive implementation either crashes on the first failure or silently drops it and answers with incomplete data. In production I combine three patterns.

Pattern 1: Per-tool timeouts and retries

Each tool gets its own timeout and retry policy, and failures surface to the model as structured error payloads.

# pip install tenacity
from tenacity import retry, stop_after_attempt, wait_exponential_jitter
 
class ToolError(Exception): ...
 
@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(initial=0.2, max=2.0),
    reraise=True,
)
async def fetch_shipping_with_retry(zip: str) -> dict[str, Any]:
    try:
        return await asyncio.wait_for(fetch_shipping(zip), timeout=2.0)
    except asyncio.TimeoutError as exc:
        raise ToolError(f"fetch_shipping timeout for zip={zip}") from exc
 
async def safe_dispatch(fc: types.FunctionCall) -> types.Part:
    impl = RETRY_REGISTRY.get(fc.name) or TOOL_REGISTRY[fc.name]
    try:
        result = await impl(**(fc.args or {}))
    except Exception as exc:
        result = {
            "error": "tool_unavailable",
            "message": str(exc),
            "retry_allowed": False,
        }
    return types.Part.from_function_response(name=fc.name, response=result)

When you return {"error": ...} instead of raising, Gemini 2.5 Pro will compose answers like "We couldn't reach the shipping service, but your order shipped yesterday and there are 42 units in stock." That graceful degradation is the single biggest reason I return errors to the model instead of hiding them.

Pattern 2: Distinguish critical from optional tools

Not all tools are equally important. Classify them and let non-critical failures pass through.

CRITICAL_TOOLS = {"search_orders"}
OPTIONAL_TOOLS = {"get_inventory", "fetch_shipping"}
 
async def run_parallel_with_criticality(calls: list[types.FunctionCall]) -> list[types.Part]:
    results = await asyncio.gather(*(safe_dispatch(c) for c in calls))
    for c, r in zip(calls, results):
        if c.name in CRITICAL_TOOLS:
            resp = r.function_response.response if r.function_response else {}
            if "error" in resp:
                raise RuntimeError(f"critical tool {c.name} failed: {resp}")
    return results

Agreeing with your product team on what is critical vs. optional ahead of time makes SLAs and incident response much easier. It also lets you answer "should we apologize and escalate, or serve a partial answer?" with code, not a debate.

Pattern 3: Cached stale fallback

For high-read tools, a tiny cache can save you during upstream hiccups.

import time
 
_CACHE: dict[str, tuple[float, dict]] = {}
TTL_SECONDS = 60
 
async def get_inventory_cached(sku: str) -> dict[str, Any]:
    key = f"inv:{sku}"
    now = time.time()
    cached = _CACHE.get(key)
    try:
        fresh = await asyncio.wait_for(get_inventory(sku), timeout=1.5)
        _CACHE[key] = (now, fresh)
        return fresh
    except Exception:
        if cached and now - cached[0] < TTL_SECONDS * 10:
            return {**cached[1], "stale": True, "cached_at": cached[0]}
        raise

Passing stale: True back to the model lets Gemini explain the situation honestly: "Live inventory is temporarily unavailable, but a minute ago we had 42 units in stock." This kind of candor is what separates trustworthy agents from guessers.

Rate Limits and Cost Control

Parallelization helps latency, but it also concentrates load. I layer three controls on top.

Layer 1: Per-tool semaphores

Bound the concurrency per tool to avoid overwhelming fragile downstream services.

SEMAPHORES = {
    "fetch_shipping": asyncio.Semaphore(5),
    "get_inventory":  asyncio.Semaphore(20),
    "search_orders":  asyncio.Semaphore(20),
}
 
async def bounded_dispatch(fc: types.FunctionCall) -> types.Part:
    sem = SEMAPHORES.get(fc.name, asyncio.Semaphore(10))
    async with sem:
        return await safe_dispatch(fc)

Layer 2: Per-turn parallelism cap

Cap how many tools can run in one agent turn. This also protects against a runaway model emitting ten calls in one shot.

MAX_PARALLEL_PER_TURN = 5
 
async def run_bounded(calls: list[types.FunctionCall]) -> list[types.Part]:
    if len(calls) > MAX_PARALLEL_PER_TURN:
        return [
            types.Part.from_function_response(
                name=c.name,
                response={"error": "too_many_parallel_calls",
                          "limit": MAX_PARALLEL_PER_TURN},
            )
            for c in calls[MAX_PARALLEL_PER_TURN:]
        ] + [await safe_dispatch(c) for c in calls[:MAX_PARALLEL_PER_TURN]]
    return await asyncio.gather(*(safe_dispatch(c) for c in calls))

Layer 3: Budget metering

Parallel execution does not inflate input tokens directly, but large tool responses feed back into the next turn and can blow up your bill. Cap tool response sizes (e.g., summarize RAG results beyond 2 KB), and flip your agent from parallel to sequential when your per-user token budget nears its limit.

Three Bugs You Will Only See in Production

These are not in the docs. They come from owning parallel agents on-call.

Pitfall 1: Duplicate calls to the same tool

Models occasionally call the same tool twice with the same arguments, especially when a prompt contains words like "definitely" or "to be sure." For read-only tools this is merely wasteful. For billing or notifications it's a production incident.

Fix: deduplicate calls by (name, args) before dispatching.

def dedupe_calls(calls: list[types.FunctionCall]) -> list[types.FunctionCall]:
    seen: set[tuple] = set()
    uniq: list[types.FunctionCall] = []
    for c in calls:
        key = (c.name, tuple(sorted((c.args or {}).items())))
        if key not in seen:
            seen.add(key)
            uniq.append(c)
    return uniq

Pitfall 2: Result ordering scrambled

If you switch from asyncio.gather to asyncio.as_completed to process results as they finish, the ordering of response parts no longer matches the ordering of the original functionCall parts. Gemini uses the ordering as a strong signal for binding results back to calls; scramble it and you get hallucinated summaries.

Fix: keep asyncio.gather (or track indexes yourself) so that "call index N ↔ response index N" holds.

Pitfall 3: Forced-mode kills parallelism

Setting tool_config.function_calling_config.mode = ANY with a single allowed_function_names is a great debugging tool, but it disables parallelism. If you flipped on forced mode during development and noticed your agent got slower, that's expected.

Fix: use forced mode only during the "first turn onboarding" of an agent, and switch back to AUTO for subsequent turns. I call this "gripping the wheel during takeoff" and let the agent fly itself once it's airborne.

Worked Example: A Financial Report Agent

To ground all of this, here's the design of a financial report agent I helped build. It answers user questions by pulling from an internal P&L DB, BigQuery, a news API, and internal RAG, then optionally drafting an email.

Tool inventory, tagged by side effect and parallelism:

• fetch_pl(period) — read-only, parallel-safe
• run_bq_query(sql) — read-only, parallel-safe (but costly — rate-limited)
• search_news(keywords) — read-only, parallel-safe with semaphore
• search_internal_docs(query) — read-only, parallel-safe
• draft_email(recipient, body) — side-effecting, sequential only

Turn 1 runs the four read tools in parallel. Turn 2 produces an answer. Turn 3 optionally drafts the email. Measured latency:

• Sequential: ~6.8 s (each read tool 1.5–2.0 s)
• Parallel: ~2.1 s (bounded by the slowest tool plus model overhead)

That is the "it feels like a product, not a demo" threshold. Users stop noticing the wait.

If you are building something in this space, I'd pair this with the Gemini 3 Pro × LangGraph data-analysis agent guide, which goes deeper on parallel node execution inside LangGraph. And if you are still shaking out the basics of tool calls, the Gemini API Function Calling complete guide is a good warm-up.

Doing the Same Thing in TypeScript

Not every team ships Python. The @google/genai Node.js SDK has the same primitives, and the same parallel patterns translate cleanly to a TypeScript backend.

// npm install @google/genai p-limit p-retry
import { GoogleGenAI, Type, type FunctionCall, type Part } from "@google/genai";
import pLimit from "p-limit";
import pRetry from "p-retry";
 
const client = new GoogleGenAI({ apiKey: process.env.YOUR_GEMINI_API_KEY! });
 
const tools = [{
  functionDeclarations: [
    {
      name: "search_orders",
      description:
        "Look up orders for a user. Safe to run in parallel with get_inventory and fetch_shipping.",
      parameters: {
        type: Type.OBJECT,
        properties: { user_id: { type: Type.STRING } },
        required: ["user_id"],
      },
    },
    {
      name: "get_inventory",
      description:
        "Return stock for a SKU. Safe to run in parallel with search_orders and fetch_shipping.",
      parameters: {
        type: Type.OBJECT,
        properties: { sku: { type: Type.STRING } },
        required: ["sku"],
      },
    },
    {
      name: "fetch_shipping",
      description:
        "Return shipping ETA. Safe to run in parallel with search_orders and get_inventory.",
      parameters: {
        type: Type.OBJECT,
        properties: { zip: { type: Type.STRING } },
        required: ["zip"],
      },
    },
  ],
}];
 
type Impl = (args: Record<string, string>) => Promise<Record<string, unknown>>;
const impls: Record<string, Impl> = {
  search_orders: async ({ user_id }) => ({
    orders: [{ id: "o_9001", sku: "ABC-001", status: "shipped", user_id }],
  }),
  get_inventory: async ({ sku }) => ({ sku, stock: 42 }),
  fetch_shipping: async ({ zip }) => ({ zip, carriers: ["JP Post"], eta_days: 2 }),
};
 
// Per-tool concurrency caps map cleanly to p-limit instances.
const limits: Record<string, ReturnType<typeof pLimit>> = {
  fetch_shipping: pLimit(5),
  get_inventory: pLimit(20),
  search_orders: pLimit(20),
};
 
async function dispatch(call: FunctionCall): Promise<Part> {
  const limit = limits[call.name] ?? pLimit(10);
  return limit(async () => {
    try {
      const impl = impls[call.name];
      if (!impl) throw new Error(`unknown tool: ${call.name}`);
      const result = await pRetry(() => impl(call.args as Record<string, string>), {
        retries: 2,
        factor: 2,
        minTimeout: 200,
      });
      return { functionResponse: { name: call.name, response: result } };
    } catch (err) {
      return {
        functionResponse: {
          name: call.name,
          response: { error: "tool_unavailable", message: String(err) },
        },
      };
    }
  });
}
 
export async function runAgent(userMessage: string): Promise<string> {
  const history: { role: "user" | "model"; parts: Part[] }[] = [
    { role: "user", parts: [{ text: userMessage }] },
  ];
  for (let turn = 0; turn < 4; turn++) {
    const resp = await client.models.generateContent({
      model: "gemini-2.5-pro",
      contents: history,
      config: { tools, temperature: 0.2 },
    });
    const parts = resp.candidates?.[0]?.content?.parts ?? [];
    const calls = parts
      .map((p) => (p as { functionCall?: FunctionCall }).functionCall)
      .filter((c): c is FunctionCall => Boolean(c));
    if (calls.length === 0) {
      return parts.map((p) => (p as { text?: string }).text ?? "").join("");
    }
    history.push({ role: "model", parts });
    const toolParts = await Promise.all(calls.map(dispatch));
    history.push({ role: "user", parts: toolParts });
  }
  return "Conversation looped without convergence; escalating.";
}

Two small notes for Node users. First, Promise.all preserves ordering the same way asyncio.gather does, so the "pitfall #2" warning about result ordering applies identically — never migrate to Promise.race or Promise.allSettled without re-binding indexes. Second, if you deploy to Cloudflare Workers, be aware that aggressive parallel fan-out can hit the subrequest limit; batch accordingly or move the tool calls to a durable object.

Testing Parallel Tools Without Burning Tokens

One thing I learned the hard way: you cannot reliably test parallel agents by calling the real Gemini API every time. Responses are non-deterministic, and you will burn your monthly quota on CI. What has worked for me is a two-layer test strategy.

Layer A: unit-test the dispatcher

Feed fake FunctionCall objects into your dispatcher and assert the behavior.

import pytest
import asyncio
from agent import run_parallel_tools, TOOL_REGISTRY
from google.genai import types
 
@pytest.mark.asyncio
async def test_one_tool_failure_does_not_cancel_others(monkeypatch):
    async def failing(**_): raise RuntimeError("boom")
    monkeypatch.setitem(TOOL_REGISTRY, "fetch_shipping", failing)
 
    calls = [
        types.FunctionCall(name="search_orders", args={"user_id": "u_1"}),
        types.FunctionCall(name="get_inventory", args={"sku": "A"}),
        types.FunctionCall(name="fetch_shipping", args={"zip": "150-0001"}),
    ]
    results = await run_parallel_tools(calls)
    assert len(results) == 3
    errors = [r.function_response.response for r in results if
              "error" in r.function_response.response]
    assert len(errors) == 1

This is the test I have gotten the most value from. It takes one failure and proves the other two calls still complete.

Layer B: golden-trace replay

Record a handful of real Gemini responses (as JSON fixtures), then replay them through the agent loop with the live model client mocked. This lets you change tool code freely while keeping the model-side logic pinned.

@pytest.mark.asyncio
async def test_agent_uses_all_three_tools_in_first_turn(gemini_fake):
    gemini_fake.queue_response(fixture="triple_parallel_call.json")
    gemini_fake.queue_response(fixture="final_answer.json")
    reply = await agent_turn("Status of my order, please")
    assert "shipped" in reply
    assert gemini_fake.call_count == 2   # 1 tool turn, 1 answer turn

A pair of fixtures per agent flow is usually enough. The goal is not to test the model — it's to test that your dispatcher uses the model's output correctly.

Composing Streaming Answers with Parallel Tool Calls

One question I get after every talk I give on this topic: can we stream the final answer to the user while tools are still executing in parallel? The short answer is yes, with a caveat. Gemini needs all tool results back before it can compose the final response, but you can stream the response once it starts. The user-visible effect is:

1. User sends a message.
2. Tools fan out in parallel (400–800 ms of silence on the UI).
3. The model starts streaming the answer word-by-word.

That 400–800 ms pause is a fine place to render a small "Looking things up…" indicator. I advise against faking token streaming during the tool phase; users notice the trick once the real stream arrives at a different cadence. Be honest: "Checking three systems…" followed by a real stream feels more professional than a synthesized fake stream.

A practical pattern I use with server-sent events:

async def sse_agent(user_message: str):
    tools = declare_tools()
    history = [types.Content(role="user", parts=[types.Part.from_text(user_message)])]
 
    yield "event: status\ndata: searching\n\n"
    for _ in range(4):
        resp = client.models.generate_content(
            model="gemini-2.5-pro",
            contents=history,
            config=types.GenerateContentConfig(tools=tools),
        )
        parts = resp.candidates[0].content.parts
        fcalls = [p.function_call for p in parts if p.function_call]
        if fcalls:
            history.append(types.Content(role="model", parts=parts))
            # While tools run in parallel, keep the connection alive
            yield f"event: status\ndata: calling_{len(fcalls)}_tools\n\n"
            tool_parts = await run_parallel_tools(fcalls)
            history.append(types.Content(role="user", parts=tool_parts))
            continue
 
        # Final turn: switch to streaming
        stream = client.models.generate_content_stream(
            model="gemini-2.5-pro",
            contents=history,
            config=types.GenerateContentConfig(tools=tools),
        )
        for chunk in stream:
            for p in chunk.candidates[0].content.parts:
                if p.text:
                    yield f"event: token\ndata: {p.text}\n\n"
        yield "event: done\ndata: ok\n\n"
        return

The two separate model calls (one non-streaming while tools run, then a streaming one for the answer) feel slightly wasteful, but in practice the first call is short because it only produces functionCall parts, and the streaming payoff on the second call is worth it.

Monitoring: What to Watch After You Parallelize

Once your agents run tools in parallel, the median latency hides a lot. I always instrument these five signals:

• Turn-count distribution — fraction of conversations resolved in one turn vs. three or more
• Parallelism per turn — average tools per turn (>2 means parallelism is actually kicking in)
• Per-tool error rate — fraction of function_response parts carrying error
• Per-tool p95 latency — so you know if your semaphores or cache are effective
• Stale fallback rate — how often cached stale values were returned

Feed these into the OpenTelemetry setup described in the Gemini API observability and production monitoring guide. That lets you tell at a glance whether the bottleneck is Gemini or your own infrastructure.

Closing

Give yourself one hour this week. Pick an agent, rewrite the tool dispatcher to use asyncio.gather, and add "safe to call in parallel with X" to each tool's description. The moment your latency drops in half is genuinely fun. From there, the two rules you must never break are: let the model see the errors you recover from, and keep write-side effects out of the parallel batch.

I've come to think of parallelization less as a performance optimization and more as an agent-design discipline. Tool independence, idempotency, and side-effect containment — the questions distributed systems engineers have been chewing on for decades — turn out to be exactly the right vocabulary for building reliable LLM applications too.

If you build agents long enough, you notice that "the model is slow" is almost never the bottleneck. It is your orchestration layer deciding what to do next, and whether to wait for three things or one. Parallel Function Calling is the feature that most rewards careful engineering on that layer. The payoff shows up in every conversation your users have, forever, in the form of shorter waits and answers that feel considered rather than guessed. That compounding effect is why I keep coming back to this as the first thing I optimize in any new agent.

Try it on one agent this week. Measure the median latency before and after. Write down the two or three things that surprised you. That small experiment has a way of reshaping how you think about everything else you build on top of the Gemini API.