Gemini × DSPy: Retire from Prompt Craftsmanship — Automated Prompt Optimization

Separate Signatures from Modules

What's elegant about DSPy's design is the clean split between Signature (what goes in and out) and Module (how to reason). You can swap Modules on the same Signature, so "start with Predict, upgrade to ChainOfThought if accuracy isn't enough" becomes a trivial experiment.

# signature_and_module.py — same Signature, different Modules
import dspy
 
class ClassifyInquiry(dspy.Signature):
    """Classify user inquiries into one of four categories: payment / bug / feature / other."""
    inquiry: str = dspy.InputField(desc="The user's inquiry")
    category: str = dspy.OutputField(
        desc="One of: payment / bug / feature / other"
    )
 
# 1) Predict: straightforward single-shot answer. Fast and cheap.
quick_classifier = dspy.Predict(ClassifyInquiry)
 
# 2) ChainOfThought: emits reasoning steps. Helps on ambiguous cases.
thoughtful_classifier = dspy.ChainOfThought(ClassifyInquiry)
 
sample = "Since last week's update, my iPhone app crashes when I open the settings screen."
 
print(quick_classifier(inquiry=sample).category)
# Expected: "bug"
 
out = thoughtful_classifier(inquiry=sample)
print(out.reasoning)
# Expected: "The word 'crashes' strongly suggests a defect, not a feature request..."
print(out.category)
# Expected: "bug"

In practice, I run Flash with Predict first and only fall back to ChainOfThought when Predict is confused. Applying ChainOfThought to every request boosts accuracy but inflates output tokens by 3–5×, which hits the bill directly. I once misread this tradeoff and quadrupled the token bill on the first production rollout.

The clean way to express this two-stage strategy in DSPy is a custom class inheriting from dspy.Module, holding both classifiers and routing based on confidence. You can, for instance, read the output token probabilities from quick_classifier (available on Gemini 2.5 Pro and newer) and call thoughtful_classifier only when confidence drops below a threshold. The key benefit: submodules of a Module are all optimizable. Compiling the parent trains both classifiers together — something you simply cannot replicate with hand-written prompts.

This compositional property is what makes DSPy feel like PyTorch. You can build a retrieval-augmented classifier by composing a retriever Module with a classifier Module and a verification Module. Each submodule has its own Signature and can be optimized independently or jointly. I've built pipelines with five or six modules chained together, and the beauty is that the top-level compile handles all of them without any special plumbing from me. Contrast that with hand-written prompts, where coordinating few-shot examples across stages quickly becomes a maintenance nightmare.

Move to "I No Longer Write Prompts" with Optimizers

Here's where DSPy really earns its keep. An Optimizer takes a small labeled dataset and automatically searches for the best prompt wording and few-shot examples. For combinatorial searches, a machine will outrun any human.

Two optimizers cover most situations:

• BootstrapFewShot: extracts a handful of successful examples from your training data and bakes them into a few-shot prompt. Cheap and fast.
• MIPROv2: simultaneously optimizes the prompt wording and the few-shot exemplars. More expensive, but the highest-quality option.

Here's a complete example of BootstrapFewShot applied to ClassifyInquiry.

# optimize_classifier.py — auto-tune with just 20 labeled examples
import dspy
from dspy.teleprompt import BootstrapFewShot
 
# Set GEMINI_API_KEY in your environment beforehand.
dspy.configure(
    lm=dspy.LM(model="gemini/gemini-2.5-flash", temperature=0.0, max_tokens=256)
)
 
class ClassifyInquiry(dspy.Signature):
    """Classify user inquiries into payment / bug / feature / other."""
    inquiry: str = dspy.InputField()
    category: str = dspy.OutputField(desc="One of: payment, bug, feature, other")
 
# Training examples (50–200 is more stable in production)
trainset = [
    dspy.Example(inquiry="My credit card was charged twice for the same order.", category="payment").with_inputs("inquiry"),
    dspy.Example(inquiry="Please add a dark mode.", category="feature").with_inputs("inquiry"),
    dspy.Example(inquiry="The app crashes right after launch.", category="bug").with_inputs("inquiry"),
    dspy.Example(inquiry="How do I cancel my account?", category="other").with_inputs("inquiry"),
    # ...(more realistically 20+ examples)
]
 
# Metric: 1 if the predicted category matches, else 0
def exact_match(example, pred, trace=None):
    return pred.category.strip().lower() == example.category.strip().lower()
 
student = dspy.Predict(ClassifyInquiry)
optimizer = BootstrapFewShot(
    metric=exact_match,
    max_bootstrapped_demos=4,   # number of successful examples embedded in the prompt
    max_labeled_demos=16,       # maximum labeled examples considered
    max_rounds=1,
)
compiled = optimizer.compile(student=student, trainset=trainset)
 
# Persist and reuse (DSPy saves JSON)
compiled.save("compiled_classifier.json")
 
# Load and infer
loaded = dspy.Predict(ClassifyInquiry)
loaded.load("compiled_classifier.json")
print(loaded(inquiry="I'd like to reactivate my Premium plan.").category)
# Expected: "payment" (billing-related inquiry)

What compile actually does isn't obvious on first read. In short, it runs the student against your training set and selects only examples where exact_match returns 1 to embed as few-shot exemplars. By definition, every retained example is one the Signature solved correctly, so the few-shot set is intrinsically compatible with the prompt DSPy is building. That's fundamentally different from hand-curated few-shot.

Switching to a stronger optimizer only requires swapping the class name.

from dspy.teleprompt import MIPROv2
 
optimizer = MIPROv2(
    metric=exact_match,
    auto="light",           # "light" / "medium" / "heavy" for cost control
    num_threads=4,
)
compiled = optimizer.compile(student=student, trainset=trainset, requires_permission_to_run=False)

MIPROv2 searches over both the instruction text and the few-shot exemplars in parallel, which typically yields better results than BootstrapFewShot — but at the cost of hundreds of Gemini calls. Even auto="light" triggers 50–200 API calls. A practical setup is to optimize on Flash during development and run production inference on Pro.

When choosing between the two, I use this rule of thumb: start with BootstrapFewShot. If you end up within 5 points of your target accuracy, ship it. Only reach for MIPROv2 if that gap stays open. Running MIPROv2 on every task gets expensive quickly — it can shift your monthly API bill by tens of thousands of yen. Treat optimizer choice as a function of task importance.

Something that isn't obvious from the docs: DSPy optimizers are stateful when it comes to the student argument. If you hand the same Module instance to two consecutive compile calls, the second run builds on the first's progress. That's useful when you want to extend a compiled program with a larger dataset without discarding prior work, but surprising if you expect fresh state. My recommendation is to always instantiate a fresh student = dspy.Predict(Signature) before each compile, unless you explicitly want the accumulation. It avoids hard-to-debug cases where today's "baseline" is secretly benefiting from yesterday's optimization.

Roll Your Own LLM-as-a-Judge Metric

Not every task can be graded by exact match. For summarization, translation, FAQ responses, and other open-ended tasks, LLM-as-a-Judge is the go-to pattern — and it's just another DSPy Signature.

# llm_judge.py — a Gemini-based judge for summary quality
import dspy
 
class SummaryJudge(dspy.Signature):
    """Compare a source article against its summary and score the overall quality 1–5."""
    article: str = dspy.InputField(desc="Original article text")
    summary: str = dspy.InputField(desc="Summary to evaluate")
    rationale: str = dspy.OutputField(desc="Reasoning for the score (under 100 words)")
    score: int = dspy.OutputField(
        desc="Integer 1–5 where 5 is best and 1 is worst."
    )
 
# Use Pro for the judge — consistency matters more than throughput
judge_lm = dspy.LM(model="gemini/gemini-2.5-pro", temperature=0.0, max_tokens=512)
judge = dspy.Predict(SummaryJudge)
 
def summary_metric(example, pred, trace=None):
    with dspy.context(lm=judge_lm):
        result = judge(article=example.article, summary=pred.summary)
    try:
        score = int(result.score)
    except (TypeError, ValueError):
        return 0.0
    # Return a continuous 0–1 value to the optimizer
    return max(0.0, min(1.0, (score - 1) / 4.0))

The trick is dspy.context(lm=judge_lm): it lets you swap in a stronger model for the judge only. While Flash grinds through hundreds of candidates, the judge running on Pro stays stable. dspy.context temporarily overrides the global LM — it reverts at the end of the with block.

Always sanity-check that your judge isn't biased in some specific direction (e.g., does it systematically favor longer summaries?). I pair 100 human-rated samples against the judge's scores, and I consider the pair production-ready if the correlation exceeds 0.7.

One strong suggestion: log the judge's rationale field alongside the score. Pipe them into BigQuery or a similar store, then spot-check weekly. I've caught judges that were implicitly scoring "length" rather than "quality" this way — it's much harder to notice from scores alone.

Also, make the grading rubric explicit in the judge's docstring. "Score overall quality 1–5" lets Gemini be generous. Something like "If the summary contains information not present in the article, max score is 2. Deduct one point per grammatical error." immediately grounds the scores in something usable. I dive deeper into judge prompt design in A Production System for Gemini API Prompt Template Management, which will shorten your iteration time when you put a judge in production.

As a rule of thumb, a single judge is less stable long-term than an ensemble of three judges evaluating different axes (accuracy, style, conciseness) combined via a weighted average. The reason is simple: any single judge's biases cancel out across the ensemble. In DSPy, this is just a metric function that calls several judges and averages the scores — almost no additional code. "Judge ensembling" has become a standard pattern in my pipelines.

Case Study: Training a Support Classifier on Two Weeks of Real Logs

Let's assemble the pieces into a realistic pipeline. Say you have 500 support logs from the last two weeks and you want to improve classification accuracy from 87% to 95%.

# production_pipeline.py — optimize on real data, evaluate on a holdout
import json
import random
import dspy
from dspy.teleprompt import MIPROv2
 
random.seed(42)
 
dspy.configure(
    lm=dspy.LM(model="gemini/gemini-2.5-flash", temperature=0.0, max_tokens=256)
)
 
class ClassifyInquiry(dspy.Signature):
    """Classify user inquiries into payment / bug / feature / other."""
    inquiry: str = dspy.InputField()
    category: str = dspy.OutputField(desc="One of: payment, bug, feature, other")
 
# Load 500 labeled logs
with open("support_logs.json", encoding="utf-8") as f:
    raw = json.load(f)
 
examples = [
    dspy.Example(inquiry=r["text"], category=r["label"]).with_inputs("inquiry")
    for r in raw
]
random.shuffle(examples)
 
# 70 / 20 / 10 split (train / val / test)
n = len(examples)
trainset = examples[: int(n * 0.7)]
valset = examples[int(n * 0.7) : int(n * 0.9)]
testset = examples[int(n * 0.9) :]
 
def exact_match(example, pred, trace=None):
    return pred.category.strip().lower() == example.category.strip().lower()
 
# Baseline
baseline = dspy.Predict(ClassifyInquiry)
baseline_acc = sum(1 for e in testset if exact_match(e, baseline(inquiry=e.inquiry))) / len(testset)
print(f"Baseline accuracy: {baseline_acc:.3f}")
 
# Optimize
optimizer = MIPROv2(metric=exact_match, auto="light", num_threads=4)
compiled = optimizer.compile(
    student=dspy.Predict(ClassifyInquiry),
    trainset=trainset,
    valset=valset,
    requires_permission_to_run=False,
)
 
# Post-optimization accuracy
optimized_acc = sum(
    1 for e in testset if exact_match(e, compiled(inquiry=e.inquiry))
) / len(testset)
print(f"Optimized accuracy: {optimized_acc:.3f}")
 
compiled.save("inquiry_classifier_v1.json")
# Example output:
# Baseline accuracy: 0.874
# Optimized accuracy: 0.954

One run of this script typically produces 7+ percentage points of improvement. What matters is that the improvement isn't "I polished prompts by hand" — it's "I automatically searched the space given the data." As the dataset grows, you just rerun compile to ratchet quality upward.

The overlooked hazard with real data is label quality. In a past project, I found that human-applied labels had a 6% error rate, and that's what was destabilizing MIPROv2. Mislabeled examples get selected as few-shot exemplars, and the optimizer starts faithfully reproducing the wrong answers. At minimum, carry a confidence score on each label and filter out examples below a threshold. No framework can rescue dirty data.

One more detail from the trenches: class imbalance can quietly break your metric. If 80% of the training set is bug and 20% is split across the others, exact_match will lead the optimizer toward a prompt that plays it safe by predicting bug more often. Accuracy will look fine in aggregate while recall on the minority classes silently tanks. The fix is either rebalancing the training set or using a per-class metric like macro-F1 inside your metric function. I default to macro-F1 for any classification task with more than two classes, and the optimizer output is visibly healthier across the confusion matrix.

Common Pitfalls and How to Handle Them

Five concrete pitfalls I've walked into.

Pitfall 1: The optimizer dies mid-run and you've burned tokens for nothing

MIPROv2 issues dozens to hundreds of inferences. If you hit 429 Resource Exhausted, you lose all progress to that point. The fix: don't rely on the free tier for optimization. Provision a separate billing project and point GEMINI_API_KEY at it. Also, high num_threads will trip rate limits instantly. num_threads=4 is a safe starting point.

Pitfall 2: A too-lax metric leaves you unsure what you actually optimized

Something like return 1.0 if some_keyword in pred.output else 0.0 will technically "optimize," but it's optimizing your stated objective, not the user satisfaction you actually care about. Compose the metric from multiple criteria (exact match + length constraint + forbidden-word check) and validate it against at least 100 hand-checked examples.

Pitfall 3: Changing the Signature breaks saved examples

The few-shot exemplars saved in JSON are tied to the Signature they were produced against. Rename a field or change a type, and loading either errors out or silently discards the examples. If you change the Signature, compile again. Version saved files explicitly (compiled_classifier_v2.json) so you never confuse generations.

There's also a Gemini-specific quirk: LiteLLM's JSON-validation pass for Structured Output is a bit loose. I've seen a Signature with score: int come back as "4.0" — which then explodes on int(). The defense is either a two-stage cast in the metric (int(float(x))) or an explicit docstring note that integers (not decimals) are required. I cover this pattern in Gemini API Prompt Regression Testing with Pytest, which also lays out the broader testing approach before going to production.

Pitfall 4: On Japanese data, few-shot ordering matters more than you'd expect

The English benchmark folklore says few-shot order is a minor factor. Japanese data tells a different story. Running the same four exemplars in three orderings (by category, random, by ascending difficulty) gave me 92% / 94% / 96% respectively. DSPy randomizes ordering by default, but raising max_bootstrapped_demos too high amplifies ordering sensitivity. Empirically, 3–5 exemplars works best to reduce order dependence.

Pitfall 5: Gemini's safety filter blocks few-shot examples

If your trainset happens to contain sharply negative language, Gemini's safety filter may block those few-shot exemplars, and the Optimizer will register those as "failures." Important patterns end up under-represented. Relax safety_settings to BLOCK_ONLY_HIGH and pre-filter the trainset to strip extreme wording. I maintain an NG-word list as a preprocessing step. The complete treatment of safety-filter behavior is in Fixing Blocked Responses from the Gemini API Safety Filter.

Cost Reality — What DSPy on Gemini Actually Bills You

DSPy's optimizers call the API more than you'd think. Before adopting it, you should know the real numbers. These figures are from my own measurements against April-2026 Gemini API pricing (USD).

For BootstrapFewShot on 50 training examples, internal inferences land between 70 and 120 calls. Flash's low input price keeps a single compile run well under a dollar. Running it daily costs pocket change.

MIPROv2 at auto="medium" triggers 300–600 calls for that same dataset, plus Pro-based meta-inferences for prompt generation. A single compile runs anywhere from a few dollars to low double digits. auto="heavy" has cost me over $30 for some tasks. Monthly re-optimization is fine; running it repeatedly during development is where costs get noticeable fast.

The practical playbook: use a small dataset (20–30 examples) for fast iteration during development, and only run auto="medium" on the full dataset once before deploying. Gemini supports batch inference, and enabling it via LiteLLM can reduce cost by roughly 50%. DSPy doesn't yet expose a stable batching API, but bumping num_threads for parallelism is usually enough.

If you need another order of magnitude off the bill, use Gemini 2.5 Flash Lite as the student model during compile. Ship Flash or Pro for production, but let the optimizer loop run on Lite. Tiering Gemini models this way is the key to cost-efficient DSPy in practice.

One more lever: cache the Optimizer's intermediate results. DSPy uses diskcache under the hood if you enable it, so repeated runs of the same (Signature, trainset) pair can reuse earlier inferences. This is invaluable when you're iterating on the metric function and rerunning compile with only the metric changed. I've seen 3–5× speedups on iterative runs with the cache enabled, and the cost savings line up roughly with the speedup.

Three Tricks for Production

Making it work locally is not the same as keeping it running in production. Three practices I actually use.

Trick 1: Version-control compiled_*.json

DSPy's save() emits plain JSON. Prompt text and few-shot exemplars are both human-readable, which means they're reviewable. Put them under prompts/, run them through your normal PR flow, and track "who optimized what against which data" in commit history. When production accuracy degrades, rollback is one Git command away.

Going further, configure CI to auto-comment the before/after accuracy delta on every compiled_*.json PR. GitHub Actions can run pytest --benchmark against testset on both the old and new versions, label the PR for auto-merge if the delta is under 1%, and kick it to human review otherwise. Once prompt changes show up as hard numbers, you can re-optimize often without fear.

Trick 2: Separate the optimization model from the inference model

As noted earlier, optimize on Flash and infer on Pro — the cost/quality balance is much better. DSPy works fine when the optimization and inference models differ, but accuracy will shift slightly, so always re-evaluate on your production model. My workflow finishes by running testset on Pro and only deploying if Pro also clears the accuracy bar. The gap between Flash-optimized-tested-on-Flash and Flash-optimized-tested-on-Pro is usually a point or two in either direction, so leave that headroom in your accuracy targets.

Trick 3: Re-compile monthly on fresh data

User phrasing drifts over time. An optimization from two weeks ago might not be optimal three months from now. I keep a workflow_dispatch GitHub Actions job that reruns monthly and pings Slack when accuracy drops below a threshold (say, 93%). Writing prompts becomes monitoring prompts.

To make this sustainable, the monthly job needs a fresh labeled dataset each time. I automate this by sampling 5% of production traffic daily, sending it to a lightweight labeling UI (I use Streamlit; anything works), and appending confirmed labels to training_data/. Over a month, that accumulates a few hundred new labeled examples — enough for a meaningful re-optimization. The loop is: production sees the world change, you label a small slice of new examples, the monthly job bakes them in. Once this is wired up, prompt quality self-heals. You mostly notice DSPy when something else in the pipeline breaks, not when it's working.

The directory layout I've settled on: app/ for production code, prompts/ for versioned compiled_*.json, training_data/ for labeled datasets, and .github/workflows/ for the monthly re-optimization job. DSPy's own code footprint is small — operational quality hinges on how cleanly this metadata is managed. If your team has a code-review culture, assigning dedicated reviewers to prompts/ changes adds a lot of stability.

For the broader production architecture that surrounds this setup, Gemini API Production Architecture Patterns 2026 covers the full picture. Reading it before you roll DSPy out to a team makes design decisions faster.

A Recommended Adoption Order

For readers thinking "I want this in my project," here's the order I'd suggest. Get this order wrong and people bounce off DSPy before they feel the benefit.

1. Pick the one existing prompt that causes the most rework. Summarization or classification is fine. Scope small; pick something with a clear evaluation axis.
2. Hand-label 30–50 evaluation examples. This is human work. Cutting corners here caps the quality of every later step.
3. Port the existing prompt into a Signature and measure baseline accuracy. Plain Predict, no few-shot, no optimization.
4. Run BootstrapFewShot once. At this point, you'll almost always see a multi-point improvement.
5. If you're still below target, switch to MIPROv2. If even that doesn't close the gap, the Signature design or the metric function is likely the problem.
6. Deploy, then set up a monthly re-optimization job. That's how you graduate from "writing prompts forever" to "monitoring prompts."

Run this six-step loop for a week and you'll feel DSPy's value first-hand. Once you experience it, you'll want to migrate your other prompts too. Eventually, your entire product's prompt layer lives inside compiled_*.json files, and "the person who writes prompts" becomes "the person who writes metric functions." That's a productivity step change that's hard to ignore — whether you're an indie developer or on a team.

One parting suggestion: when choosing your first target, pick tasks where evaluation is easy to quantify. Classification, extraction, and normalization give you quick wins and show the value clearly. Leave creative summarization and long-form generation for later — designing good metrics for those is the real challenge, and most people bail out at that step if they start there.

The biggest thing DSPy changed for me was escaping the endless polish loop of "I bet I can get another two points if I tweak this prompt a bit more." With a metric function and a small dataset in place, the automation handles the search, and you can spend your attention on what you're actually measuring. If you're currently spending weekends fine-tuning a production Gemini prompt, label 50 examples today and run BootstrapFewShot once. The moment you feel it working, the way you allocate your development time will never look the same again.

Concretely, here's the smallest commitment I'd ask you to make after closing this page. Pick one Gemini-backed feature in your product, open a new branch, install DSPy, and port that feature's prompt into a Signature. Don't optimize yet — just get it running. Measure baseline accuracy with 30 hand-labeled examples you gather in an afternoon. That baseline measurement alone will tell you more about your current prompt's real performance than a month of vibes-based tweaking. From there, running BootstrapFewShot is a 30-minute investment and a small spend on the API. If nothing else, you'll have a number to compare against — which is more than most of us had a year ago. Everything I've described above followed from that first baseline measurement. The rest is just repetition.