Gemini × DSPy: Retire from Prompt Craftsmanship — Automated Prompt Optimization

Keep Signature and Module apart

The best design decision in DSPy is separating the Signature (what goes in, what comes out) from the Module (how the answer is produced). You can write a task with Predict and swap in ChainOfThought later without touching the Signature.

# signature_and_module.py — one Signature, two Modules
import dspy
 
class ClassifyInquiry(dspy.Signature):
    """Classify a user inquiry as payment / bug / feature / other."""
    inquiry: str = dspy.InputField(desc="The user's message")
    category: str = dspy.OutputField(desc="One of: payment / bug / feature / other")
 
quick_classifier = dspy.Predict(ClassifyInquiry)            # one shot: fast and cheap
thoughtful_classifier = dspy.ChainOfThought(ClassifyInquiry)  # adds reasoning
 
sample = "Since last week's update, the app crashes when I open Settings."
 
print(quick_classifier(inquiry=sample).category)   # e.g. "bug"
 
out = thoughtful_classifier(inquiry=sample)
print(out.reasoning)   # older versions call this field "rationale"
print(out.category)

In practice I run the cheap Predict first and escalate only the doubtful cases to ChainOfThought. Running everything through chain-of-thought buys accuracy, but output tokens multiply, and the bill goes with them. I misjudged that trade-off on my first production rollout and my stomach dropped when the invoice arrived.

The two-stage version is a small dspy.Module. For the escalation rule, I avoid values that may not be available (like token probabilities) and use a check that always is: did the model return something outside the allowed set?

# two_stage.py — escalate only when the first answer is invalid
import dspy
 
ALLOWED = {"payment", "bug", "feature", "other"}
 
class TwoStageClassifier(dspy.Module):
    def __init__(self):
        super().__init__()
        self.quick = dspy.Predict(ClassifyInquiry)
        self.careful = dspy.ChainOfThought(ClassifyInquiry)
 
    def forward(self, inquiry: str):
        first = self.quick(inquiry=inquiry)
        if first.category.strip().lower() in ALLOWED:
            return first
        return self.careful(inquiry=inquiry)

What matters is that every sub-module inside a Module becomes an optimization target. Compile the parent and both quick and careful get tuned together — something hand-written prompts can't give you.

Let the Optimizer replace the prompt writing

This is the core of DSPy. An Optimizer takes a small labeled set and searches for instructions and few-shot examples. For combinatorial search, machines are far faster than people.

Two optimizers cover most needs:

• BootstrapFewShot: keeps examples the program solved correctly and embeds them as few-shot demos — cheap and fast
• MIPROv2: searches instructions and demos together — more calls, more headroom

First, BootstrapFewShot on ClassifyInquiry:

# optimize_classifier.py — optimize with a small labeled set
import os
import dspy
from dotenv import load_dotenv
from dspy.teleprompt import BootstrapFewShot
 
load_dotenv()
dspy.configure(
    lm=dspy.LM(
        model=os.environ["GEMINI_FAST_MODEL"],
        api_key=os.environ["GEMINI_API_KEY"],
        temperature=0.0,
        max_tokens=256,
    )
)
 
class ClassifyInquiry(dspy.Signature):
    """Classify a user inquiry as payment / bug / feature / other."""
    inquiry: str = dspy.InputField()
    category: str = dspy.OutputField(desc="payment, bug, feature, or other")
 
# Real use wants dozens to a couple hundred examples
trainset = [
    dspy.Example(inquiry="I was charged twice on my card.", category="payment").with_inputs("inquiry"),
    dspy.Example(inquiry="Please add a dark mode.", category="feature").with_inputs("inquiry"),
    dspy.Example(inquiry="The app dies right after launch.", category="bug").with_inputs("inquiry"),
    dspy.Example(inquiry="How do I close my account?", category="other").with_inputs("inquiry"),
    # ... (truncated)
]
 
def exact_match(example, pred, trace=None):
    return pred.category.strip().lower() == example.category.strip().lower()
 
student = dspy.Predict(ClassifyInquiry)
optimizer = BootstrapFewShot(
    metric=exact_match,
    max_bootstrapped_demos=4,   # demos the program solved itself
    max_labeled_demos=16,       # upper bound on labeled examples consulted
    max_rounds=1,
)
compiled = optimizer.compile(student=student, trainset=trainset)
 
compiled.save("compiled_classifier.json")
 
loaded = dspy.Predict(ClassifyInquiry)
loaded.load("compiled_classifier.json")
print(loaded(inquiry="I'd like to go back to the premium plan.").category)

What happens inside compile isn't obvious on first read. It runs student over the training examples and keeps only the ones where exact_match passes. So the demos are examples DSPy itself solved — which is a different selection logic from picking demos by eye.

To go further, swap in MIPROv2:

from dspy.teleprompt import MIPROv2
 
# Roles can be split: a stronger model proposes instructions, the fast one does the task
prompt_lm = dspy.LM(model=os.environ["GEMINI_STRONG_MODEL"], api_key=os.environ["GEMINI_API_KEY"])
 
optimizer = MIPROv2(
    metric=exact_match,
    prompt_model=prompt_lm,
    task_model=dspy.settings.lm,
    auto="light",          # "light" / "medium" / "heavy" sets search depth
    num_threads=4,
)
compiled = optimizer.compile(
    student=student,
    trainset=trainset,
    requires_permission_to_run=False,
)

MIPROv2 tends to gain more than BootstrapFewShot, but the call count jumps by an order of magnitude. Even auto="light" can mean a lot of requests in one compile. Optimizing with a fast model and switching only production inference to a stronger one is a realistic setup.

When I can't decide, I follow one order: try BootstrapFewShot once, and keep it if it meets the target; only if it doesn't, run MIPROv2. Matching the optimizer to the task's importance also keeps spending predictable.

Write your own metric with LLM-as-a-Judge

Not every task has an exact answer. Summaries, translations and reply drafts need a judge, and the judge is itself a DSPy Signature:

# llm_judge.py — grade summary quality with Gemini
import os
import dspy
 
class SummaryJudge(dspy.Signature):
    """Compare an article and its summary; give an integer score from 1 to 5.
    If the summary adds information not in the article, score 2 or lower.
    Subtract one point for noticeable grammar errors. Never use decimals."""
    article: str = dspy.InputField(desc="Source text")
    summary: str = dspy.InputField(desc="Summary to grade")
    rationale: str = dspy.OutputField(desc="Reason for the score, 100 characters or fewer")
    score: int = dspy.OutputField(desc="Integer 1-5")
 
# Judge with the stronger model: a steadier yardstick makes a steadier optimization
judge_lm = dspy.LM(
    model=os.environ["GEMINI_STRONG_MODEL"],
    api_key=os.environ["GEMINI_API_KEY"],
    temperature=0.0,
    max_tokens=512,
)
judge = dspy.Predict(SummaryJudge)
 
def summary_metric(example, pred, trace=None):
    with dspy.context(lm=judge_lm):
        result = judge(article=example.article, summary=pred.summary)
    try:
        score = int(float(result.score))   # survive "4.0"-style strings
    except (TypeError, ValueError):
        return 0.0
    return max(0.0, min(1.0, (score - 1) / 4.0))   # 0-1 is easier on Optimizers

The point is dspy.context(lm=judge_lm): only the judge switches models, so while the task model is optimized hundreds of times, the yardstick doesn't move. Leaving the with block restores the global LM.

Write the grading rubric into the docstring. "Rate overall quality" alone makes Gemini lenient; explicit deductions bring its scores closer to a human reader's. For managing judge prompts themselves, see Production Prompt Template Management for the Gemini API.

Also check whether the judge over-rewards long summaries. Rather than fixing a numeric target in advance, I read a few dozen cases and confirm the rationale gives reasons I agree with — that surfaces bias faster than a correlation figure.

Finally, instead of one judge, split by criterion (accuracy, tone, brevity) and take a weighted average. A bias in one judge gets diluted by the others, and the cost is just a few extra calls inside metric.

Doubt the labels before you optimize

This is where I lost the most time. The Optimizer treats your labels as truth. If some are wrong, it will happily adopt them as "correctly solved" demos and grow a prompt that is faithful to the mistakes.

I once spent a long while chasing weak accuracy before finding that the culprit was neither the metric nor the model, but inconsistent human labels. Since then I always audit first. The method is simple: re-read every row where a plain Predict disagrees with the human label.

# audit_labels.py — list rows where the model and the label disagree
import csv
import dspy
 
def audit_labels(program, examples, out_path="label_suspects.csv"):
    suspects = []
    for ex in examples:
        pred = program(inquiry=ex.inquiry)
        got = pred.category.strip().lower()
        want = ex.category.strip().lower()
        if got != want:
            suspects.append({"inquiry": ex.inquiry, "label": want, "model": got})
 
    with open(out_path, "w", encoding="utf-8", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["inquiry", "label", "model"])
        writer.writeheader()
        writer.writerows(suspects)
 
    print(f"{len(suspects)} / {len(examples)} disagreements -> {out_path}")
    return suspects
 
# audit_labels(dspy.Predict(ClassifyInquiry), trainset)

Open the CSV and the disagreements sort into three piles: model mistakes, label mistakes, and messages that could honestly go either way. If the third pile is big, the problem isn't the prompt — it's the category definitions. Fixing one line of a definition often beats a hundred search iterations.

The same pass can also reveal Safety Filter blocks on some training examples. A blocked example looks like a failure to the Optimizer, so an important pattern silently drops out of learning. If the disagreements cluster in one category, Layered Safety Settings for the Gemini API in Production is a good place to look next.

Measure what one compile costs

An Optimizer calls the API far more than you'd expect. Before committing, you want to know how many calls and tokens a single compile uses — and the reliable way is to run it once and count.

DSPy's LM keeps a call log in history. Take the length before and after compile and the difference is your call count.

# budget_probe.py — count calls and tokens for one compile run
import dspy
 
# Fill these from the official pricing page; no numbers are hard-coded here
PRICE_IN_PER_MTOK = 0.0   # per 1M input tokens
PRICE_OUT_PER_MTOK = 0.0  # per 1M output tokens
 
def probe_compile(optimizer, student, trainset, lms, **kwargs):
    """lms: every LM involved (task model, instruction-proposal model, ...)"""
    marks = [len(lm.history) for lm in lms]
    compiled = optimizer.compile(student=student, trainset=trainset, **kwargs)
 
    calls = tok_in = tok_out = 0
    for lm, start in zip(lms, marks):
        for h in lm.history[start:]:
            usage = h.get("usage") or {}
            calls += 1
            tok_in += usage.get("prompt_tokens", 0)
            tok_out += usage.get("completion_tokens", 0)
 
    cost = tok_in / 1e6 * PRICE_IN_PER_MTOK + tok_out / 1e6 * PRICE_OUT_PER_MTOK
    print(f"calls={calls}  in={tok_in:,} tok  out={tok_out:,} tok  est_cost={cost:.4f}")
    return compiled

Two cautions. If MIPROv2 uses a separate model to propose instructions, include that LM in lms and sum both. And history is capped, so for large runs check the related dspy.settings value. Cached calls may be recorded differently, so do the run you use for budgeting with cache=False and the numbers read cleanly.

With real numbers, decisions get concrete: iterate on 20–30 examples during development, run the heavier setting on the full data once before shipping, search with a lighter model if you need to save, and consider batch processing where it's cheaper (check the pricing page for the discount). For putting a token check into your routine, Using countTokens as a Budget Gate for Batches is a useful companion.

Run it end to end on real data

Here is everything combined into one script: split your support log into train, validation and test sets, then compare the baseline and the optimized program on the same test set.

# production_pipeline.py — optimize on real data, evaluate on unseen data
import json
import os
import random
import dspy
from dotenv import load_dotenv
from dspy.teleprompt import MIPROv2
 
load_dotenv()
random.seed(42)
 
lm = dspy.LM(
    model=os.environ["GEMINI_FAST_MODEL"],
    api_key=os.environ["GEMINI_API_KEY"],
    temperature=0.0,
    max_tokens=256,
)
dspy.configure(lm=lm)
 
class ClassifyInquiry(dspy.Signature):
    """Classify a user inquiry as payment / bug / feature / other."""
    inquiry: str = dspy.InputField()
    category: str = dspy.OutputField(desc="payment, bug, feature, or other")
 
# support_logs.json: [{"text": "...", "label": "bug"}, ...]
with open("support_logs.json", encoding="utf-8") as f:
    raw = json.load(f)
 
examples = [
    dspy.Example(inquiry=r["text"], category=r["label"]).with_inputs("inquiry")
    for r in raw
]
random.shuffle(examples)
 
n = len(examples)
trainset = examples[: int(n * 0.7)]
valset = examples[int(n * 0.7) : int(n * 0.9)]
testset = examples[int(n * 0.9) :]
 
def exact_match(example, pred, trace=None):
    return pred.category.strip().lower() == example.category.strip().lower()
 
def accuracy(program, dataset):
    hits = sum(1 for e in dataset if exact_match(e, program(inquiry=e.inquiry)))
    return hits / len(dataset)
 
baseline = dspy.Predict(ClassifyInquiry)
print(f"baseline  : {accuracy(baseline, testset):.3f}")
 
optimizer = MIPROv2(metric=exact_match, auto="light", num_threads=4)
compiled = optimizer.compile(
    student=dspy.Predict(ClassifyInquiry),
    trainset=trainset,
    valset=valset,
    requires_permission_to_run=False,
)
 
# The only number that counts: data the optimizer never touched
print(f"optimized : {accuracy(compiled, testset):.3f}")
 
compiled.save("inquiry_classifier_v1.json")

The two numbers it prints are whatever your data produces. Gains someone else saw on their data are a rough expectation at best, never a promise. Measure the real result only on a test set the optimizer has never seen. Holding that single line changes how much you can trust your own numbers.

Pitfalls I actually hit

Pitfall 1: the run dies and the calls are lost. MIPROv2 makes many requests, and a 429 Resource Exhausted partway through can throw away progress. Use a key with rate-limit headroom and start with a modest num_threads (around 4); push it higher and you hit the ceiling fast.

Pitfall 2: a loose metric, so you can't tell what was optimized. Something like "1 if a keyword appears" will still be reported as optimized — but it only satisfied the goal you wrote, not real usefulness. Combine exact match with a length check and banned-word check, then read a few dozen outputs yourself.

Pitfall 3: changing the Signature after compile. The few-shot demos in the saved JSON are tied to the Signature at the time. Rename a field or change a type and loading may fail, or worse, silently ignore them. Recompile after any Signature change and version the files (compiled_classifier_v2.json).

Pitfall 4: structured output is less strict than it looks. Through LiteLLM, a field declared score: int can still come back as "4.0". Cast in two steps (int(float(x))) or say "no decimals" in the docstring. For the wider picture, see Fixing Structured Output Schema Validation in the Gemini API.

Pitfall 5: underestimating demo order. English benchmarks often suggest order barely matters; in Japanese classification tasks I've felt it matter. I haven't run a study I'd quote numbers from, so I'll only say: test it on your own data. DSPy's default order is random, and the more demos you allow, the more order matters. Start with 3–5 and compare several compiles with shuffled order.

Three habits that keep it running in production

Running on your laptop and running for months are different things.

Habit 1: keep compiled_*.json in Git. save() writes plain JSON with instructions and demos in readable form. Put it under prompts/, send it through normal PRs, and the commit log records who optimized what, on which data. When accuracy drops in production, going back one version is a single step. Better still, have the PR show before-and-after test accuracy automatically, so small differences pass quickly and only big moves need a human read.

Habit 2: separate the optimization model from the inference model — and re-evaluate on the inference model. Search with a light model, serve with another. Accuracy shifts a little between models, so before deploying, re-run the test set on the model you'll actually serve.

Habit 3: recompile on a schedule and alert on a threshold. User messages drift. Schedule a re-optimization job and notify yourself when test accuracy falls below a floor you chose, and prompt work turns from "keep writing" into "keep watching."

My layout settled into app/ for production code, prompts/ for versioned compiled files, training_data/ for labeled data, and the job definition for re-optimization. The DSPy code itself stays small; it's how carefully you handle this surrounding metadata that decides quality.

For regression testing, see Prompt Regression Testing with Pytest for the Gemini API; for the architecture overall, Gemini API Production Architecture Patterns 2026.

The order I'd adopt it in, starting today

1. Pick the one prompt that causes the most rework — classification, extraction or formatting, where the right answer is clear
2. Label 30–50 evaluation examples yourself; this part is human work, and cutting corners shows later
3. Run the label audit: re-read rows where a plain Predict disagrees, and fix definitions and mistakes
4. Move the existing prompt into a Signature and measure the baseline with Predict and no few-shot
5. Run BootstrapFewShot exactly once
6. If you're short of the target, move to MIPROv2 — and if that still falls short, suspect the Signature or the metric
7. Ship it and set up the recompile job

Start with a task that's easy to score. Long creative generation can eat all your time in metric design and leave you discouraged.

What changed most after switching to DSPy was getting out of the endless "a bit more polish and it'll improve" loop. With a metric and data in hand, the search belongs to the machine, and you spend your time deciding what counts as good.

If you're currently nudging a Gemini-powered prompt every weekend, tonight is a good time to label 50 examples and run the label audit and BootstrapFewShot once each. By the time you've read the results, I suspect your next weekend will look different.