When Sampling Parameters Were Deprecated, Diversity Broke Before Determinism Did

What broke was the diversity generator

The real problem lived in src/variation/paraphrase.py.

That file produces paraphrase candidates for search intents. People type wildly different things into a wallpaper app's search box, so instead of hand-growing a synonym dictionary I generate paraphrases from representative intents and feed them into the regression suite as test inputs.

There I had temperature at 1.2 and pulled the same prompt eight times. The temperature was the diversity generator.

Turn it down and re-run, and of the eight candidates only two or three are meaningfully different. As test input, that is thin. Verifying the same phrasing over and over is guaranteed to pass.

What struck me was that the diversity had been working by accident. I never once declared what I wanted to vary. I delegated it to the width of a distribution, so when the width was taken away, there was no fallback.

Determinism was protected by design. Diversity was left to chance. Laid side by side, the priorities look inverted.

Move diversity into the input layer

I changed the approach. Instead of pulling one prompt eight times, I pull eight different prompts once each.

That requires naming what you want to vary. In my case three axes: vocabulary, length, and noise.

#!/usr/bin/env python3
"""variation_axes.py — build diversity in the input layer, not the sampler."""
import itertools
import random
 
AXES = {
    "vocabulary": ["everyday words only", "official in-app terms", "slang and abbreviations"],
    "length": ["3 words or fewer", "one sentence", "two or more explanatory sentences"],
    "noise": ["clean spelling", "phonetic spelling", "one typo included"],
}
PREFIX = "Paraphrase the following search intent. Intent: '{intent}'\nConstraints: "
 
 
def build(intent, n, seed=0):
    """Naive: shuffle the full product and take the first n."""
    combos = list(itertools.product(*AXES.values()))
    random.Random(seed).shuffle(combos)
    keys = list(AXES)
    out = []
    for combo in combos[:n]:
        cond = ", ".join(f"{k} = {v}" for k, v in zip(keys, combo))
        out.append(PREFIX.format(intent=intent) + cond)
    return out

Three axes at three levels each gives 27 combinations. I take eight of them.

Because the seed is fixed, the same seed reproduces the same input set. For regression testing that reproducibility is the decisive difference from temperature-driven generation: when results change between runs, you can tell whether the inputs moved or the model did.

Balancing the axes created duplicates

The naive output looked off immediately.

combinations: 27  / selected: 8
1. vocabulary = official in-app terms, length = one sentence, noise = one typo included
2. vocabulary = official in-app terms, length = one sentence, noise = clean spelling
3. vocabulary = slang and abbreviations, length = 3 words or fewer, noise = phonetic spelling
4. vocabulary = official in-app terms, length = one sentence, noise = phonetic spelling
...

Five of the eight used "official in-app terms." Drawing eight at random from a pool this small skews as a matter of course.

So I wrote a stratified version that cycles each axis: shuffle the levels per axis, then assign with i % number_of_levels. The skew vanished exactly as intended.

In exchange, I got duplicates.

All three axes have three levels, so all three cycle with the same period, which makes i and i+3 identical combinations. Five of the eight prompts were repeats.

Trying to spread the set evenly destroyed the very diversity I was after.

I settled on greedy selection instead. At each step, pick the unused combination containing the levels used least so far. Fixing only the tie-break order with a seed preserves reproducibility.

def build_balanced(intent, n, seed=0):
    """Minimize level skew without emitting duplicates."""
    keys, levels = list(AXES), list(AXES.values())
    pool = list(itertools.product(*levels))
    random.Random(seed).shuffle(pool)        # seed fixes tie-break order only
    used = {k: {v: 0 for v in vals} for k, vals in AXES.items()}
    chosen = []
    for _ in range(min(n, len(pool))):
        best = min(pool, key=lambda c: (sum(used[k][v] for k, v in zip(keys, c)),
                                        pool.index(c)))
        pool.remove(best)
        for k, v in zip(keys, best):
            used[k][v] += 1
        cond = ", ".join(f"{k} = {v}" for k, v in zip(keys, best))
        chosen.append(PREFIX.format(intent=intent) + cond)
    return chosen
 
 
def coverage(prompts):
    """How evenly the selected set touches each axis level."""
    rep = {}
    for k, vals in AXES.items():
        c = {v: sum(1 for p in prompts if f"{k} = {v}" in p) for v in vals}
        rep[k] = (c, max(c.values()) - min(c.values()))
    return rep

The three strategies compared over the same seed and the same eight prompts:

Writing the axes down turned out to be a more valuable by-product than the code. While I was running on temperature, I never articulated what variation I actually wanted. Once it was written out, I noticed I had never deliberately tested an input containing a typo.

Judge the output by effective diversity, not by count

Spreading the inputs is pointless if the returned candidates still look alike. So the pass condition is not how many candidates came back, but how many genuinely different phrasings survive.

I built a two-layer gate: collapse near-identical candidates with a SimHash over character bigrams, then sieve the survivors again with token-set Jaccard. I did not want to pull in a morphological analyzer for Japanese, so character trigrams stand in for words.

#!/usr/bin/env python3
"""diversity_gate.py — measure the effective diversity of paraphrase candidates."""
import hashlib
import re
import sys
import unicodedata
 
SIMHASH_BITS = 64
HAMMING_MAX = 12      # at or below this distance, treat as the same phrasing
JACCARD_MAX = 0.72    # above this overlap, treat the token set as redundant
 
 
def normalize(s):
    s = unicodedata.normalize("NFKC", s).lower()
    return re.sub(r"[\s、。,.!?！？「」『』（）()・\-—]+", "", s)
 
 
def shingles(s, n=2):
    s = normalize(s)
    return [s[i:i + n] for i in range(max(len(s) - n + 1, 1))]
 
 
def simhash(s):
    v = [0] * SIMHASH_BITS
    for sh in shingles(s):
        h = int.from_bytes(hashlib.blake2b(sh.encode(), digest_size=8).digest(), "big")
        for i in range(SIMHASH_BITS):
            v[i] += 1 if (h >> i) & 1 else -1
    out = 0
    for i, x in enumerate(v):
        if x > 0:
            out |= 1 << i
    return out
 
 
def hamming(a, b):
    return bin(a ^ b).count("1")
 
 
def tokens(s):
    # approximation that avoids a morphological analyzer: trigrams stand in for words
    t = normalize(s)
    return {t[i:i + 3] for i in range(max(len(t) - 2, 1))}
 
 
def jaccard(a, b):
    return len(a & b) / len(a | b) if (a | b) else 0.0
 
 
def gate(cands):
    kept, dropped = [], []
    for c in cands:
        h, tk = simhash(c), tokens(c)
        clash = next((k for k in kept
                      if hamming(h, k[1]) <= HAMMING_MAX
                      or jaccard(tk, k[2]) >= JACCARD_MAX), None)
        if clash:
            dropped.append((c, clash[0]))
        else:
            kept.append((c, h, tk))
    return [k[0] for k in kept], dropped
 
 
if __name__ == "__main__":
    cands = [l.rstrip("\n") for l in sys.stdin if l.strip()]
    kept, dropped = gate(cands)
    print(f"candidates          : {len(cands)}")
    print(f"survivors (effective): {len(kept)}  = {len(kept) / len(cands):.0%}")
    for c in kept:
        print(f"  + {c}")
    for c, by in dropped:
        print(f"  - {c}  <- collapsed into: {by}")

Running ten Japanese candidates through it:

candidates          : 10
survivors (effective): 8  = 80%
  + 壁紙を検索する
  + 壁紙をさがす
  + 背景画像を検索する
  + 待ち受けを変えたい
  + ホーム画面の画像を差し替える
  + スマホの背景を選ぶ
  + かべがみをけんさくする
  + ロック画面に使う画像がほしい
  - 壁紙を探す  <- collapsed into: 壁紙を検索する
  - 壁紙 を 検索 する  <- collapsed into: 壁紙を検索する

Choosing the threshold took some deliberation. Sweeping HAMMING_MAX over the same candidate set:

Loosen it to 20 and "background image" gets collapsed into "wallpaper." For validating search synonyms, that is a candidate I want to keep. Tighten it to 6 and a genuine verb synonym slips through as if it were a distinct phrasing.

Twelve was the knee. Past that point, what gets removed shifts from redundant spelling variants to meaningful paraphrases.

Your threshold will depend on your use case, so rather than copying my number, run the sweep on your own candidate set once. Watching which candidates start disappearing makes the choice obvious.

One honest limitation: character-level methods have a ceiling. In Japanese, the kanji and kana spellings of the same verb differ sharply at the bigram level, so at 12 they survive as separate entries. Without reading data you cannot collapse that class of variation by string distance alone. I chose to keep them, because users really do type both.

What it does to the bill

You often see the claim that moving diversity to the input layer is cheaper. Decomposing it, the claim is conditional.

Pulling the same prompt eight times bills the shared prefix — instructions, output contract, few-shot examples — in full every time. Varying only the tail lets the prefix sit in cache.

So the saving depends entirely on whether prefix caching applies. Discount rates shift with the spec, so I made it a parameter rather than hard-coding it.

#!/usr/bin/env python3
"""diversity_cost.py — decompose the bill by which layer produces diversity."""
IN_PER_1M = {"gemini-3.6-flash": 1.50, "gemini-3.5-flash-lite": 0.30}
OUT_PER_1M = {"gemini-3.6-flash": 7.50, "gemini-3.5-flash-lite": 2.50}
 
SHARED_PREFIX = 1500   # instructions, output contract, few-shot examples
VARIABLE_TAIL = 300    # the part that differs per target
OUT_TOKENS = 220       # output per candidate
N_VARIANTS = 8
N_TARGETS = 120
DAYS = 30
 
 
def cost(model, cache_discount):
    ip, op = IN_PER_1M[model], OUT_PER_1M[model]
    calls = N_VARIANTS * N_TARGETS * DAYS
 
    # A: same long prompt n times; prefix billed in full every call
    in_a = (SHARED_PREFIX + VARIABLE_TAIL) * calls
 
    # B: prefix billed once per target, then at the cached rate
    per_target = (SHARED_PREFIX
                  + SHARED_PREFIX * (N_VARIANTS - 1) * (1 - cache_discount)
                  + VARIABLE_TAIL * N_VARIANTS)
    in_b = per_target * N_TARGETS * DAYS
 
    out = OUT_TOKENS * calls
    return in_a / 1e6 * ip + out / 1e6 * op, in_b / 1e6 * ip + out / 1e6 * op
 
 
if __name__ == "__main__":
    for model in IN_PER_1M:
        print(f"[{model}] in ${IN_PER_1M[model]}/1M, out ${OUT_PER_1M[model]}/1M")
        for d in (0.0, 0.50, 0.75, 0.90):
            a, b = cost(model, d)
            print(f"  discount {d:>4.0%} | A ${a:>7,.2f} | B ${b:>7,.2f} "
                  f"| delta ${a - b:>5,.2f} | {(a - b) / a:>5.1%}")

The assumption is 8 candidates over 120 targets across 30 days, or 28,800 calls a month.

The first row is the important one. At a 0% discount, A and B differ by nothing.

Which makes sense: moving diversity into the input layer does not reduce total input tokens. It only reduces the unit price of the portion you were resending. If your prefix is not structured to land in cache, the redesign buys you no money at all.

Read the other way: the longer and more stable your shared prefix, the more this pays. I took the opportunity to reorder my prompts so instructions and the output contract sit at the front and everything target-specific collects at the tail.

For high-volume, low-stakes work, the $21 to $31 per month band on gemini-3.5-flash-lite is also worth weighing. Regression-test input generation does not demand top-tier quality, so I moved that job over.

The order I would migrate in

For anyone about to do the same, here is the sequence I ended up glad about.

1. Count call sites by effect, not by declaration. Propagate carriers so you catch files that never mention the keyword
2. Clear the paths that already have structured output first. With response_schema in place, dropping the temperature is usually the whole job
3. Find the paths where temperature was producing diversity. This is the real work. Anything set above 1.0 almost certainly belongs here
4. Write down the axes you actually want to vary. The act of articulating what the distribution was doing for you surfaces gaps in coverage
5. Define the pass condition as effective diversity before you swap the generator. Without a measure in place first, you will not notice the set going thin

Schemas protect determinism for you. Diversity only survives if you build the mechanism yourself. This deprecation was a useful occasion to see that asymmetry clearly.

Specs and prices move, so please confirm against the Gemini API changelog and the pricing page before you implement. Every figure above is raw output from the scripts as printed. Swap in your own assumptions and run them again.

I am still learning where the right knee sits for my own workloads, and I would be glad if any of this saves you a detour.