When Transcription Keeps Mangling Proper Nouns: Choosing What Earns a Custom Vocabulary Slot

The first thing I saw in the transcript was a woodblock artist's name spelled half in kanji and half in kana.

I keep a wallpaper app that publishes ukiyo-e prints, and while I work through which pieces to ship I talk to a voice recorder. Artist names and series titles are the unit of the work — if those break, I cannot follow my own notes a week later.

I already knew about custom_vocabulary. The documentation says you can supply up to 1,000 entries. So I did what I suspect most people do: I sat down, listed every artist and series title I could think of, and handed the whole thing over.

The results were not good. Some names came back clean, others did not, and I had no idea why the list helped in one place and not another.

Writing the list first meant spending the budget wrong

Looking back, I had the order backwards.

I built my list by remembering which proper nouns were likely to show up. That is an inventory of what lives in my head. But transcription does not need an inventory. It needs the words that actually fell over.

When a limit says 1,000, the instinct is to fill it. Yet the practical budget is closer to a tenth of that — I wrote about the mode combinations and their limits separately, in why smart mode returns neither speaker labels nor word timestamps. You will dilute the list with entries that do nothing long before you run into the hard ceiling.

A vocabulary list is not something you recall — it is something you collect from what broke. It took me three rewrites of the same list to arrive at that sentence.

A failure log is just reference-and-hypothesis pairs

To collect anything, you need a record that shows what broke. Mine turned out to be much smaller than I expected.

You need two things per segment: what was actually said (the reference) and what transcription returned (the hypothesis). Keep them side by side, in short chunks.

{
  "terms": ["Utagawa Hiroshige", "Kitagawa Utamaro", "Toshusai Sharaku",
            "Katsushika Hokusai", "One Hundred Famous Views of Edo",
            "ukiyo-e", "wallpaper"],
  "pairs": [
    {"ref": "Three plates from Utagawa Hiroshige's One Hundred Famous Views of Edo",
     "hyp": "Three plates from Utagawa Hiroshigi's One Hundred Famous Views of Edo"},
    {"ref": "Kitagawa Utamaro's bijin-ga will become wallpaper",
     "hyp": "Kitagawa Utamaru's bijin-ga will become wallpaper"},
    {"ref": "Toshusai Sharaku only worked for ten months",
     "hyp": "Toshusai Sharaku only worked for ten months"},
    {"ref": "Katsushika Hokusai's Thirty-six Views can wait for another batch",
     "hyp": "Katsushika Hokusai's Thirty-six Views can wait for another batch"},
    {"ref": "Keep Utagawa Hiroshige's rain strokes, so do not crush the ukiyo-e grain",
     "hyp": "Keep Utagawa Hiroshigay's rain strokes, so do not crush the ukiyo-e grain"},
    {"ref": "Kitagawa Utamaro's lines are thin and the wallpaper goes soft when zoomed",
     "hyp": "Kitagawa Utamara's lines are thin and the wallpaper goes soft when zoomed"},
    {"ref": "Ship the bridge set from One Hundred Famous Views of Edo first",
     "hyp": "Ship the bridge set from One Hundred Famous Views of Edo first"}
  ]
}

If the thought of writing references sounds like a chore, I felt the same way at first. You do not need a full transcript, though. Only the segments that contain proper nouns matter, and five of them from a single recording are already enough to see a pattern.

The terms array can simply be the inventory list you wrote in the first place. Nothing is wasted — the destination just changes. It goes to the script below instead of to the API.

Sixty lines that rank the candidates

The logic is plain. For each term, count how often it appears in the reference and how often it survives into the hypothesis, then take the difference. Only terms with a difference become candidates.

import json, sys, unicodedata
from collections import defaultdict
 
BUDGET = 100  # what I actually ship (1,000 is the API ceiling)
 
def norm(s):
    return unicodedata.normalize("NFKC", s).strip()
 
def build(pairs, terms, budget=BUDGET):
    seen = defaultdict(int)   # times the term appeared in the audio
    miss = defaultdict(int)   # times it did not survive recognition
    for ref, hyp in pairs:
        r, h = norm(ref), norm(hyp)
        for t in terms:
            n = r.count(norm(t))
            if n == 0:
                continue
            seen[t] += n
            got = h.count(norm(t))
            if got < n:
                miss[t] += n - got
 
    rows = []
    for t in terms:
        if seen[t] == 0:
            rows.append((t, 0, 0, None, "not in this audio"))
            continue
        rate = miss[t] / seen[t]
        if miss[t] == 0:
            rows.append((t, seen[t], 0, 0.0, "already correct"))
        else:
            rows.append((t, seen[t], miss[t], rate, "candidate"))
 
    cand = [r for r in rows if r[4] == "candidate"]
    cand.sort(key=lambda r: (-r[2], -r[3], r[0]))
    picked = [r[0] for r in cand[:budget]]
    dropped = [r for r in rows if r[4] != "candidate"]
    return picked, cand, dropped
 
if __name__ == "__main__":
    data = json.load(open(sys.argv[1], encoding="utf-8"))
    picked, cand, dropped = build(
        [(d["ref"], d["hyp"]) for d in data["pairs"]], data["terms"]
    )
    print(f"candidates {len(cand)} / picked {len(picked)} / dropped {len(dropped)}\n")
    print("rank  miss  seen   rate  term")
    for i, (t, s, m, rate, _) in enumerate(cand, 1):
        print(f"{i:>3}   {m:>3}   {s:>3}  {rate:>5.2f}  {t}")
    print("\n-- not given a slot --")
    for t, s, m, rate, why in dropped:
        print(f"  {t} ({why})")
    print("\ncustom_vocabulary =", json.dumps(picked, ensure_ascii=False))

The norm() call applies NFKC normalisation so that the same term written with full-width and half-width characters is not counted as two different things. Skip it and your counts drift the moment a term contains digits or Latin letters.

The important line is the sort. I rank by the number of failures first and fall back to the failure rate only for ties. Ranking by rate alone floods the top of the list with terms that appeared exactly once. A term that keeps falling over is worth more, because fixing it pays off repeatedly.

Feeding the sample above through the script gives this:

candidates 2 / picked 2 / dropped 5

rank  miss  seen   rate  term
  1     2     2   1.00  Kitagawa Utamaro
  2     2     2   1.00  Utagawa Hiroshige

-- not given a slot --
  Toshusai Sharaku (already correct)
  Katsushika Hokusai (already correct)
  One Hundred Famous Views of Edo (already correct)
  ukiyo-e (already correct)
  wallpaper (already correct)

custom_vocabulary = ["Kitagawa Utamaro", "Utagawa Hiroshige"]

Seven terms went in. Two of them earned a slot.

Where the picked list actually goes

The output of the script is the value, not the config. custom_vocabulary sits directly under transcription_config, at the same level as language_codes — not inside the mode object, which is where the speaker and timestamp settings live. I have mixed those two levels up more than once.

transcription_config = {
    "language_codes": ["en-US"],
    "custom_vocabulary": picked,          # straight from the script
    "mode": {"type": "verbatim",
             "timestamp_granularities": ["word"]},
}

Setting language_codes when you already know the language is worth doing alongside the vocabulary. A list of names is a weaker signal on its own; pinning the language narrows what the model is choosing between before the names even come into play. On recordings where I switch between two languages mid-sentence, I gave up on pinning it and accepted a slightly noisier result instead of a confidently wrong one.

One caveat I want to state plainly rather than imply: I have not measured how the entry count relates to accuracy, and I would not trust a number I made up for it. What I can say is what the log shows me — that most of the terms I was shipping never failed in the first place, so they were never doing any work.

The terms you leave out fall into three groups

Not everything is excluded for the same reason, and separating the reasons saves you from rethinking the whole list next time.

The third group deserves a note. In my sample, one name broke in two distinct ways, and the first half of it survived every time. That tells me the model is only unsure about the tail of the word — which is exactly the situation where a vocabulary entry helps.

Some terms scatter into a different shape on every pass instead. Those are the ones I used to keep re-adding, convinced the list was almost right. When that happens, the cause is usually pace: words running into each other, or no pause between a name and the word after it. I once fixed a whole cluster of failures simply by re-reading a rushed passage more slowly, without touching the vocabulary at all.

Leave the rest of the budget open for the next recording

Against a budget of a hundred, the first recording yields very little. That empty space can feel like a mistake, but it is not one.

Every time I record, I add a few more reference-and-hypothesis rows and run the same script again. Writing five lines takes a couple of minutes, which is little enough that I actually do it, and that turned out to matter more than any refinement to the script itself. The set of failing terms shifts slowly, and any artist I have just started working with will naturally break on the first pass. The list is not a finished artefact — it is a ledger, something you keep updating rather than something you complete.

One more rule I follow: once a term has gone three recordings without failing, I take it back out. Models get updated, and words that used to need help stop needing it. Holding the slot means I cannot hand it to something that still does. The removals are logged the same way as the additions, so if a term starts failing again I can see that it was once on the list and why it came off — that history has saved me from re-litigating the same decision twice.

You do not need to start with anything long. Pick a ten-minute recording you already have, write out five segments that contain proper nouns, and run them through the script.

That alone shows you how many terms genuinely deserve a slot right now, and it shows it in a form you can hand to the API without editing. For me it was two — and it gave me far more confidence than the long list I had been shipping.

Thank you for reading. If you also build your work out of voice notes, I hope some part of this overlaps with yours.