One evening I turned the screen reader on and walked down the list screen of my wallpaper app from the top. The category headings read out the way I expected.
Then I reached the row of icons in the corner, and the device said "button", "button", "image", and moved on. Listening alone, there was no way to tell which one would save anything.
The translations for that screen were complete. I ship in several languages, so I always check for untranslated keys before a release. The reading layer still had that many holes in it.
A Translation Check Cannot See a Label That Was Never Written
The reason turned out to be simple. A localization check compares one set of keys against another. It finds keys that exist in Japanese but not in English. What it cannot find is a place where no key exists at all — and a label you forgot to write has no key. Nothing untranslated is being counted, because nothing is there.
SwiftUI has a second path that is easy to miss. When you pass a string literal, as in .accessibilityLabel("Save"), it is treated as a LocalizedStringKey. If no entry named Save exists in your strings files, there is no warning and no error — the literal itself is read aloud, in every language. From the translation side, that line was never visible.
Android's @string/ references behave in a related way. A reference that exists nowhere fails at build time, but one that is missing from a single language quietly falls back to the default, and the default gets spoken to a reader who expected their own language.
Whether a label is required can be settled statically; what it should say cannot be settled without looking at the screen. Separating those two questions is what finally let me start working. The first half I can answer in my own environment. The second half is worth asking a model about.
Step One: Gather Every Icon Element into One Table
Before judging anything, gather the material. On iOS I start from Image(...) in SwiftUI and look at the modifiers that follow it. On Android I read ImageButton and ImageView out of the layout XML. At this stage I deliberately decide nothing about quality. Deciding nothing here matters later, as you will see.
# collect_labels.py - gather icon elements and their screen-reader labels from source
import json
import re
import sys
from pathlib import Path
from xml.etree import ElementTree
ANDROID_NS = "{http://schemas.android.com/apk/res/android}"
ICON_VIEWS = {"ImageButton", "ImageView"}
# In SwiftUI a string literal is a LocalizedStringKey too, so look it up
# in the strings table whether or not it is wrapped in Text(...)
LABEL_RE = re.compile(r'\.accessibilityLabel\(\s*(?:Text\(\s*)?"([^"]*)"')
IMAGE_RE = re.compile(r'\bImage\(\s*(?:systemName:\s*)?"([^"]+)"')
STRINGS_RE = re.compile(r'"([^"]+)"\s*=\s*"([^"]*)"\s*;')
def load_lproj(root: Path):
table = {}
for path in sorted(root.glob("*.lproj/Localizable.strings")):
locale = path.parent.name.replace(".lproj", "")
locale = "default" if locale == "en" else locale
for key, value in STRINGS_RE.findall(path.read_text(encoding="utf-8")):
table.setdefault(key, {})[locale] = value
return table
def collect_ios(root: Path):
strings = load_lproj(root)
records = []
for path in sorted(root.rglob("*.swift")):
lines = path.read_text(encoding="utf-8").splitlines()
for i, line in enumerate(lines):
icon = IMAGE_RE.search(line)
if not icon:
continue
# modifiers sit right below, so a four-line window is enough
window = "\n".join(lines[i : i + 5])
label = LABEL_RE.search(window)
# resolved means it is a key; unresolved means the literal is spoken as-is
resolved = strings.get(label.group(1)) if label else None
is_button = any("Button(" in lines[j] for j in range(max(0, i - 3), i))
records.append(
{
"platform": "ios",
"element": "Button+Image" if is_button else "Image",
"icon": icon.group(1),
"interactive": is_button,
"label_raw": label.group(1) if label else None,
"label_kind": ("key" if resolved else "literal") if label else None,
"values": resolved or {},
}
)
return records
def load_android_strings(res_dir: Path):
table = {}
for xml in sorted(res_dir.glob("values*/strings.xml")):
locale = xml.parent.name.replace("values-", "")
locale = "default" if locale == "values" else locale
for node in ElementTree.parse(xml).getroot().findall("string"):
table.setdefault(node.get("name"), {})[locale] = (node.text or "").strip()
return table
def collect_android(res_dir: Path):
strings = load_android_strings(res_dir)
records = []
for xml in sorted((res_dir / "layout").glob("*.xml")):
for node in ElementTree.parse(xml).iter():
if node.tag not in ICON_VIEWS:
continue
desc = node.get(ANDROID_NS + "contentDescription")
key = desc[8:] if desc and desc.startswith("@string/") else None
values = strings.get(key, {}) if key else {}
records.append(
{
"platform": "android",
"element": node.tag,
"icon": (node.get(ANDROID_NS + "src") or "").replace("@drawable/", ""),
"interactive": node.tag == "ImageButton",
"label_raw": desc,
"label_kind": ("key" if values else "literal") if desc else None,
"values": values,
}
)
return records
if __name__ == "__main__":
root = Path(sys.argv[1])
records = collect_ios(root / "ios") + collect_android(root / "android" / "res")
json.dump(records, sys.stdout, ensure_ascii=False, indent=2)On the small project I used to work this through, the collector found 11 icon elements: 9 buttons and 2 images.
I Searched for Role Words Before Resolving, and Failed My Own Naming
My first version of the check looked for role words — "button", "image", "icon" — directly in whatever string it had collected. A screen reader announces the role of an element on its own, so repeating it in the label means the user hears it twice.
Here is how that run started.
ROLE_WORD | ios/Button+Image | arrow.down.circle | role word found: save_button
MISSING | ios/Button+Image | heart | interactive element has no label
ROLE_WORD | ios/Button+Image | square.and.arrow.up | role word found: Button
ROLE_WORD | ios/Button+Image | shuffle | role word found: shuffle_buttonThe first and fourth lines stopped me. save_button and shuffle_button are key names that I chose. What the screen reader actually speaks is "Save" and "Shuffle", not the name of the key. My own naming convention was failing my own check.
The fix was one change of order. Role words are only searched for in the display text after the key has been resolved. A @string/ reference, or a key name, must never be the input to that check.
Looking back, this was a seam in the design rather than a typo. I had told myself that the collector decides nothing — and then let the judging side accept raw strings anyway. What was missing was a written rule about what may cross between the two.
Step Two: Settle Everything Mechanical Before Any API Call
The reordered check is below. It asks four questions only: does a label exist, does the reference resolve, are all required languages present, and does the text repeat a role word.
# prescreen.py - settle the mechanical failures before anything reaches the model
import json
import sys
ROLE_WORDS = ("button", "image", "icon", "graphic")
REQUIRED_LOCALES = ("default", "ja")
def display_texts(record):
"""Return only what is actually spoken. Keys and references are excluded."""
if record.get("label_kind") == "key":
return list((record.get("values") or {}).values())
if record.get("label_kind") == "literal":
return [record["label_raw"]]
return []
def verdicts(record):
out = []
kind = record.get("label_kind")
values = record.get("values") or {}
if not kind:
if record["interactive"]:
return [("MISSING", "interactive element has no label")]
return [("DECORATIVE?", "hide it from the reader explicitly if decorative")]
if kind == "literal":
out.append(("HARDCODED", f"same string in every language: {record['label_raw']}"))
elif not values:
out.append(("UNRESOLVED", f"{record['label_raw']} does not resolve"))
else:
gaps = [loc for loc in REQUIRED_LOCALES if not values.get(loc)]
if gaps:
out.append(("LOCALE_GAP", f"undefined for: {', '.join(gaps)}"))
# role words are checked against resolved display text only
for text in display_texts(record):
if any(word in (text or "").lower() for word in ROLE_WORDS):
out.append(("ROLE_WORD", f"role word found: {text}"))
break
return out
if __name__ == "__main__":
records = json.load(open(sys.argv[1], encoding="utf-8"))
passed = []
for rec in records:
found = verdicts(rec)
head = f"{rec['platform']}/{rec['element']}"
if not found:
passed.append(rec)
print(f"{'PASS':<11}| {head:<22}| {rec['icon']:<20}| sending to wording review")
continue
for code, detail in found:
print(f"{code:<11}| {head:<22}| {rec['icon']:<20}| {detail}")
stopped = len(records) - len(passed)
print(f"{len(records)} total / {stopped} settled locally / {len(passed)} sent to the model")
json.dump(passed, open("to_judge.json", "w", encoding="utf-8"), ensure_ascii=False, indent=2)Against the same 11 elements, this is what came back.
PASS | ios/Button+Image | arrow.down.circle | sending to wording review
MISSING | ios/Button+Image | heart | interactive element has no label
HARDCODED | ios/Button+Image | square.and.arrow.up| same string in every language: Button
ROLE_WORD | ios/Button+Image | square.and.arrow.up| role word found: Button
HARDCODED | ios/Button+Image | square.grid.2x2 | same string in every language: Open category list button
ROLE_WORD | ios/Button+Image | square.grid.2x2 | role word found: Open category list button
LOCALE_GAP | ios/Button+Image | shuffle | undefined for: default
HARDCODED | ios/Image | hero_banner | same string in every language: Image
ROLE_WORD | ios/Image | hero_banner | role word found: Image
PASS | android/ImageButton | ic_save | sending to wording review
MISSING | android/ImageButton | ic_heart | interactive element has no label
PASS | android/ImageButton | ic_share | sending to wording review
UNRESOLVED | android/ImageButton | ic_shuffle | @string/shuffle_button does not resolve
ROLE_WORD | android/ImageView | hero_banner | role word found: Image
11 total / 8 settled locally / 3 sent to the modelEight of the eleven were settled without calling anything. The remaining three expand to six rows once you open them per language. Here is what each verdict means.
| Verdict | Meaning | Fix |
|---|---|---|
MISSING | Interactive element has no label | Add one. There is nothing to discuss here |
DECORATIVE? | An image carries no label | If it is decorative, hide it from the reader explicitly |
HARDCODED | Does not resolve as a key, so the literal is spoken | Define a key and move it into each language |
UNRESOLVED | The reference exists in no language | Check the spelling, or the missing definition |
LOCALE_GAP | Defined for some languages only | Fill in the languages that are missing |
ROLE_WORD | Repeats what the reader already announces | Drop the role word and describe the action |
I run the first two steps before every release, and only the third one occasionally. MISSING and UNRESOLVED are the two that make the script exit non-zero, because they are the ones I can be certain about; the rest are printed and left for me to read. That split has held up well. A checker that blocks a build on a judgement call gets disabled within a month, and a disabled checker finds nothing at all.
One more habit that saved me time: I keep the collector's output committed as a small JSON file. When a label goes missing later, the diff tells me which commit dropped it, without my having to walk the screen again with the reader on.
The question mark on DECORATIVE? is deliberate. Whether an image is decorative is not something a script can decide, so that one line always comes back to a person.
Step Three: Send Only the Wording to Gemini
Nothing is statically wrong with the three that survived. "Save" and "Share" both resolve, and both exist in every language I require. They can still be confusing inside a particular screen, and they can still be too long in a language I do not read well. That judgement is what I hand over.
# judge_labels.py - only labels that passed the static gate reach the model
import json
import os
import sys
from google import genai
from google.genai import types
MODEL = "gemini-3.8-flash"
SCHEMA = {
"type": "object",
"properties": {
"results": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"locale": {"type": "string"},
"verdict": {
"type": "string",
"enum": ["ok", "ambiguous", "duplicate", "too_long", "wrong_register"],
},
"reason": {"type": "string"},
"suggestion": {"type": "string"},
},
"required": ["id", "locale", "verdict", "reason"],
},
}
},
"required": ["results"],
}
INSTRUCTION = """You are reviewing screen-reader labels.
You will receive the labels that appear together on one screen. Judge only on these terms.
- ok: in that language, a listener can tell what will happen
- ambiguous: indistinguishable from another label on the screen, or the target is unclear
- duplicate: effectively the same meaning as another label on the same screen
- too_long: takes more than three seconds to read, and the opening words are not enough
- wrong_register: unnatural in that language, or inconsistent in tone with the screen
Do not discuss label quality in general terms. Compare only within the screen you were given.
Write suggestion only when the verdict is not ok, and write it in that label's language."""
def build_payload(records):
screen = []
for rec in records:
for locale, text in (rec.get("values") or {}).items():
screen.append(
{
"id": f"{rec['platform']}:{rec['icon']}",
"locale": locale,
"role": "button" if rec["interactive"] else "image",
"label": text,
}
)
return screen
if __name__ == "__main__":
path = sys.argv[1] if len(sys.argv) > 1 else "to_judge.json"
records = json.load(open(path, encoding="utf-8"))
payload = build_payload(records)
if not payload:
print("nothing to judge - the static gate settled everything")
raise SystemExit(0)
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
response = client.models.generate_content(
model=MODEL,
contents=json.dumps({"screen": payload}, ensure_ascii=False),
config=types.GenerateContentConfig(
system_instruction=INSTRUCTION,
temperature=0,
response_mime_type="application/json",
response_schema=SCHEMA,
),
)
for item in sorted(json.loads(response.text)["results"], key=lambda r: (r["id"], r["locale"])):
mark = " " if item["verdict"] == "ok" else "> "
print(f"{mark}{item['id']:<26}{item['locale']:<8}{item['verdict']:<14}{item['reason']}")
if item.get("suggestion"):
print(f"{'':<34}suggested: {item['suggestion']}")Deciding what to send took me longer than writing the request. I do not send icon names. Show a model something like ic_share and it starts describing the icon instead of reviewing the label. The other labels on the same screen, on the other hand, always go along — whether a label is confusing is only decidable inside its own screen.
temperature is 0 and the shape is pinned with response_schema. If the verdict names drift, I cannot line this week's output up against last week's, and the whole point is to look at the difference.
What the Label Should Say Is Still Mine to Decide
I took about half of the suggestions as they came back. The rest I rewrote myself.
"Save" reads fine on the list screen, but the detail screen has a second button that would carry the same label, and the two become hard to tell apart. The model offered a longer phrasing. I settled on "Save this wallpaper" instead, because I wanted the purpose to land in the first words rather than the last.
Length is the other thing you cannot notice from one language. A word that takes two beats in Japanese can run considerably longer elsewhere, and a suggestion that reads well on the page can feel slow in the ear. Playing them back, I found that putting the shorter word first got the listener to the point sooner — a small ordering choice that no static check would have raised.
If you want the wider version of this — auditing a whole screen and keeping false positives under control — I wrote that up separately in Automating Accessibility Audits with the Gemini API — A Design That Survives False Positives. Narrowing the scope to labels alone, as I did here, keeps the checker surprisingly short.
The Smallest Thing Worth Doing Today
Turn the screen reader on and walk one screen of your own app from top to bottom. Writing the script can wait until after that. I had no idea those holes were there until I listened to them.
None of this replaces listening. The script tells me where a label is absent, and the model tells me where a phrase is muddy, but neither of them knows that the row of icons in the corner is the part people actually reach for.
Thank you for reading this far. I hope it helps with your first screen.