Finding the questions your help docs never answer, by asking Gemini to write the quiz

Implementing the generation pass

Google's Python SDK plus structured output, with a fixed schema so the downstream stage cannot silently break.

# pip install google-genai pydantic
from google import genai
from pydantic import BaseModel
from typing import List
 
client = genai.Client(api_key="YOUR_API_KEY")
MODEL = "gemini-3.6-flash"
 
 
class Question(BaseModel):
    question_id: str      # stable key for matching against the judgment pass
    text: str             # phrased the way a user would phrase it
    user_words: List[str] # the user-side vocabulary it contains
 
 
class QuestionSet(BaseModel):
    questions: List[Question]
 
 
GEN_PROMPT = """You are a first-time user of this app.
Read the help text below and write 8 questions you would realistically send to support.
 
Constraints:
- Do not reuse the headings from the help text. Rephrase in everyday language
- At least half should start from a symptom: "won't", "disappeared", "can't find"
- Include questions the text probably does not answer, if any come to mind
- Number question_id from q01 through q08
 
Help text:
---
{doc}
---"""
 
 
def generate_questions(doc_text: str) -> QuestionSet:
    res = client.models.generate_content(
        model=MODEL,
        contents=GEN_PROMPT.format(doc=doc_text),
        config={
            "response_mime_type": "application/json",
            "response_schema": QuestionSet,
        },
    )
    return QuestionSet.model_validate_json(res.text)
 
 
# Expected output (excerpt):
# questions=[Question(question_id='q01',
#                     text="I saved a wallpaper but it isn't in my photos",
#                     user_words=['saved', 'not there', 'photos']), ...]

The line that earns its keep is "do not reuse the headings." Drop it and the generated questions become paraphrased table-of-contents entries, the judgment pass returns answered across the board, and you learn nothing.

Seeding the prompt with a handful of real review texts helps too. The vocabulary users actually reach for is blunter than anything you would invent for them.

Implementing the judgment pass

The judge sees the questions and the document, nothing else. Outside knowledge is explicitly forbidden and a verbatim quote is mandatory for any positive verdict. Requiring the quote noticeably cuts down on the model drifting toward a comfortable answered.

class Verdict(BaseModel):
    question_id: str
    verdict: str      # "answered" | "partial" | "missing"
    quote: str        # verbatim span from the document, empty if none
    reason: str       # under 20 words
 
 
class VerdictSet(BaseModel):
    verdicts: List[Verdict]
 
 
JUDGE_PROMPT = """Using only the document below as evidence, decide whether each question can be answered.
 
Verdicts:
- answered: one passage in the document settles it
- partial: evidence exists but requires combining separate passages
- missing: the document contains no basis for an answer
 
Rules:
- Never use knowledge from outside the document
- For answered and partial, put a verbatim span from the document in quote
- Always echo back the same question_id
 
Document:
---
{doc}
---
 
Questions:
{questions}"""
 
 
def judge(doc_text: str, qs: QuestionSet) -> VerdictSet:
    listed = "\n".join(f"{q.question_id}: {q.text}" for q in qs.questions)
    res = client.models.generate_content(
        model=MODEL,
        contents=JUDGE_PROMPT.format(doc=doc_text, questions=listed),
        config={
            "response_mime_type": "application/json",
            "response_schema": VerdictSet,
            "system_instruction": "You are a document reviewer. Never fill in what the document does not say.",
        },
    )
    vs = VerdictSet.model_validate_json(res.text)
 
    # Detect dropped items. A short response goes back for a retry
    got = {v.question_id for v in vs.verdicts}
    missing_ids = [q.question_id for q in qs.questions if q.question_id not in got]
    if missing_ids:
        raise ValueError(f"verdict missing for: {missing_ids}")
    return vs

Skip the question_id reconciliation and eight questions can come back as seven verdicts with the eighth quietly gone. A call whose counts do not line up is safer retried, or discarded along with that document's result. I have written up how structured output drifts in production in when Gemini structured output quietly leaves the schema and guarding Gemini API responses with schema validation.

Writing the replies taught me the shortfall was vocabulary, not explanation

The idea came out of the reply work itself.

For the recurring question my first read was "the explanation must be too thin," so I added paragraphs to the help page. The tickets kept coming. The next month, the same phrasing landed again.

Writing replies one at a time, the reason surfaces slowly. The words a user reaches for and the words in my headings simply do not overlap. I wrote "save location." They write "where did it go." Adding explanation without adding a path to that explanation changes nothing.

Which is why the value of this device sits in the partial results more than the missing ones. A partial means the evidence is there but no single stop delivers it — a defect you fix by renaming a heading and consolidating two sections, not by writing more. Less work, faster payoff.

I prefer running the audit right after a batch of review replies rather than before a feature launch. The user vocabulary is still fresh, which makes the generated questions much easier to judge for realism.

If you are running replies across several languages, I covered keeping the tone consistent in building multilingual Google Play review replies with Gemini.

Judging one question at a time costs 7x the input tokens

How you batch the calls moves the bill visibly. Assume 40 help documents, roughly 1,800 characters of body text each, 8 questions per document. Japanese runs close to one token per character, and the rates used here are Gemini 3.6 Flash at $1.50 per 1M input tokens and $7.50 per 1M output tokens.

The gap is about 7.0x, entirely on the input side. In the judgment pass the document dominates, and sending it per question means paying for the same body text eight times over. Batched, it goes across once.

Batching has a cost of its own. Later questions in a batch tend to drift toward whatever the earlier ones were judged, so verdicts clump. Two guards keep that in check:

1. Batch only within a single document. Never pack multiple documents into one call
2. Require question_id on every verdict and reconcile by ID, never by position

If your setup genuinely has to resend the same body text repeatedly, context caching is the cleaner answer. I laid out how to decide in cutting Gemini API costs with context caching.

Keep a ledger and only re-run what changed

A one-off audit is not worth much. The loop closes when you edit the page and confirm the same question has moved to answered.

import hashlib
import json
from pathlib import Path
 
LEDGER = Path("docs_answerability.jsonl")
 
 
def doc_fingerprint(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
 
 
def load_ledger() -> dict:
    if not LEDGER.exists():
        return {}
    rows = {}
    for line in LEDGER.read_text(encoding="utf-8").splitlines():
        if line.strip():
            row = json.loads(line)
            rows[row["doc"]] = row
    return rows
 
 
def run(doc_paths):
    ledger = load_ledger()
    out = []
    for path in doc_paths:
        text = Path(path).read_text(encoding="utf-8")
        fp = doc_fingerprint(text)
        prev = ledger.get(path)
        if prev and prev["fingerprint"] == fp:
            continue  # unchanged documents stay off the bill
 
        qs = generate_questions(text)
        vs = judge(text, qs)
        gaps = [v for v in vs.verdicts if v.verdict != "answered"]
        out.append({
            "doc": path,
            "fingerprint": fp,
            "total": len(vs.verdicts),
            "gaps": [
                {"id": v.question_id, "verdict": v.verdict, "reason": v.reason}
                for v in gaps
            ],
            "prev_gap_count": len(prev["gaps"]) if prev else None,
        })
 
    with LEDGER.open("a", encoding="utf-8") as f:
        for row in out:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")
    return out
 
 
# Expected output (excerpt):
# [{'doc': 'help/en/save-wallpaper.md', 'fingerprint': '9f1c...', 'total': 8,
#   'gaps': [{'id': 'q03', 'verdict': 'partial', 'reason': 'save path lives in another section'}],
#   'prev_gap_count': 3}]

Carrying prev_gap_count makes it obvious when a page you thought you fixed still has the same number of holes. In my own rotation about five documents change in a month, which under the batched layout works out to roughly $0.08 monthly. Cost is not the reason to skip this.

Drop the fingerprint check and you pay full price on unchanged documents every run. That is not a line worth cutting.

Where to stop handing work to the model

All this device returns is the fact that a question went unanswered. Deciding what to rewrite stays with you. My split:

• Model's job: generating the questions, judging reachability, aggregating gaps per document, diffing against last time
• My job: renaming headings, merging sections, and deciding whether the product itself is what needs fixing

A cluster of missing verdicts is also worth pausing on before you start writing. An operation that only the help page can rescue usually points at a rough edge in the app. For an indie developer, prose looks like the cheap fix, and then you pay for it monthly in review replies.

Start with a single help page

There is no need to point this at everything at once. Take the page behind your most frequent ticket, generate the eight questions, and read them yourself. If they do not read like sentences a real user would type, the constraints in your generation prompt are too loose — and tightening those is the first real piece of work.

I am still undecided about the right granularity for the audit. If you run something similar, I would genuinely like to hear how you handle it. Thank you for reading this far.