A Provenance Gate That Never Asks Gemini to Name the Country

At full-frame scale, those marks do not survive as pixels

Having decided to rely on physical marks, I ran into the next problem. Signatures and censor seals occupy a very small region of the frame.

A typical source scan here is around 1200×1800. If a censor seal fits inside roughly 60 by 60 pixels, that is about 0.17% of the frame by area. Downscale to a long edge of around 768 pixels before the call, and the seal drops to roughly 25 pixels on a side. The outline survives. The characters do not.

Instruct a model to base its answer on something unreadable and it will still try to comply. Compliance, in that situation, produces a plausible-sounding invented reading. That, I suspect, was the source of the confident answers I saw at the start.

The fix is unglamorous: crop before you ask. I build a sheet of the four corners and the margins at native resolution and send that sheet. The full image still goes along for compositional context, but the mark reading is answered from the crops alone. I wrote separately about the resolution and token trade-off in controlling image tokens by resolution for batch classification.

This gate asks for character-reading precision rather than reasoning depth, so the call goes to gemini-3.1-pro. Flash models handle the higher-volume preprocessing earlier in the same pipeline.

Ask for what was seen, not for a verdict

That reasoning drops directly into a schema. What comes back is not a country name or a boolean. It is a list of marks.

import json
import time
from pathlib import Path
 
from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_API_KEY")
 
MARK_SCHEMA = {
    "type": "object",
    "properties": {
        "marks": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    # Kinds of visible marks only. No verdict words such as
                    # "japanese" or "chinese" belong in this enum.
                    "kind": {
                        "type": "string",
                        "enum": [
                            "kana_script",
                            "publisher_seal",
                            "censor_seal",
                            "artist_signature",
                            "red_seal_large",
                            "long_colophon",
                            "latin_botanical_name",
                        ],
                    },
                    "where": {"type": "string"},           # which tile of the crop sheet
                    "transcription": {"type": "string"},   # empty when unreadable
                    "legibility": {
                        "type": "string",
                        "enum": ["legible", "partial", "illegible"],
                    },
                },
                "required": ["kind", "where", "legibility"],
            },
        },
        # True when nothing at all is visible. Keeps this case distinct
        # from an empty array caused by a missed reading.
        "no_marks_visible": {"type": "boolean"},
    },
    "required": ["marks", "no_marks_visible"],
}
 
PROMPT = """This image is a contact sheet of the four corners and margins of a
single woodblock print, cropped at native resolution.
 
List only what is physically present on the sheet.
 
- Do not write anything inferred from the subject matter or style
- If characters are too degraded to read, set legibility to illegible and
  leave transcription empty
- If no marks are present at all, return an empty marks array and set
  no_marks_visible to true
- Do not judge which country the work is from
"""
 
 
def read_marks(sheet_path: Path, retries: int = 3) -> dict:
    image = types.Part.from_bytes(
        data=sheet_path.read_bytes(),
        mime_type="image/jpeg",
    )
    last_error = None
    for attempt in range(retries):
        try:
            response = client.models.generate_content(
                model="gemini-3.1-pro",
                contents=[image, PROMPT],
                config=types.GenerateContentConfig(
                    response_mime_type="application/json",
                    response_schema=MARK_SCHEMA,
                    # Reading marks yields short output. A long response is
                    # itself a signal that the model started inventing.
                    max_output_tokens=1024,
                ),
            )
            return json.loads(response.text)
        except json.JSONDecodeError as e:
            # Even with a schema, truncated JSON occasionally comes back.
            # Do not swallow this silently.
            last_error = e
        except Exception as e:
            last_error = e
            time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed to read marks from {sheet_path.name}: {last_error}")

The critical detail is that the kind enum contains no verdict vocabulary. I tried a version with an option like japanese_style in the list. The moment it was there, the model stopped reading marks and started filling the array from stylistic impression. Vocabulary invites judgment.

no_marks_visible exists as a separate field because an empty array is ambiguous. Either nothing was there, or something was there and the model missed it. The first is a valid observation; the second is a failure. Making the model state which one it means lets the next stage tell them apart.

The verdict comes from a table on my side, not from the model

The observed marks feed into a decision that preserves the asymmetry described earlier. This part is ordinary code, with no model involved.

from dataclasses import dataclass
 
JAPANESE_MARKS = {"kana_script", "publisher_seal", "censor_seal"}
SUSPECT_MARKS = {"red_seal_large", "long_colophon"}
OUT_OF_SCOPE_MARKS = {"latin_botanical_name"}
 
 
@dataclass
class Verdict:
    route: str      # "accept" / "hold" / "reject"
    reason: str
 
 
def decide(observation: dict) -> Verdict:
    marks = observation["marks"]
    # An illegible mark tells us only that something is there.
    # It cannot serve as grounds for acceptance.
    usable = {m["kind"] for m in marks if m["legibility"] in ("legible", "partial")}
    seen = {m["kind"] for m in marks}
 
    if usable & OUT_OF_SCOPE_MARKS:
        return Verdict("reject", "Not a woodblock print of the target kind")
 
    if usable & JAPANESE_MARKS:
        # One readable Japanese mark settles it, even alongside suspect features.
        found = sorted(usable & JAPANESE_MARKS)
        return Verdict("accept", f"Japanese marks confirmed ({found})")
 
    if seen & SUSPECT_MARKS:
        return Verdict("hold", "Suspect features present with no readable Japanese mark")
 
    if observation["no_marks_visible"]:
        return Verdict("hold", "Likely a partial enlargement with no provenance marks in frame")
 
    return Verdict("hold", "Marks are present but could not be read")

Splitting usable from seen is the part that earned its place in production. An illegible mark cannot support acceptance, because accepting on that basis means treating an unread seal as read. It can, however, support a hold. A source image covered in unreadable seals is exactly the kind a person should look at.

The practical consequence is that this gate accepts only when a Japanese mark was actually read. Everything else holds. That sounds severe, and it turned out not to be a problem, for reasons in the section after next.

I stopped asking for a confidence score

The first version had the model return a confidence between 0 and 1, with a threshold splitting accept from hold. It is the obvious design, and it ran that way for a while.

I dropped it not because of accuracy but because the output could not be checked.

When 0.82 comes back, my only options are to believe it or not. Too conservative, so lower the threshold. An error slipped through, so raise it. Each adjustment meant reviewing everything again, and before long I could not articulate what I was actually tuning against.

With mark listing, the response is a claim of the form "censor seal in the lower-right tile, legible, reads as such and such." Open that tile on the crop sheet and I can verify the claim in seconds. The model's output became falsifiable.

That change was not what I expected. I had been editing the schema to improve accuracy. What actually improved was the time it took to notice a mistake. I doubt the raw error count fell dramatically. The errors simply stopped hiding.

Losing threshold tuning was a quieter win but a real one. The decision table reads as code, so changing policy means editing the table. The cycle of nudging a number and inferring behavior from the results disappeared entirely.

Putting held items somewhere you can actually verify them

The first thing that tripped me up was not the model output but where held items went.

Coming back to a held image later, I could no longer tell which region of the original a claim like "censor seal in the lower-right tile" referred to. Rebuilding the crop sheet recovers it, but that overhead compounds, and the hold pile stops getting opened. I had gone to the trouble of making the output verifiable and then stored it somewhere inconvenient to verify.

The workaround was to persist three things alongside every verdict.

1. The crop sheet filename, plus a map of which coordinates in the original each tile was cut from
2. The raw observation the model returned, kept as a list of marks rather than the post-decision conclusion
3. The reason string explaining which branch of the decision table produced the outcome

The third matters more than it looks. Without it, revisiting an item weeks later leaves no way to reconstruct why it was held. That is why decide() carries a reason on Verdict.

In production, the hold folder also needs to sit somewhere you will pass through anyway. Mine lives outside the intake directory, on the path taken before assembling the next batch.

Holding is not a failure, it is the third output

Alongside accept and reject, this gate has a hold path built in from the start. Held images move to a _hold/ folder in the source directory and wait for a person.

I first read a high hold rate as a design failure. I now read it the other way. Images land in hold because the image alone cannot settle the question.

Consider a source that is a partial enlargement of a print. The title slip and signature are outside the frame. No amount of resolution recovers something that was never captured. A model that returns an answer for that input is guessing, not reading.

And most held images resolve through information outside the image. If the source site has a work page, the artist and date are printed right there. Capture that metadata at acquisition time and provenance is settled without any image judgment at all.

Writing the order down removes hesitation at implementation time. I would recommend routing only metadata-less images into image judgment at all. My earlier pipeline pushed everything through one path, running image judgment even on images whose metadata I already had. I was asking a model to guess at something I could simply look up.

The abstain path had to come before the accuracy work

The whole rebuild reduces to one move: demoting the model from judge to observer.

A judge is asked to produce a verdict, and will produce one, with or without visible grounds. An observer states what is visible, and can say that nothing is. That difference is exactly the difference between catching a problem before distribution and not.

The same shape applies broadly to any stage that infers attributes from images. I wrote about filtering visually similar images in building a near-duplicate gate before publishing, but that case differs: similarity is itself expressible as a number. For attributes like provenance, where the answer may live outside the image entirely, building the abstain path first is the faster route.

If you are building an image attribute stage right now, check whether any verdict vocabulary has crept into your schema enum. A single such word is enough to make the model stop reading and start answering. That was the turning point for me.