Wiring a Security-Focused Model Into a Solo Developer's Audit — The Extraction Layer and Fingerprints That Stop Re-Reporting

Own the extraction layer

Write this layer yourself. Outsourcing it costs you both money and reproducibility.

Python's ast module is enough to find outbound-request sinks whose first argument traces back to request-derived input.

# scan_sinks.py — extract outbound-request sinks and fingerprint them
import ast, hashlib, json, sys
from pathlib import Path
 
SINKS = {
    ("requests", "get"), ("requests", "post"),
    ("httpx", "get"), ("httpx", "post"),
    ("urllib.request", "urlopen"),
}
TAINT_ROOTS = ("request",)
 
 
def dotted(node):
    if isinstance(node, ast.Attribute):
        left = dotted(node.value)
        return f"{left}.{node.attr}" if left else node.attr
    if isinstance(node, ast.Name):
        return node.id
    return ""
 
 
def taint_key(node):
    """Normalise request.args.get("src") and request.json["url"] into a stable key."""
    if isinstance(node, ast.Call):
        base = dotted(node.func)
        if base.startswith(TAINT_ROOTS) and node.args and isinstance(node.args[0], ast.Constant):
            return f"{base}({node.args[0].value})"
        return None
    if isinstance(node, ast.Subscript):
        base = dotted(node.value)
        if base.startswith(TAINT_ROOTS) and isinstance(node.slice, ast.Constant):
            return f"{base}[{node.slice.value}]"
        return None
    if isinstance(node, ast.Attribute):
        base = dotted(node)
        return base if base.startswith(TAINT_ROOTS) else None
    return None
 
 
class FnScanner(ast.NodeVisitor):
    def __init__(self, relpath):
        self.relpath = relpath
        self.findings = []
 
    def visit_FunctionDef(self, fn):
        env = {}  # local name -> taint key
        for node in ast.walk(fn):
            if isinstance(node, ast.Assign):
                key = taint_key(node.value)
                if key:
                    for t in node.targets:
                        if isinstance(t, ast.Name):
                            env[t.id] = key
        for node in ast.walk(fn):
            if not isinstance(node, ast.Call):
                continue
            fq = dotted(node.func)
            if "." not in fq or not node.args:
                continue
            mod, _, fname = fq.rpartition(".")
            if (mod, fname) not in SINKS:
                continue
            arg = node.args[0]
            local = arg.id if isinstance(arg, ast.Name) else ""
            src = env.get(local) if local else taint_key(arg)
            if src is None:
                continue
            self.findings.append({
                "file": self.relpath, "line": node.lineno,
                "sink": f"{mod}.{fname}", "source": src,
                "local": local, "fn": fn.name,
            })
        self.generic_visit(fn)
 
 
def h(*parts):
    return hashlib.sha1("|".join(parts).encode()).hexdigest()[:12]
 
 
def scan(root: Path):
    out = []
    for p in sorted(root.rglob("*.py")):
        rel = str(p.relative_to(root))
        s = FnScanner(rel)
        s.visit(ast.parse(p.read_text()))
        for f in s.findings:
            f["fp_line"] = h(f["file"], str(f["line"]), f["sink"])        # scheme A
            f["fp_v1"] = h(f["file"], f["fn"], f["sink"], f["local"])     # scheme B
            f["fp_v2"] = h(f["sink"], f["source"], f["fn"])               # scheme C
            f["fp_v3"] = h(f["sink"], f["source"])                        # scheme D
            out.append(f)
    return out
 
 
if __name__ == "__main__":
    print(json.dumps(scan(Path(sys.argv[1])), ensure_ascii=False, indent=2))

Taint is followed exactly one hop, and that is deliberate rather than lazy. A two-hop pattern — payload = request.json followed by target = payload["callback_url"] — slips through. On my fixture of 4 files containing 9 sinks, the scanner found 8 and missed that two-hop case, for a recall of 88.9%.

Deepening the taint analysis to close that gap immediately widens the candidate set feeding the model. A slightly leaky extractor that runs fast and never argues with you turned out to be the version I could actually keep running. I close gaps by adding sink definitions, not by chasing longer taint chains.

Throughput, measured against real Python trees:

Four thousand files in under thirty seconds is cheap enough to run on every CI job. Putting it in front of the model drops what reaches the hypothesis layer to a few dozen candidates.

The ledger broke on day one because I keyed it by line number

Findings need identifiers. Without a stable key, everything is new every day and the ledger means nothing.

I started with path + line + sink. Scheme A. The next morning I added a module docstring, two TODO comments, and one logging helper — and every entry in the ledger was orphaned.

So I switched to path + function + sink + local variable. Scheme B. Line numbers were gone, so I assumed I was safe. Then I renamed src to image_url and hook to endpoint, and scheme B collapsed just as completely. The local variable name carries no meaning, and I had baked it into the identity.

What finally held was the triple sink + taint source + enclosing function. Scheme C. However the local variable is spelled, request.args.get(src) stays request.args.get(src).

Here is the fixture rescanned after adding comments, adding a helper function, and renaming local variables:

Scheme B failing as badly as scheme A was the result that stung. I had assumed line numbers were the only volatile ingredient and never asked what else in my key moves for reasons unrelated to meaning. A fingerprint may only contain things that change when the meaning changes — never things that change when the author changes their mind about naming.

Coarser is more stable, and that is exactly the trap

This is the part that ran against my expectations.

Scheme C still contains the function name, so renaming a function breaks it. I renamed api/proxy.py to api/gateway.py and api_proxy to proxy_get: 6 of 8 findings survived, 2 arrived as new.

Dropping the function name to leave sink + taint source looks like the obvious fix. Scheme D. Measured, it tracks 7 of 7 — flawless.

The interesting question is why 8 findings became 7.

collision: 97d2524cdca2 -> ['services/thumbnail.py:8  fetch_remote_thumbnail',
                            'services/thumbnail.py:16 fetch_allowlisted_thumbnail']

One file held two functions calling requests.get with request.args.get("src"). One fetches the URL directly. The other checks the host against an allowlist first. Scheme D flattens them into a single identity.

Review the safe one, mark it handled, and the vulnerable one never surfaces again. Tracking went up; what disappeared was the finding that mattered.

Fingerprint design pulls you toward stability, but stability and discriminative power trade against each other. I chose discrimination, kept scheme C, and absorbed rename churn a different way: the ledger carries an aliases array, and when I rename a function I add the old fingerprint by hand. That happens a few times a year. One manual line beats an automatic merge that silently unifies two findings that were never the same.

Acceptance is decided by reproduction, not by wording

The hypothesis layer returns exactly two things:

1. The smallest test that reproduces the failure
2. A patch that makes that test pass

Acceptance means the test fails before the patch and passes after it. The model's prose is never read by the gate.

# verify_gate.py — accept only on a fail-then-pass transition
import subprocess, tempfile, pathlib
 
def run(cmd, cwd):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True).returncode
 
def verify(repo: pathlib.Path, repro_test: str, patch: str) -> dict:
    """Write to the ledger only when accepted is True."""
    test_path = repo / "tests" / "test_repro_generated.py"
    test_path.parent.mkdir(exist_ok=True)
    test_path.write_text(repro_test)
 
    before = run(["python", "-m", "pytest", "-q", str(test_path)], repo)
    if before == 0:
        # A repro that already passes has not reproduced anything
        test_path.unlink()
        return {"accepted": False, "reason": "repro_did_not_fail"}
 
    with tempfile.NamedTemporaryFile("w", suffix=".patch", delete=False) as fp:
        fp.write(patch)
        patch_file = fp.name
 
    if run(["git", "apply", "--check", patch_file], repo) != 0:
        test_path.unlink()
        return {"accepted": False, "reason": "patch_does_not_apply"}
 
    run(["git", "apply", patch_file], repo)
    after = run(["python", "-m", "pytest", "-q", str(test_path)], repo)
    suite = run(["python", "-m", "pytest", "-q"], repo)
    run(["git", "checkout", "--", "."], repo)   # always restore
    test_path.unlink(missing_ok=True)
 
    return {
        "accepted": after == 0 and suite == 0,
        "reason": "ok" if after == 0 and suite == 0 else f"after={after} suite={suite}",
    }

The before == 0 branch earns its keep more than anything else here. A reproduction that passes before the fix is proof that nothing was reproduced, however confidently the explanation was written.

Always restoring with git checkout -- . matters just as much. I originally placed that call where an exception could skip it, left a patch applied, and started the next verification from a dirty tree. That is when I moved the whole gate onto a throwaway clone rather than the working repository. The same instinct — never carry a broken state into the next run — shows up in Self-Healing Architecture for Gemini Computer Use.

What the ledger holds, and what it deliberately does not

Model prose and confidence scores are intentionally absent. If they are stored, the next person to read the ledger will reason from them. The only admissible basis is the test sitting in evidence.

wontfix requires a one-line human reason. Leave that field optional and the suppression list quietly becomes a graveyard for findings nobody read.

Porting this to another stack

1. Define 3 to 5 sinks and get the extraction layer running. No model calls yet
2. Fingerprint with scheme C and do nothing but rescan for two weeks, watching whether re-reports converge to zero
3. Only after they converge, connect the hypothesis layer
4. Write to the ledger only past the verify gate, and make the wontfix reason mandatory

Reverse that order and findings pile up before the ledger stabilises, which is precisely how my first attempt died at the two-week mark. With extraction and fingerprinting settled, the model becomes a component you can replace at will — which is what makes each new purpose-built model an upgrade rather than a rebuild.

Back to the Agent Studio notice: the useful move was never to enumerate apps built before a given date. It was to reach a state where a machine can tell me how many /api-proxy-shaped paths exist right now. With the extractor in place, the next notice of this kind takes about fifteen seconds to answer.

Run scan_sinks.py against one of your own repositories and look at the count. Mine came back with more than twice what I expected, which was a quiet and unwelcome surprise.

Thank you for reading through a fairly long set of notes — I hope some of it saves you an afternoon.