Folding a Local Gemma 4 into Daily Work — Practical Notes on the Ollama API and Response Speed

Improve perceived response speed

What bothers people most with a local model is the slowness of the very first call. The wait for the model to load into memory feels long, but only that first time. This improves markedly with the keep_alive setting.

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4:e2b",
  "prompt": "ping",
  "keep_alive": "30m"
}'

Specifying a longer keep_alive keeps the model resident in memory, so second and later calls run faster. I recommend slipping in a warm-up call before a burst of repeated requests. If it still feels slow, drop to a smaller model size or keep prompts short. To cap output, limiting the token ceiling with num_predict in options made the wait more predictable.

Isolate the cause of the slowness — the timing fields Ollama returns

A response with stream set to false carries more than the generated text: it also ships a breakdown of where the time went, in nanoseconds. Choosing a fix from "it feels slow" alone, without reading that breakdown, is how you spend an afternoon on a lever that was never connected to anything.

These are the fields worth watching.

I keep one small measurement script on hand that does nothing but convert this breakdown to seconds and line it up.

import json, time, urllib.request
 
NS = 1_000_000_000
 
def measure(prompt: str, model: str = "gemma4:e2b") -> dict:
    payload = json.dumps({"model": model, "prompt": prompt, "stream": False}).encode()
    req = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=payload, headers={"Content-Type": "application/json"},
    )
    started = time.perf_counter()
    with urllib.request.urlopen(req, timeout=300) as res:
        body = json.loads(res.read())
    wall = time.perf_counter() - started
 
    load = body.get("load_duration", 0) / NS
    gen = body.get("eval_duration", 0) / NS
    tokens = body.get("eval_count", 0)
    return {
        "wall_sec": round(wall, 2),
        "load_sec": round(load, 2),
        "load_share": round(load / wall, 2) if wall else 0.0,
        "tokens": tokens,
        "tok_per_sec": round(tokens / gen, 1) if gen else 0.0,
    }
 
if __name__ == "__main__":
    for i in range(3):
        print(i, measure("List three benefits of this feature as bullet points"))

The trick is to send the same prompt three times in a row. The first call shows a large load_share, and from the second call it drops to near zero. If load_sec is still there on the third run, either keep_alive is not taking effect, or another model's call has evicted yours from the resident slot. The reverse case matters just as much: if load_sec is near zero while wall_sec stays long, the cost is in generation, and stretching keep_alive will not move it at all.

tok_per_sec becomes your yardstick when comparing models and sizes. For each model I was considering, I ran this three-call loop exactly once and wrote load_sec and tok_per_sec into a local note. Since then I decide "which model gets this job" by reading that note rather than by guessing. You measure once; you reuse the judgment many times.

Switch between local and cloud automatically

A local Gemma 4 is plenty for light tasks, but on involved instructions or work needing the latest information, it can feel underpowered. So a two-tier setup — try local first, escape to the cloud Gemini API only on failure or insufficient quality — holds down cost while preventing dropped work.

def ask(prompt: str) -> str:
    try:
        out = ask_local(prompt)
        if out and len(out.strip()) > 20:   # reject clearly empty or too-short replies
            return out
    except Exception:
        pass
    # if local is unwell or insufficient, fall back to the cloud
    return ask_gemini_cloud(prompt)   # a separate implementation calling the Gemini API

The axis here is simple: judge whether the local reply is "usable" with an easy condition, and escalate to the cloud if not. In my operation, I throw a few rough options locally first and leave only the final polish to the cloud. Automating this switch saves me from stopping at every decision. If you fold it into a production script, assume the cloud side can also fail and prepare a path that always returns something in the end.

Leaving the accept-or-escalate decision to length alone, though, lets failures through. A lightweight model can hit its token ceiling and return output cut off mid-sentence, and that sails past a length check. Looking at how the text ends alongside how long it is raises the hit rate. While you are there, write one line recording which path the call took, so you can count the effect later.

import json, time, pathlib
 
LOG = pathlib.Path("ask_route.jsonl")
 
def looks_usable(out: str) -> bool:
    s = (out or "").strip()
    if len(s) < 20:
        return False
    return not s.endswith((",", ";", "and", "…"))   # reject output cut off mid-sentence
 
def ask(prompt: str) -> str:
    route = "local"
    try:
        out = ask_local(prompt)
        if not looks_usable(out):
            route, out = "cloud", ask_gemini_cloud(prompt)
    except Exception:
        route, out = "cloud", ask_gemini_cloud(prompt)
    with LOG.open("a", encoding="utf-8") as f:
        f.write(json.dumps({"ts": time.time(), "route": route, "chars": len(out)}) + "\n")
    return out

That single line is enough to answer "what share is local actually handling" after the fact. A falling share is your cue to revisit either how the prompt is written or which model you picked. The opposite reading is just as useful: a high share paired with growing manual cleanup means the acceptance test is too lenient. Only once the numbers existed did the two-tier setup become something I could tune rather than merely hope about.

Three pitfalls people stumble on

From actually wiring it in, here are three cautions I hit.

Mistaking the first download's wait for a failure

Fetching a model can take minutes depending on your connection. While progress advances it is fine, so wait for the success line. When calling from a script for the first time, note that an un-fetched model makes that first call extremely long. Dropping it ahead of time with ollama run is the workaround.

Memory pressure making responses crawl

Stack too large a model and you run out of memory, swapping kicks in, and responses turn heavy at once. The remedy is to limit which models stay resident and to avoid sizes excessive for the task. Not using a large model for work a small variant handles paid off.

Unstable output formatting

A lightweight model can stray from the output format you specified. When you want bullets or JSON, show the format concretely in the prompt and lightly validate on the receiving side too. Not leaving formatting entirely to the model and adding minimal post-processing on the script side keeps downstream steps stable.

How far to leave to local

Finally, the line. A local Gemma 4 suits drafts where you want more trials, consulting on notes you would rather not send out, and light work in places with shaky connectivity. Conversely, work needing the latest information or a huge context steadies up if you escalate to the cloud rather than forcing it locally.

In my case, automating this division let me cut API cost while adding trials. Mass-produce options freely on your own machine, and entrust only the decisive polish to the cloud. For using tools over the long haul as an indie developer, this two-tier setup feels the most realistic. Start by adding the single-call wrapper from today to the script from the previous article.