I Wanted to Build a Tool-Calling Repair Proxy. My Own Benchmark Killed the Idea

After the last post — that brutal fight with xl-ai and a local Gemma — I walked away with one conclusion burned into my brain: local models simply can't do tool calling. Gemma 3 27B couldn't even emit stable JSON, and any library that "expects the model to politely speak tool-call format" (like xl-ai) exploded on contact. I only salvaged it by hand-building a backend adapter.

And then a lightbulb went off: if everyone building local AI hits this same wall, wouldn't a "tool-calling repair proxy" be an open market?

The idea was simple: an OpenAI-compatible middle layer sitting between your app and the local model. The app sends standard tools requests as usual; the proxy translates tool definitions into prompts the model can chew on, fishes the broken JSON out of the response and repairs it, validates against the schema, and auto-retries on failure — making a model that can't do tool calling look, from the outside, like one that can.

Sounded beautiful. Everyone wiring local models into Vercel AI SDK / LangChain complains about exactly this, and no standalone repair layer existed.

But this time I'd learned my lesson. Before writing any product code, spend one day building a benchmark to validate the assumption.

That benchmark personally executed my idea. This post is the full autopsy report — plus a practical scoreboard for picking a local tool-calling model in June 2026.

(The reason I bothered to validate first was the lesson from that last war: a library's beautiful demo always assumes a strong backend. This time I was about to become the library author — so I'd better confirm that the "weak backend" my whole idea assumed actually exists.)

How the benchmark works#

25 test cases, six categories, all shapes you actually hit in real products:

Each model runs two conditions:

• native: the standard OpenAI-compatible API with the tools parameter — i.e., what Vercel AI SDK and LangChain actually send today.
• shim: a simulated "repair proxy" — no tools param; a system prompt demands JSON-only output, then the response goes through fence stripping, <think>-block removal, balanced-brace JSON extraction, jsonrepair, ajv schema validation, and one retry with error feedback.

🟨jsCopy
// The shim's core: extract JSON (strip think blocks, strip fences,
// balanced-brace scan) → repair → validate → retry
function parseShimOutput(text) {
  const raw = extractJson(text)        // find the first balanced {...}
  let obj
  try {
    obj = JSON.parse(raw)
  } catch {
    obj = JSON.parse(jsonrepair(raw))  // the ER for broken JSON
  }
  // ...followed by ajv schema validation; on failure, retry once with the error message
}

Scoring is not "any call counts": the tool name must be right, arguments must pass ajv validation against the declared schema, content checks must pass (did the email really go to bob@example.com), parallel cases must emit the full set of calls, and a single call on a nocall case is instant death. temperature: 0, everything on one RTX 5060 Ti 16GB via Ollama, default Q4 quants.

(Full disclosure: 25 cases, single pass, Q4 quants — this is a smoke test, not an academic paper. But once you see the numbers below you'll agree: the gap between 0/25 and 25/25 is not something noise explains.)

The scoreboard#

Sorted by native score:

(✨ = released May–June 2026, pulled and tested the week of writing)

Finding 1: the last war wasn't my fault — Gemma 3 really scores zero#

Start from the bottom two rows. Gemma 3's native score is 0/25 — and not because it answered wrong, but because Ollama's API straight-up returns 400:

registry.ollama.ai/library/gemma3:12b does not support tools

The model's chat template has no concept of tools at all. So that war I fought with xl-ai in the last post was unwinnable from day one: it wasn't that my integration was bad — the thing literally didn't exist at the API level. Seeing that 0/25 felt like long-overdue vindication.

And the shim lifts gemma3:12b from 0 to 23/25. In other words — my repair proxy idea was completely valid on the 2025 model generation. Prompt strategy + JSON repair + retry really can make a model whose API rejects tools outright score 92%.

If I had run this benchmark in mid-2025, I'd probably have started building.

Finding 2: the generation gap — the window closed the day Qwen3.5 shipped#

Now look at the top two rows.

qwen3.5:4b. Native 25/25. Perfect. A 4B model. 3.4GB. Runs effortlessly on my middle-class GPU.

Not "roughly usable" — all 25 correct: every nested schema, every HTML-in-JSON case (including the one that must preserve a <b>final</b> tag), parallel when it should, silent when it should, four-way tool selection all correct. I stared at the results JSON trying to find one failure. There isn't one.

The within-family comparisons are even more brutal:

• qwen3 → qwen3.5 (both 4B): 16/25 → 25/25
• gemma3 → gemma4 (both 12B): 0/25 → 18/25 (from "API rejects it" to "mostly works")

Within one year, native tool calling went from "the collective disability of local models" to "table stakes for the new generation." That line I wrote in the last post — "our local Gemma simply can't do stable tool calling" — was true of 2025 models and is stale intel for 2026 ones.

Finding 3: a repair layer actively hurts the new models#

This is the second knife into the proxy idea, and it cuts deeper.

Look at qwen3.5:4b: native 25/25, shim 22/25. The same model, wrapped in my "repair layer," drops three cases. granite4:tiny-h is worse: 22/25 chopped to 12/25.

The reason isn't mysterious: these new models were deeply aligned to the official tool-calling format during training (their own chat templates, their own special tokens). My shim translates tool definitions into a homespun "respond with JSON only" prompt — like forcing someone with formal training to speak a dialect I invented. Its native ability gets disrupted by my "help."

For the proxy's value proposition, this is fatal:

• On old models (gemma3): the shim wins huge, 0 → 23. But why wouldn't you just ollama pull qwen3.5?
• On new models (qwen3.5): the shim is a net negative.

The repair proxy's only habitat is users locked onto old models, who need tool calling, and who can't switch models. That intersection is too small to support a project.

Finding 4: case studies — every model dies (or lives) in its own way#

deepseek-r1:8b's split personality: native 4/25 (it emits almost no native tool calls — after reasoning, it just answers in prose), shim 23/25 (ask it for JSON and it obliges, with high quality). Thinking models are the shim condition's biggest beneficiaries — once you strip the <think> block, the JSON inside is clean.

command-r7b and glm4:9b score by silence: native 4/25 each, and all four points come from the nocall category — meaning they never emitted a single tool call all benchmark, and collected free points on the "should NOT call" cases by being mute. They wear the tools tag as decoration.

HTML-in-JSON is the Gemma family curse: all three gemma4 sizes score 0/4 on the html category; the typical failure is schema validation catching a missing type field in operations[0] — the structure falls apart under HTML-escaping pressure. This is exactly the xl-ai scenario: even if you wire gemma4 into BlockNote, the blood and tears replay. qwen3.5 scores 4/4 on the same cases. Pick the wrong model and no amount of framework glue saves you.

mistral-nemo, old soldiers never die: a mid-2024 12B veteran scoring native 22/25. Mistral made function calling a first-class training target back then, and two years later it still arm-wrestles the new generation. Whether tool calling was a training objective matters more than parameter count.

The June 2026 recruits: an entire generation lined up against the idea#

The week I wrote this, I pulled every May–June 2026 model on the Ollama library that fits a 16GB card and tested them all. If qwen3.5's perfect score was a data point, the recruits draw a line through it:

granite4.1:8b and ministral-3:8b both score 24/25. IBM's and Mistral's new generations, each one case short of perfect. Tool calling is no longer Qwen's party trick — it's the entry fee for the new generation.

lfm2.5:8b is the most dramatic row on the table: Liquid AI's official copy literally says it's built for "fast, reliable tool calling on consumer hardware," and native 22/25 honors that claim — but the shim craters it to 5/25, the table's biggest reversal. Over-specialize a model on the native format, force-feed it a homespun JSON prompt, and it forgets how to talk. This proves the same point as qwen3.5's three dropped cases, but more violently: a repair layer isn't neutral for new models — it's poison.

qwen3.6:27b, the most expensive case in this run, scores 21/25 — losing to its own 4B little brother. A 17GB model on a 16GB card: 14% of the layers get evicted to the CPU, inference speed drops an order of magnitude, and just finishing the 50 runs took me almost two hours. And the score is still four cases below qwen3.5:4b (3.4GB, fully in VRAM, fast). One sentence for everyone doing local AI on consumer GPUs: small-and-new that fits in VRAM beats big-and-evicted. Parameter vanity is worthless in the face of offloading.

functiongemma:270m is the Easter egg: Google's 270M function-calling specialist, native 7/25 (respectable at that size), shim 0/25 — the entire model is distilled around the native format, and outside that format it can't survive a single case. The extreme version of a specialist, conveniently serving as the control group for the "format lock-in" phenomenon.

Verdict: the idea is dead, and it died well#

Back to that repair proxy. Three lines in the death certificate:

1. The problem is naturally disappearing. Tool-calling repair is a model-generation problem, not a structural one. The new open-model generation (led by qwen3.5) has solved it natively, and it only gets better from here.
2. A repair layer is a net negative on new models. The moment your target users upgrade their model, your product flips from help to interference.
3. The alternative is one command. ollama pull qwen3.5:4b. 3.4GB. Free. No proxy's install cost beats that.

So the project was executed before the first line of product code. The executioner: 25 test cases I wrote myself. Total time: one day.

If you need local tool calling today, my practical advice in four lines:

• Default pick: qwen3.5 (4b already scores perfect; take 9b if you have the VRAM); granite4.1:8b and ministral-3:8b are solid second choices
• Don't force models that don't fit in VRAM: qwen3.6:27b on a 16GB card is slow and loses to a 4B
• Thinking models: deepseek-r1 needs your own JSON output layer; don't touch its native tool calls
• Avoid: the entire gemma family for structured output (especially with HTML/markup in the payload), and anything from before mid-2025

Closing thoughts#

The last post ended with: "a library's beautiful demo usually assumes a very strong backend." This post is its mirror image: a startup idea's beautiful blueprint usually assumes a world that doesn't change. I assumed "local models can't do tool calling" was a durable pain — but model generations turn over faster than I could have written the proxy.

One day of benchmarking killed a project that would have eaten one or two months. It sounds like a failure; it's actually the best trade I've made all year — the value of validation isn't proving you're right, it's discovering you're wrong while it's still cheap.

The benchmark code and raw result JSONs for all 22 models are on GitHub. The 25 cases cover the shapes you'll actually hit in real products (including the xl-ai HTML-in-JSON killer). Run your own models against it — and if one of them slaps my table around, tell me. I'm happy to be slapped; I'm used to it by now.