How large must a local tool-calling model be?
Pick the smallest model that survives your recovery and argument tests, not the one that nails clean tool selection in a demo run. That's the whole rule. Everything that follows is just me making the case for it, because a rule without a reason gets ignored the moment someone waves a bigger benchmark number at you.
Most conversations about local tool-calling models start from the wrong end: which checkpoint, how many billion parameters, what some leaderboard says about function-calling accuracy. None of that tells you whether the thing holds up on an ordinary Tuesday when your agent chains several tool calls with optional arguments. The honest starting point is your own tool surface: a private chat, a coding loop, and a nightly batch extraction can share one machine and still need three different answers.
five tools is not the same test as fifteen
Give a small model five well-scoped tools with distinct names and argument shapes that don't overlap, and it will pick the right one almost every time. That's not luck. Tool selection over a narrow menu is close to pattern matching, which small models handle fine. Change the shape of the problem: add tools that could plausibly serve the same request, chain a second call that depends on the first call's output, or hand it a schema with several optional fields and one required field buried in the middle. The picture shifts fast, and parameter count stops predicting anything useful. A bigger model dropped onto a badly scoped tool surface will still pick wrong tools; a smaller model with a tight surface and a real recovery path will beat it. What separates them is schema complexity, argument precision under ambiguity, and what the model does after it gets something wrong, not whether it can name the right function in a clean run.
the six-line notebook, not the six-page report
Before you compare anything, capture one honest baseline: a single named model, a fixed prompt set, and the exact command that started the server, with the model artifact and prompt template saved alongside it. Then build a fixture set that isn't just the happy path: calls with every required argument present, calls missing one, requests ambiguous about which tool applies, and cases where the tool itself throws an error. Run it and follow the request through the whole stack. Time to first token tells you about loading and prompt processing; steady token rate describes decoding once underway; completion time is what the person on the other end feels, and it's the one people forget to measure. Add peak memory, queue delay, and wall power when they matter. If the job is a nightly batch, count completed valid jobs per hour instead of tokens per second, because nobody schedules an extraction run by throughput.
The record itself stays short: artifact, runtime, launch flags; workload and a fixed input set; cold start, warm start, p50, p95; peak memory and wall energy; the quality failures and the abstentions; a decision, an owner, a retest date.
Skip the decision and the owner, and what you've built is trivia with good production values.
the failure that never throws an exception
The tempting shortcut is testing only the happy path and calling the model agentic because the output still reads plausible. It survives review because nothing looks broken. Local inference has a long list of soft failures that don't announce themselves: a model partly offloads onto CPU, a cache stops hitting, swap creeps up, a queue keeps holding requests whose caller already disconnected, or a fallback path changes where the data goes. None of that raises an error. You catch it by watching runtime logs and OS-level counters while the test runs, not by reading the summary afterward. Change one variable at a time unless you're comparing two complete configurations, and run each version enough times to tell a durable improvement from a lucky sample, because a single fast run proves almost nothing. Read the actual outputs: speed is not evidence of equivalence, and if a change makes some task worse, write that cost down next to whatever it gained. A properly built eval harness surfaces that instead of leaving it to vibes.
what a boring config buys you later
Operational simplicity deserves its own line in the record too. An optimization that needs manual repair after every driver update or model swap is not free, it just bills you later. Prefer the boring version: something reproducible from a service file, a container definition, or a short script, even if it's a little slower. Keep the raw run artifacts out of the written report, but keep the hashes and the exact commands, so a rerun later means the same thing it meant today.
The decision rule survives contact with all of this unchanged: pick the smallest model that passes the recovery and argument tests, not just tool-name selection on easy inputs. It's a less exciting choice than reaching for the largest model or the newest accelerator, but it gives you a stack whose limits you can see, and a visible limit can be scheduled around or priced. An invisible one just turns into an unexplained wait and an emergency upgrade. Stop once the workload clears its quality and latency bar with some headroom left over, and don't treat that margin as waste. It's what absorbs a longer document, one more concurrent user, or the next runtime release that changes memory behavior underneath you.
So before you swap in a bigger model because a demo felt slow: rerun your ambiguous-request fixtures against the current one, and check whether it's failing the argument tests, or just failing your patience.