Local tool calling is mostly an interface contract

Someone hands a local agent a general shell tool, watches it write clean commands for a while, and stops reading arguments before they fire. A prompt goes sideways, the model proposes something confident and syntactically correct, and the tool fires anyway, because nothing was set up to say no. I've watched a coding loop wipe the wrong directory this way, trusting fluent text as authorization. It didn't need a bad model, just an unguarded gap between what it said and what the machine let it do.

Here's what never makes the roadmap slide: the model doesn't execute anything, it emits an argument proposal, and the application in between is the only thing standing between "sounds right" and "did the thing." Local tool calling comes down to schemas and orchestration more than model training: tool surface, argument validation, who signs off on mutations. Less exciting than a bigger context window. Also the part that holds up.

Same hardware, three different bars to clear

A private chat, a coding loop working through a repo, and a nightly extraction job can share one box and still want different trade-offs. The chat cares about the first few hundred milliseconds. The coding loop cares whether the tenth call in a chain still has the right context. The nightly job barely notices latency and cares whether every output parses. Treating all three as one local-inference problem is how people tune for the wrong thing.

So write a baseline before touching a config file: one named model, one fixed prompt set, the exact launch command, plus the model artifact and prompt template, both easy to lose and easy to let wreck a comparison. Keep tools narrow. Validate arguments against a schema first. Return compact, structured errors, not a stack trace the model will paraphrase into something confident-sounding. Anything that mutates state waits on a human saying go.

What you record depends on the job: time to first token (loading, prompt processing), steady token rate (decoding), completion time (what the user feels, the only number anyone asks about). Peak memory, queue delay, wall power earn a column only when they change the decision. Nightly batch: completed valid jobs per hour. Chat: the slow waits a person remembers.

artifact + runtime + launch flags
workload + fixed input set
cold start / warm start / p50 / p95
peak memory + wall energy
failures and abstentions, read not just counted
decision, owner, retest date

A number that doesn't lead to a decision is trivia. A setting nobody owns drifts back to whatever it was before.

Nothing here throws an error when it should

The mistake that keeps recurring: handing a local agent one generic shell tool and treating fluent, well-formed text as authorization on its own. It survives review a long time, because the system keeps producing plausible output right up until the one bad argument gets through. Local inference has a family of failures that never raise an exception:

• a model partly offloads to CPU and quietly slows down
• a cache misses and nobody notices until throughput drops
• swap grows under memory pressure with no warning
• a queue keeps serving requests whose caller already disconnected
• a fallback path swaps in a different backend, privacy boundary included

None of that raises an error; it shows up later as a graph that looks fine until you go looking, which means looking is the job. Watching a local pipeline while it runs is most of what agent observability means in practice: logs and counters live, one variable at a time unless comparing whole configurations, enough repeats to separate a real gain from luck. Read the outputs too. A faster run that quietly gets more answers wrong is a worse trade dressed up as a win.

Operational simplicity belongs in that report too. A clever optimization needing a manual fix after every driver or model update isn't free, whatever the benchmark said. Boring wins: something that comes back from a service file, a container definition, or a script exactly as it left. Keep raw artifacts out of the write-up; keep hashes and commands so a rerun still means the same thing.

Designing that boundary for failure, retries, and a human in the loop is most of what sandboxing coding agents reduces to, marketing stripped off. Less fun than picking the biggest model on the spec sheet, but it makes limits visible, and a visible limit gets scheduled around instead of becoming a 2am page. Stop tuning once the workload clears its quality and latency bar with headroom left: that margin absorbs the longer document, the extra user, the hot afternoon, the next runtime update.

The rule I'd keep if I kept nothing else: never let an argument reach a mutation without a schema check and a human who can still say no.