Return compact tool results to the model

So no, the fix isn't a bigger context window, and it isn't tokens-per-second either. What wrecks an agent loop is duller: a grep returns pages of matches, a failed test buries the one fact that matters in framework noise. A blown context budget is rarely about size. It's about clutter.

Talk to enough people about local models and the conversation drifts to a model name or a throughput number fast. Wrong starting point. What should set the rules is the work on the other side of the call: a private chat checked twice an hour, a coding agent looping tool calls all afternoon, an overnight batch job, same box, three different answers for what a tool result should hold.

The cut happens at the boundary, not the tail

Before touching a truncation rule, pin down what you're running, one named model, one fixed prompt set, the exact launch command, since skipping any of that invalidates the next comparison. The real work is filtering at the tool boundary, not upstream in the prompt or downstream in the model's head: keep provenance on whatever survives the cut (which command, which file, which line range), and give the caller a way to page through what got trimmed. Treat the return value as an interface, not a shell around a print statement, the case the tool design crowd keeps making. Done right, a paginated result reasons like a full one; done wrong, you've just moved the noise deeper.

The clock has more than one hand

Once results flow through a real filter, measure the whole path, not just the easy part. Time to first token is loading plus prompt processing, steady token rate is decoding, completion time is what the person or the calling agent experiences. Add peak memory, queue delay, and wall power only when one will change the decision. For a batch job, what matters is completed valid jobs per hour. For anything interactive, it's the slow wait a human remembers and complains about next week, whatever the average says.

It fits on an index card

I keep the record for each run embarrassingly plain, on purpose:

model + runtime + exact launch flags
workload + fixed prompt/input set
cold start, warm start, p50, p95
peak memory + wall power draw
quality failures / silent abstentions
decision + owner + retest date

That last line is the one people skip and the one that matters most. A benchmark with no decision attached becomes trivia by next quarter, and a setting with no owner turns into folklore nobody can explain. That habit of hoarding tokens is what the token diet piece argues for, one layer lower in the stack.

Silence is not the same as correctness

The tempting shortcut is truncating blindly from the tail, taking the error summary or file identity with it. It survives because the output reads as plausible text. Local inference fails quietly: a model partly offloads to CPU, a cache misses without complaint, swap creeps up, a queue keeps serving requests nobody's waiting on, a fallback silently moves where your data is allowed to go, and none of it throws an error. Watch runtime logs and OS counters while a test runs, change one variable at a time, and rerun to rule out a lucky sample. Read the outputs, not just the clock. An optimization that needs hand-repair after every driver bump isn't free, so a boring, reproducible setup beats a clever one; keep raw logs out of the report but keep hashes and commands so a rerun means the same thing twice.

What I'm giving up to get this

Build the summary around the decision it supports, and keep the raw output one click away for anyone who doesn't trust it. Less fun than picking the biggest model or newest card, but it leaves visible limits: something you can route around, not an invisible one that becomes an outage. Stop tuning once the workload clears its quality and latency bar with headroom to spare, because that margin absorbs the longer document, the extra user, or the next runtime upgrade nobody warned you about. The tradeoff I'm making on purpose: some detail that might matter gets cut before the model sees it, and once in a while that costs a debugging session where the answer was sitting in the lines I dropped. I'll take that loss over paying the full context tax on every call, forever.