Keep shell output from eating the context window

An agent three edits into a refactor watches its own test runner dump pages of dot-progress, a coverage table, and a cheerful green summary straight back into its context window. None of that was written for the agent. It's a human-shaped report, built for someone glancing at a terminal between compiles, and every line of it costs tokens the model will never spend on the one failure that actually matters. A smaller model won't fix that, and neither will a bigger context window. What actually helps is admitting that CI output and agent output are different products built from the same underlying run, and that almost none of the tooling on your machine was written with the second product in mind.

Write down the baseline before you touch anything

Before changing a flag or a runtime, I write down one baseline: one named model file, one fixed prompt set, and the exact command that launched the server, flags included. Skip the flags and the whole comparison rots in a few weeks, because you won't remember whether the number you're staring at came from a different quantization, a different context length, or a completely different build.

Then I stop measuring the model and start measuring the request as it moves through the whole system. Time to first token tells you about loading and prompt processing. Steady token rate is decoding, and it's the number everyone quotes first, but it only describes one leg of the trip. Completion time is the one that actually lands on a person or a downstream job, because it's the one nobody gets to skip past. On shared hardware I also track peak memory, queue delay, and wall power, when any of those would change a real decision, not because a dashboard happens to have a slot for them.

The workload decides which of those numbers gets the weight. A nightly extraction job cares about completed, valid jobs per hour, full stop. An interactive coding loop cares about the pauses long enough for a person to switch tabs and lose the thread, and those two things do not trade off against each other the same way, so tuning for one on the assumption it helps the other is a quiet way to make both worse.

I keep the record itself deliberately boring:

Skip the retest date and that row calcifies into a fact nobody rechecks. A benchmark without a decision attached to it is trivia. A configuration without an owner attached to it is folklore, repeated at standup by whoever remembers it existed until nobody remembers why it was ever chosen.

Pasting the whole transcript back in is the tell

Pasting the whole transcript back in after every small edit is the tell. It happens because it works, in a narrow sense: the model still produces plausible-sounding next steps, so nobody notices that the context is now mostly successful-build noise and only a little bit signal. Local inference fails softly more often than it fails loudly, and that's the part people underestimate. A model partly offloads onto the CPU and nothing crashes, it just gets slower. A KV cache misses quietly. Swap creeps up over an afternoon of runs. A queue keeps holding requests whose client already gave up and disconnected. A fallback path kicks in and quietly changes which machine your data is leaving the room on. None of it throws an error. Not one alarm fires.

So watch the machine, not just the chat window. Tail the runtime logs and the operating-system counters while the suite runs, and change exactly one variable at a time, unless the whole point of that particular run is comparing two complete configurations against each other head to head. Repeat enough times to know whether an improvement is real or whether you caught a lucky sample, and actually read a handful of the outputs instead of treating a faster clock as proof that the answers stayed correct. If a change makes some real task meaningfully worse, that cost goes in the same row as the gain, not in a footnote you write later when someone asks.

Operational simplicity is a cost too, and it's the one people forget to price in. A clever optimization that needs hand-repair after every driver update or model swap wasn't free just because it was fast on the day you built it. Not free. I'd rather run something boring that comes back to life from a service file, a container definition, or a short script than something fast that only one person on the team knows how to resurrect. Keep the raw logs out of the written report itself, but keep the hashes and the exact commands, so a rerun weeks from now means the same thing it meant today.

What actually gets sent to the coding model, once the rest of this exists, is small on purpose: the delta, a status line, and the one failure that's actionable right now. That's a less impressive choice than reaching for the biggest model on the shelf or the newest card in the machine, but it produces a system whose limits you can actually see. A visible limit gets routed around, scheduled for later, or priced into the plan. An invisible one turns into an afternoon of unexplained waiting followed by somebody approving an emergency upgrade to make the waiting stop. Stop tuning once the workload clears its quality and latency bar with room to spare, because that spare room is what absorbs the longer document, the extra concurrent user, or the runtime update you didn't ask for and can't schedule around.

None of this is worth setting up for a script you're going to delete by Friday. The baseline, the table, the retest date, all of that is overhead you're paying to keep something honest over months, and if you're just checking whether a small GGUF answers one question sanely on a laptop, the overhead costs more than the mistake you're guarding against. Read the terminal like a person would and move on.