A 128K context window is not free locally

So you drag the slider to 128K, dump half the repo into the prompt, and the model swallows it without complaint. That's the part that fools people. Acceptance and capacity aren't the same claim, and capacity isn't free on hardware you own and have to power yourself.

It's easy to tune the number everyone can see, context length in a settings panel, and leave the thing actually limiting you, memory headroom, untouched. My reference setup is a local coding model wired into a repository assistant, on hardware I'd call ordinary, no claim my numbers travel to your box.

Before touching a setting, write down what the job actually is: chat, completion, extraction, an overnight batch. Each has a different definition of good enough, and that has to exist before you measure anything.

where the extra tokens actually live

Every token you hand the model needs a key and value vector cached at every layer, sitting in the same memory as the weights. That cache grows in a straight line with context length, no clever curve saves you at the far end. Double the tokens and you roughly double what the cache costs. On a 12 GB card running a 14B-class GGUF, that cache competes with the weights for room, and the competition gets worse the longer a session runs.

A window that fits comfortably at 4K can start thrashing at 32K without you adding a single real capability to the prompt.

the tax before the first token

Before generation starts, the engine runs the whole prompt through the model once, attending over every token to fill the cache, and that pass costs you before it produces anything. Time to first token is the honest number, not tokens per second once the stream is rolling; conflating the two is how a fat context gets called fast because the back half looked fine.

Measure prompt time at more than one size, 4K, then 16K, then 32K, with identical inputs each run, and save the launch command next to the result. Take a cold start too, not just the warmed run, because cold is what your editor pays for every session.

A faster run that also changes the answer was never the same system to begin with.

logging the numbers that can move the decision

Pick one named workload, fix its inputs, version them so a rerun still means the same thing months later, and set a quality gate before touching anything: pass, fail, or abstain. Log latency cold and warm, plus the 95th percentile, because the slow tail is what a person actually notices. Log peak memory, wall power when it matters, and whether the task succeeded, then write the decision, keep, revert, or retest, beside the numbers.

the repo-dump reflex

The failure mode that opened this piece is dumping a whole repository into the window because the model will take it, then paying to re-process mostly irrelevant text on every call. Local inference runs configurations that technically work: loading tells you nothing about speed, and an answer that looks plausible tells you nothing about correctness. Check the runtime log and the operating system's own numbers instead of trusting whatever flag you passed in, and change one variable at a time to know what actually moved.

There's a maintenance cost too: a fragile five percent win tied to one model, one driver, one runtime build evaporates the moment any of those updates. The same discipline that keeps a token diet honest in a hosted tool works locally too, minus the invoice: a small script, a few prompts, a plain-text file.

spending the window like a budget

My rule: context is a budget spent on evidence, not a bin you pour tokens into because there's room. Test the smallest plausible change first. Preserve headroom on purpose: spare capacity absorbs a longer prompt next week, a background process you forgot was running, or next month's model. Stop tuning once the workflow hits its target, not before and not long after.

I've made a version of this argument before about context windows being a lie in hosted tools; locally that lie carries a memory footprint and a wall-clock number, which makes it easier to catch. Hardware matters less here than clear workload boundaries and fewer wasted tokens.

So here's the one rule I keep: don't fill the window because it's there, fill it because you measured that it helps.