Compress context with evidence-aware rules

Summarizing your context window with another model is usually the wrong first move. It works until the summarizer decides a ticket number or a config flag is filler and drops it. Deleting boilerplate and stale tool output is safer: deletion says what disappeared, a paraphrase doesn't.

The thing worth optimizing was never the model alone. It's a finished task on a machine that stays responsive: evidence in, an answer out, a check confirming the answer is good, nothing else falling over. Framed that way, plenty of clever tricks stop looking clever inside a real workflow.

Pick the number before you touch the flag

Before touching a launch flag or swapping hardware, write down the one number that has to move: first-token latency, jobs per hour, models kept resident at once, energy per task, or fewer corrections a human makes. "Make it faster" doesn't say when to stop.

Pin everything before measuring anything: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Pull inputs from the real workload, including edge cases you'd rather ignore. Run one cold pass, storage cold and model unloaded, then warm passes long enough for cache effects, queuing, and memory pressure to surface.

Sort what you're compressing into durable facts, active evidence, and disposable transcript, and reduce each differently. Record phases separately, not as one total:

queue wait
model load / activation
prompt processing (prefill)
time to first token
decode rate, completion time
peak RAM, VRAM, power, swap
quality pass / retry / abstain / repair

Raw token throughput is a diagnostic, not the product. For batch work, count valid completed records per hour. For coding, fold in review and correction time. For chat, check p50 and p95 first-token latency at realistic length. A short warm run at batch of one can still lose once model swaps and long prompts arrive.

The system is happy to lie to you

The recurring mistake is compressing exact identifiers and acceptance criteria into vague prose to save tokens. Local runtimes keep going anyway: offloading layers to the CPU, paging memory, or falling back to a generic kernel, hiding a degraded setup behind a normal answer. Check the startup log and device placement, and confirm the optimization is active for today's tensor shapes, not an old benchmark's.

Change one variable per test unless comparing two whole systems. Save the outputs, repeat every sample, and don't trust the metrics alone. Quantization, compression, sampling, or a model swap can change the answer while looking faster on paper. Run the quality gate every time: check a shorter answer for completeness, a faster extractor against records that validate, not braces that parse.

Operational cost sits beside performance, not below it: startup, upgrades, observability, rollback, rebuilding the box after a disk failure. A five-percent gain from an undocumented patch or manual cache-warming is a bad trade beyond a laptop. Boring configurations age well.

What survives the session and what doesn't

Keep critical state structured, compress narrative history hard: that's the durable rule here. Write the conclusion into the result file with the workload, the date, and why, plus the retest condition: a new model family, a driver update, longer contexts, a different user or traffic mix. Skip that and last quarter's benchmark turns into gospel nobody questions.

Leave headroom once you clear the target. Spare memory absorbs prompt variance. Spare queue capacity keeps a batch job from starving an interactive user. Thermal margin keeps the box from throttling into a worse number. Efficient means finishing predictably on the cheapest, least fragile setup that clears the bar, with something in reserve.

The real advantage of running local is that every layer stays inspectable. You're reading your own logs, not someone else's dashboard.

What I haven't worked out cleanly is the moment a decision from early in a session stops being narrative and becomes a constraint nobody wrote down, and no rule I use catches it reliably.