Sliding-window attention changes long-context expectations
People point a local model at a long document or a multi-hour agent session, trust the context window number on the model card, and get burned when something from early on quietly stops being used.
The model still accepts the tokens. It still answers. It just stops reaching back for the fact you gave it an hour ago, and nothing in the output flags that. This already happened to plenty of people.
The unit worth reasoning about was never the model on its own, it's the whole completed task: evidence goes in, an answer comes out, a quality gate passes it or doesn't. That framing catches an optimization that looks great alone and vanishes once wired into a real workflow. For local models on long documents or the kind of extended agent session covered in advanced agent architecture, attention architecture decides which older evidence stays reachable, separate from how many tokens it will swallow without complaining.
Before touching hardware or a launch flag, write down the one number that has to move: first-token latency, jobs per hour, resident model count, energy per task, or how often a human fixes the output. "Make it faster" doesn't tell you when you're done. The nice part about local: every layer's inspectable, so performance is something you measured, not something a vendor told you.
pin the run, then test distance, not size
Pin everything: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Pull test inputs from the real workload, awkward cases included. If users hit a cold model on cold storage, run that path once, then run warm long enough for cache effects, queueing, memory pressure, and heat to show up.
Read the runtime's architecture notes and test what marketing pages never do: retrieval accuracy at increasing distance, with realistic distractors. Track the run in phases, not one wall-clock total, since a single number hides which part is costing you:
| phase | what it exposes |
|---|---|
| queue wait | contention before the request starts |
| load / activation | cost of a cold or swapped model |
| prefill | how the window fills before generation begins |
| time to first token | what an interactive user feels |
| decode rate | throughput once generation is running |
| peak RAM / VRAM / power / swap | headroom left for anything else |
| pass / retry / abstain / repair | how often the gate intervened |
Raw tokens per second is useful diagnostic evidence, a poor stand-in for what you ship. Automation wants valid completed records per hour. Coding wants review and correction time folded in. Chat wants p50 and p95 first-token latency on real conversation lengths. A setup that wins a short warm run with one request in flight can lose once model swaps and long prompts show up.
resilience hides the degradation you're trying to catch
The recurring mistake is treating successful token ingestion as proof of uniform recall across the window. It isn't, and it's easy to miss because local runtimes are stubbornly good at limping along: offloading layers, paging memory, eating cache misses, queueing requests, falling back to a generic kernel. That resilience keeps answers coming back, which is exactly what makes a degraded configuration invisible. Read the startup logs, check where tensors landed, watch the OS counters, and confirm the optimization is active for your context length and tensor shapes, not someone else's benchmark.
Change one variable at a time unless you're comparing two complete systems, and keep the outputs, not just the metrics: quantization, context compression, sampling tweaks, and model swaps can make an answer worse while making it faster. Every candidate owes a pass through the quality gate first. A shorter answer needs checking for completeness, not applause for brevity. A faster extractor needs valid records counted, not credit for something that merely parses.
Weigh operational cost too: startup, upgrades, observability, rollback, rebuilding the box after a disk fails. A five-percent win on an undocumented patch, or someone keeping the model warm by hand, is a bad trade for anything shared. Boring configurations age well. Clever ones page you at two in the morning.
The rule that survives contact with production: size chunking and memory around the effective reach you measured, not the number on the model card, and write that down with the workload, date, and reasoning. Put the retest trigger beside it: a new model family, a driver update, longer contexts, a different user, different traffic. Skip that and last quarter's benchmark turns into this quarter's folklore.
Leave slack after hitting the target: free memory to absorb prompt variance, spare queue capacity so a batch job doesn't starve an interactive one, thermal margin so the box survives a hot afternoon. That slack costs you the last few points of utilization you could claim on a slide, and I'd take that trade every time over a system that only behaves correctly in a demo.