Swap is a warning light for interactive local inference

A 14B-class GGUF sitting next to three browser windows, two containers, and an IDE indexing a large repo runs fine for a few minutes, then the decode rate falls off a cliff once memory pressure crosses some invisible line and the kernel starts paging model weights to disk and back.

Task manager still shows the process alive. Tokens still come out, one at a time. What's actually running under the chat window now is a disk benchmark that happens to produce text.

That's the failure mode here. Swap is the OS's insurance policy, a way to survive memory pressure instead of getting killed outright, and it does that job well. It never substitutes for the memory bandwidth a transformer needs to move weights and activations every token; once the model touches swapped-out pages, each token becomes a disk I/O event on a far slower clock.

Decide what "better" means before you touch the launch flags

Before changing hardware or adding a launch flag, write down what actually has to improve on this box, next to this pile of browser tabs, containers, and an editor reindexing in the background. First-token latency. Accepted jobs per hour. Resident model capacity. Energy per completed task. Fewer corrections from whoever reviews the output. Pick one, maybe two. "Make it faster" is a mood, not a target: it won't tell you when to stop tuning or whether the change helped the workload or just the one benchmark you ran.

This matters more on a shared workstation than a dedicated inference box: the competition for memory isn't the model against itself, it's the model against Chrome's per-tab overhead, a couple of containers that never asked permission, an IDE reindexing the repo. None of those processes care that you're mid-generation, and neither does the kernel. It protects itself first, which means protecting the OS from abrupt failure by paging your model out, snappy reply or overnight batch job.

Time it in phases, not as one blurry total

Pin everything before you measure: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Use a small set of real-workload inputs, including the awkward cases. Run once cold, meaning cold storage and an unloaded model, if users will ever hit that path. Then run warm, repeatedly, long enough for cache effects, queueing, memory pressure, and thermal throttling to show up. A three-second warm run tells you nothing about minute twenty of a real session.

Watch the swap-in rate and memory pressure across the full session, not just at the start, and note what happens when you cut context length or drop to a smaller model. Record it phase by phase instead of one total that hides where the time went:

• queue wait before the request even gets picked up
• model load or activation
• prompt processing and prefill
• time to first token
• decode rate and total completion time
• peak RAM, VRAM, power draw, and swap activity
• the quality pass: pass, retry, abstain, or repair

Raw tokens-per-second is a fine diagnostic and a poor final answer. What you report depends on the job. For automation, count valid completed records per hour, not tokens. For coding, fold in the time spent reviewing and fixing the output. For chat, look at p50 and p95 first-token latency with realistic conversation lengths, not one prompt into an empty context; the tail matters as much as the median, which streaming latency architecture covers. A setup that wins a short, warm, one-shot test can still lose the day once model swaps and long prompts show up.

The runtime is built to hide exactly this problem from you

The recurring mistake is checking allocated RAM once and missing that the process pages continuously anyway. Local runtimes are almost too good at hiding this: they offload layers to a slower device, page memory quietly, take cache misses, queue requests, or fall back to a generic kernel when the fast path doesn't fit the shape you handed it. Stacked together, the system keeps answering while running in a degraded state nobody chose.

So go check. Read the startup logs instead of skimming past them. Look at where each tensor actually landed, not where you assumed. Watch the OS's own memory counters while a real request runs. Confirm the optimization you think is active actually is, for the tensor shapes and context length you're using now, not the ones from the tutorial you copied the flag from.

Change one variable at a time unless you're deliberately comparing two complete configurations. Repeat every sample and keep the actual outputs, not just the derived metrics: quantization, context compression, sampling changes, and model swaps can all make an answer worse while making it faster. Run the quality gate on every candidate, no exceptions. If a response got shorter, check it's complete, not just truncated. If an extractor got faster, count valid records, not braces that happen to parse.

Weigh the operational cost next to the performance number; they're the same decision. Does the change affect startup, upgrades, observability, rollback, or your ability to rebuild the server after a disk dies? A five-percent gain propped up by an undocumented patch or a human warming the cache every morning is a bad trade for a shared service. Boring configurations age well. Clever ones need a maintainer who remembers why.

Leave the memory alone once the workload stops swapping

The rule that holds up is simple: keep enough physical memory free that the steady-state workload never swaps, full stop. Write that down: the workload, the date, the reasoning, and the condition that should trigger a retest, a new model family, a driver update, longer contexts, another user, different traffic. Skip that and last quarter's benchmark quietly turns into infrastructure folklore, a number everyone repeats and nobody can reproduce.

Leave headroom once you hit the target instead of chasing the last percent of utilization. Free memory absorbs a prompt longer than anything you tested, and spare queue capacity keeps an interactive user from feeling a batch job land on top of them; thermal and power margin keeps the box from throttling deep into a run nobody was watching. None of that shows up in a benchmark number, which is why it gets cut first when someone's optimizing for a screenshot.

This is the actual point of running models locally: every layer is inspectable, so performance never has to stay a vendor's claim you take on faith. If you're building anything that keeps state across a session, the same discipline applies to agent memory architecture choices: measure the real thing running, not the demo.

Which is why I keep some capacity in reserve instead of sizing a box for exactly the model I'm running today. I'm giving up a fully packed machine, and a slightly bigger resident model than I could technically run, on purpose, in exchange for a box that behaves the same at hour four of a session as it did in the first minute.