Design around the lost-in-the-middle problem

A 12 GB card holding a 14B-class GGUF runs out of attention before it runs out of context tokens. Token capacity and evidence accessibility are different properties, and a local RAG prompt combining retrieved passages with history and instructions can bury the paragraph that answers the question.

Before touching a launch flag, decide what has to move: first-token latency, accepted jobs per hour, resident model capacity, energy per task, or fewer human corrections. "Make it faster" doesn't say when you're done.

Freeze the stack, then move the evidence

That's not retrieval quality, rag-that-retrieves covers that; it's passages sitting in the prompt, in the wrong spot. Pin model artifact, tokenizer, prompt template (see fine-tune-rag-prompt), runtime build, launch command, and sampling settings, run real-workload inputs, awkward cases included, one cold pass, warm passes until cache effects, queueing, memory pressure, and thermal throttling show.

With that baseline steady, rank evidence, drop duplicates, place the decisive passage deliberately, and test recall by position. Log phases, not one total:

queue wait
model load / activation
prefill
time to first token
decode rate
peak RAM / VRAM / swap
quality pass or retry

Raw tokens per second is diagnostic, not the product. Automation wants valid records per hour; coding wants review and correction time folded in; chat wants p50/p95 latency at realistic length. A short, warm, batch-of-one win can lose once real prompts and model swaps arrive.

Piling on chunks is the wrong reflex

The recurring mistake is retrieving more chunks after a wrong answer, burying evidence deeper. Local runtimes keep going: offloading layers, paging memory, missing cache, queueing, or dropping to a generic kernel, hiding a degraded setup. Check the startup log, then device placement and OS counters, for whether the optimization is active on the tensor shapes and context length used.

Change one variable at a time unless comparing two whole systems, and keep outputs, not just the metrics. Quantization, compression, sampling, or a model swap can change the answer while speeding it up, so the quality gate runs on every candidate. A shorter answer needs a completeness check; a faster extractor needs valid records counted, not matched braces.

Leave a margin and write down when to redo it

Operational cost sits beside speed: startup, upgrades, observability, rollback, whether you can rebuild the server after a disk dies. A five-percent win on an undocumented patch or hand-warmed cache is a bad trade for anything shared. Boring setups tend to survive.

Write the reason into the result file beside workload and date, with the retest condition:

• a new model family
• a driver or runtime change
• longer contexts than you tested
• a different user population
• a different traffic mix

Skip that and the numbers turn into folklore nobody can defend. Leave headroom past the target too. Free memory absorbs prompt variance. Spare queue capacity keeps a batch from starving an interactive user, and thermal or power margin keeps the box from throttling. None of that means packing every resource to a hundred percent; it means finishing work predictably on the cheapest, least fragile setup that clears the bar.

That's the payoff of local inference: every layer stays inspectable, so speed doesn't have to remain a vendor claim.

Fix where the evidence sits before you touch the context window. That's the one rule worth keeping.