Lower top-k until retrieval has to earn each chunk

Every team that builds a RAG pipeline eventually hits the same reflex. An answer comes back wrong, quoting the wrong doc or inventing a detail that isn't in any of them, and the fix that gets reached for first is always the same: turn up top-k. It happens one disappointing demo at a time, and nobody notices the context filling up with passages that repeat each other or flatly disagree, until the model starts hedging between them instead of answering. By the time someone asks why the assistant got worse after the "improvement," the retriever is pulling in three times what it used to, and half of it is noise the model now has to argue its way past.

I run a small RAG system, built on the idea that retrieval has to earn its keep, over a pile of internal technical notes as my reference case. Nothing exotic, no lab conditions, no claim that this setup generalizes to your machine. What it does let me do is make the top-k decision measurable instead of vibes-based. Before touching the setting, write down the actual job: interactive chat, code completion, document extraction, an overnight batch run. Each of those wants a different balance between coverage and interference, and none of them care how impressive your top-k number looks sitting on its own.

Sweep k, label the chunks, watch what breaks

The only test that answers the real question is a boring one: label which chunks actually contain the answer, sweep top-k across a range, and read the failures instead of trusting the similarity score to vouch for itself. Use the same input set on every run, and write the exact launch command down next to the result, because six weeks from now you will not remember which flag did what. One warm sample tells you nothing. Run a cold start if your users will ever hit one, keep the run going long enough that thermal throttling or request queueing has a chance to show up, and log the answer quality alongside the timing numbers every time. If a "faster" configuration changes the answer, you don't have a faster version of your system. You have a different system that happens to also be quicker.

Record only the numbers that could actually change your decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power if that matters to you, and whether the task actually succeeded. Medians describe the normal case. A slow percentile is what tells you about the pause that makes someone close the laptop and complain. Write the friction down too, the noise, the setup annoyance, the flakiness, because that's what decides whether a configuration survives a real workday and not just a benchmark run.

Here's what I pin to paper before touching any slider:

Nothing clever in there. That's the point of it.

Keep the script, skip the observability platform

The trap that actually costs people time is raising top-k every single time an answer disappoints, until the context is packed with near-duplicate chunks that contradict each other more than they inform anything. That's the same reflex some agentic RAG loops will cheerfully automate on every retry, which is worse, not better. A model loading without errors is not a performance result. An answer that looked fine on one pass is not an evaluation. Check the runtime logs and the operating-system metrics instead of trusting whatever flag you typed, because plenty of local setups will happily run in a configuration that technically works and quietly underperforms. Change one variable at a time and say what you expect to happen and why, or your benchmark turns into a pile of anecdotes with a spreadsheet attached.

None of this saves you if the embeddings can't tell two documents apart in the first place. Tuning k is a knob on top of a retriever that already works, not a substitute for one. A fragile five-percent win from last month's sweep won't survive the next model swap, driver update, or runtime bump, so don't bother chasing it. What's worth keeping is a small script, a handful of prompts you actually trust, and a plain-text file of results you can diff against the last run. That's enough to retest after an upgrade without standing up a monitoring stack for what is, in the end, one workstation under your desk.

The rule I keep is one line: pull in the smallest set of chunks that reliably contains the answer, and don't raise k again until you can name the mechanism that requires it.