When a reranker earns its latency

A reranker cannot rescue a passage your retriever never touched. That's the whole claim. It still needs qualifying, fast: when the right passage is already in your candidate list, just buried a few ranks down because the bi-encoder judges on word overlap, a cross-encoder that reorders it is doing real work, and the latency it costs is worth paying.

I run this on an ordinary local setup, a fast bi-encoder for first-pass retrieval and a small cross-encoder to rescore the top candidates. Nothing exotic, and it won't generalize to yours. It makes the decision measurable instead of a guess. Name the job before you touch a setting: interactive chat, code completion, document extraction, an overnight batch. Decide what a good answer looks like first. Skip that and you'll spend an afternoon optimizing a number nobody asked about.

what the second pass has to earn

Measure recall before reranking touches anything, then compare answer accuracy against the p95 latency the reranker adds. Same inputs every run, and save the launch command with the result: an irreproducible config isn't trustworthy. One warm sample tells you nothing. Include a cold start if a human will ever hit one, run long enough to expose thermal throttling or queueing under load, and keep the output quality next to the timing numbers, not in a spreadsheet you'll forget to update. A speedup that changes the answer isn't a faster version of your system: it's a different one wearing the old name.

Record only the numbers that could flip the decision:

first token:  cold and warm
throughput:   prompt processing, generation
memory:       peak resident
power:        wall draw, if it's the limit
outcome:      pass / fail / abstain
verdict:      keep, revert, retest

Medians tell you about the typical run. A slow percentile tells you about the run that makes someone close the laptop in frustration. Write down the noise and the setup friction too; those costs decide whether anyone keeps using it.

the compensation trap, and what survives an upgrade

The common mistake is reaching for a reranker to cover bad chunking or a document that was never indexed. That's a retrieval failure, and no reordering fixes it; you'd get more from fixing rag-that-retrieves than from a second pass. "It loaded" is not a performance result. "The answer looked fine" is not an evaluation. Check the runtime logs and OS-level metrics instead of trusting whatever flag you requested. When two runs disagree, change one variable and state the mechanism you expect to move, or you're just collecting anecdotes.

There's a maintenance bill too. A fragile five-percent win evaporates the next time a model, driver, or runtime updates. Keep a small script, a handful of representative prompts, and a plain-text result file. That's enough to retest after every upgrade without standing up a monitoring stack for one workstation.

My rule: pay for the extra pass where ordering errors, not missing-document errors, dominate your failures. Test the smallest plausible change first, leave headroom, and stop tuning once the workflow clears its latency and quality bar. Spare capacity is what absorbs a longer prompt next week, or the model you'll want to try next month.

So before you wire one in: pull the queries your system got wrong, check by hand whether the right passage was even sitting in the candidate list, and only then decide which half of the pipeline actually owes you an explanation.