Run the reranker on CPU when the GPU is busy

One GPU on the box, a generation model already resident and warm, and a coding loop keeping it busy most of the afternoon. Now retrieval needs a reranking pass, a small cross-encoder that reorders candidate chunks before they reach the prompt. The instinct is to put it on the same card as everything else. That instinct is usually wrong, and working out why matters more than any number pulled from a synthetic benchmark.

Where a step runs comes down to contention and to which model is currently resident in memory, not raw kernel speed measured on its own. Local AI conversations tend to open with a model name or a peak-throughput figure; a better question is what's already waiting on the other side of the API by the time the reranker shows up. A private chat touched a couple of times an hour shares the box with a coding loop that barely stops, and behind both a nightly extraction batch chews through documents on its own schedule. Three different placement needs, one small cross-encoder.

Why the cross-encoder fights for the same card

The headline comparison, GPU rerank against CPU rerank on a synthetic batch, isn't the number that matters. What matters is the point where the choice changes how the system behaves in an ordinary week, once two or three workloads start overlapping instead of one running by itself. Put the reranker on the GPU and it either queues behind the generation model or forces a partial reload of its weights to make room. Put it on CPU instead and it runs a little slower in isolation, cross-encoders are cheap by LLM standards, and the GPU stays committed to decoding tokens for whoever is waiting.

This is the whole case for a single-GPU RAG server sitting on spare CPU capacity nobody is using: spend it. The generation model already has first claim on the accelerator. Let the reranker be a good citizen on CPU rather than a rival for a card it doesn't need.

The baseline that makes the comparison mean something

Before moving anything, write down a baseline you can trust: one named reranker model, one fixed document and query set, and the exact server command and flags used to launch it. Keep the model artifact and prompt template attached to that record; both are easy to lose and large enough to quietly invalidate the comparison months later. Then measure its contribution to end-to-end latency under realistic concurrent load, not in isolation. Time to first token catches loading and prompt work, steady token rate describes decoding once underway, and completion time is what the user, or the downstream job, gets. Add peak memory, queue delay, and wall power draw when they bear on the decision. For a nightly extraction job, count completed valid jobs per hour; for a private chat, count the slow waits a person remembers.

I keep the record as a short table rather than a paragraph, because a paragraph makes it too easy to skip a field:

That last row is the one people skip, and it's the one that matters. A benchmark without a decision attached is trivia. A setting nobody owns turns into folklore, the kind repeated in a wiki page nobody has opened since it was written.

Plausible output hides a bad placement decision

The tempting mistake is dragging every model onto the GPU because it's right there, then living with the weight swaps that follow once reranker and generator both want residency at once. This survives in production a long while, because the system keeps producing plausible text and nobody looks closely enough to notice. Local inference fails in soft ways far more often than hard ones: a model partially offloads to CPU without anyone flagging it, a KV cache quietly starts missing, swap creeps upward, a request queue holds connections nobody is waiting on anymore, or a fallback path silently changes which process, and which machine, ends up touching the retrieved documents. None of that raises an error.

Catching it means watching runtime logs and OS-level counters while the test is running, not just reading the final latency number afterward. Change one variable at a time, unless you're deliberately comparing whole configurations against each other. Repeat the run enough times to tell a durable improvement from a lucky sample, and read some of the reranked output instead of trusting speed as proof of equivalence. If moving the reranker to CPU makes one important query worse, that cost goes next to the gain, not into a footnote nobody reads.

I wouldn't bother pinning the reranker to specific CPU cores or chasing NUMA locality here, not unless the queue numbers show contention. A reranking pass that small rarely earns that kind of attention; the baseline table earns it first.

The case for boring, and when to stop tuning

Operational simplicity deserves its own row in that table, even though it isn't a number. A fragile optimization that needs manual repair every time a driver or model updates is not free, whatever the benchmark said. I'd rather run a configuration reproducible from a service file or a short script than chase another few points from a hand-tuned placement that breaks on the next CUDA update. Keep the raw benchmark artifacts out of the report itself, but hold onto the hashes and commands, so a rerun down the line still means what it meant today.

The rule I'd carry forward: put each stage of the pipeline wherever the whole thing finishes fastest, not wherever the newest or biggest model happens to live. For a compact reranker on a single-GPU box, that's CPU almost every time, because the generation model has first claim on the card and the reranker's job is small enough not to need it. It's a less exciting answer than buying a second GPU. But it produces a system whose limits you can see. A visible limit gets scheduled around or priced into the plan. An invisible one just shows up as waiting, until it becomes an emergency upgrade.

Stop tuning once the workload clears its latency and quality target with some headroom to spare. That headroom absorbs a longer document, one more concurrent user, a warm afternoon, or whatever the next point release of the inference runtime decides to change.

Next thing I'd check on a box like this: queue depth and time to first token on the generation model with the reranker moved to CPU, measured under the heaviest overlap you see in production, not the average one. If that number holds up, leave it alone.