Hybrid search is a practical default for technical RAG

A local RAG box has one memory budget, and every process competes for a slice: the embedding model, the generator, the vector index, and whatever cache the retrieval layer keeps warm. That budget doesn't care whether the box is serving a chat interface someone opens a couple of times an hour, a coding loop that fires on every keystroke, or a nightly job extracting structure from documents while nobody's watching. Same hardware, three different tolerances for latency, three different tolerances for being subtly wrong.

Names don't paraphrase

The corpus sitting on top of that hardware has a split personality. Most of it is ordinary technical prose, tickets and docs and commit messages that say the same thing several ways, and an embedding model is good at matching across that variation. But mixed into the same corpus are strings that carry no meaning unless they match exactly: version numbers, stack traces, function names, ticket IDs, config keys. Push a raw error code through an embedding model and you'll get documents that are thematically close and factually useless, because it was trained to generalize away from exact surface form. Lexical search doesn't care about theme. It just wants the string. That's the case for running both, fusing the ranked lists, and evaluating separately by query type instead of averaging into one number that flatters neither. If your corpus leans toward code this split gets worse, and it's worth checking how embeddings-for-code-search handles identifiers before assuming a general-purpose embedding model already covers them.

Write down what you ran, not what you meant to run

Before touching the retrieval config, capture a baseline: a named model, a fixed prompt set, the exact server command used to launch it. Note the model artifact and the prompt template too, since both drift and both are large enough to invalidate a later comparison. Then retrieve lexical and semantic candidates, fuse the ranks, and evaluate by query type rather than folding everything into one score. For interactive work, track the waits a person notices: time to first token and steady decode rate feeding into total completion time. For the nightly job, track completed valid jobs per hour instead, since nobody is watching a progress bar. I keep the same fields for every run, because a comparison across runs only means something if they line up:

• artifact, runtime, and launch flags
• workload and the fixed input set
• cold start, warm start, p50, p95
• peak memory and wall energy
• quality failures and abstentions
• decision, owner, retest date

That last line is the one people skip, and the one that matters. A benchmark that doesn't end in a decision is trivia. A configuration change with no named owner turns into folklore within a month, a setting nobody remembers the reason for.

Fine, right up until you check the logs

The tempting mistake is trusting embeddings alone to recover version strings and function names, and it survives in production for a while because the output still reads fine on a glance. Local inference is full of failures that never raise an error: a model partly offloads to CPU and gets slower, a cache misses quietly, swap creeps up, a queue holds requests whose clients already disconnected, a fallback path silently changes where your data leaves the machine. None of that shows up as a crash. It shows up as a shape in the logs, if you're watching while the test runs instead of only checking after.

One variable, several runs, read the actual output

Change one thing at a time unless you're deliberately comparing two complete configurations, and run each version enough times to tell a durable improvement from a lucky sample. Read the outputs. A faster average isn't proof of equivalent quality, and if a change makes an important query worse, that cost belongs beside the speed gain, not buried in a footnote. Operational simplicity deserves the same scrutiny as latency: an optimization needing manual repair after every driver or model update isn't free, it's a recurring bill you haven't accounted for. Given the choice, I'll run something boring that reproduces from a service file or a short script over something clever only its author can fix.

Where the margin actually goes

The rule that's held up for me is to combine lexical and semantic signals before reaching for a bigger generator. It's a less interesting decision than picking the biggest model or the newest accelerator. It produces a stack whose limits you can see coming instead of finding them the hard way. A visible limit gets scheduled around or priced in. An invisible one turns into unexplained waiting and an emergency upgrade nobody budgeted for. Stop tuning once the workload hits its quality and latency target with real headroom left over, because that margin is what absorbs the next long document, the next concurrent user, a warm afternoon, the runtime update you didn't test against. None of this replaces reading what rag-that-retrieves says about evaluating retrieval quality, and I'll admit the hybrid setup is two systems to keep in sync instead of one. If your corpus is small enough that plain grep already finds everything, skip all of it.