Plan embedding upgrades as migrations

Somebody swaps the embedding model on a Friday afternoon, points the same collection at the new endpoint because the dimensions match, and weeks later retrieval quality falls apart. It happens more than people admit. From the outside it always looks the same: search results that used to be sharp turn vague, nobody can name the day it started, and the postmortem lands on the same line anyway, something like "we re-embedded in place." The two vectors sitting next to each other in that collection were never speaking the same language.

Same dimensions, different universe

The number that gets checked before a swap is dimensionality, and it's almost never the number that matters. What actually defines a vector space is the exact model checkpoint plus every preprocessing step that ran before the text hit the encoder: tokenization, truncation, whitespace handling, any prefix wrapped around the input. Change one of those and you get the same count of floats living in a different geometry, cosine-similar to nothing the old index ever understood. Local AI writeups love to open with a model name or a peak throughput figure. A more useful place to start is the work sitting on the other side of the API call: a private chat pinged a few times an hour, a coding loop firing constantly, a nightly extraction job chewing a queue, all sharing one box and wanting opposite things from it.

What a baseline actually needs to capture

Before touching a production-like local RAG index carrying millions of chunks, freeze a baseline: one named model, one fixed prompt set, the exact server command you actually ran. Put the model artifact and the prompt template inside the record itself, not a wiki page next to it. Both vanish within weeks and take your ability to compare anything with them.

What's worth putting in that baseline:

• time to first token, which mostly tells you about loading and prompt handling
• steady token rate, which describes decoding once the pipeline is warm
• full completion time, which is what a person actually experiences
• peak memory, queue delay, and wall power, when they affect the decision
• completed valid jobs per hour for batch work, slow waits a person remembers for interactive work

I keep the running log plain, nothing fancier than a text file next to the config:

model artifact + runtime + launch flags
workload and the fixed input set used to test it
cold start, warm start, p50, p95
peak memory and wall energy
quality failures and abstentions, not just latency
decision made, owner, date to retest

A row without a decision is trivia. A setting without an owner turns into folklore, and somebody reverse-engineers it from a stale comment.

Mixing vectors is how outages start quiet

The mistake I see most is re-embedding in place: swap the model, let a long batch job write new vectors into the same collection the old ones already live in, and keep serving traffic while it runs. It survives review because the system keeps answering with plausible text, and plausible text is exactly what makes this dangerous. None of the actual failure modes throw an error. A model partially offloading to CPU won't. Neither will a cache quietly missing, swap growing behind everything else, a queue holding requests from clients that already disconnected, or a fallback path that silently changes your privacy boundary and routes a request somewhere it was never supposed to go.

Watch runtime logs and OS counters while the test runs, not afterward from a summary. Change one variable at a time, unless the test is comparing two complete configurations. Run it enough times to know whether an improvement is durable, and read the outputs. Speed is not proof that a new model is equivalent to the old one. If a change makes one task worse, that cost belongs beside the gain, not buried in a footnote nobody opens.

Operational simplicity deserves a line item too. An optimization needing manual surgery every time a driver or model gets updated is not free. I'd rather run something boring that reproduces from a service file, a container definition, or a short script than something faster only one person can fix. Keep the raw run artifacts out of the report, but keep the hashes and commands, so running it again means the same experiment, not a guess.

The index is not sacred, the corpus is

The rule that's held up for me: treat the vector index as derived, versioned data, rebuilt on purpose rather than mutated quietly. Version the collections, dual-write during the transition, compare retrieval side by side, then cut over with a rollback path still sitting there. That's less exciting than picking whichever model tops a leaderboard this week, the approach most model-selection advice leads with, but it produces a stack whose limits you can see. Visible limits get routed around, scheduled, or priced into a decision. Invisible ones turn into unexplained waiting and an emergency upgrade nobody budgeted for.

Stop tuning once the workload hits its quality and latency target with real headroom to spare. That spare margin isn't waste; it absorbs a longer document than your test set had, one more user, a warm afternoon when the fans can't keep up, or whatever the next runtime release changes. Efficient local inference is mostly spending model capacity only where it changes the answer, and leaving it alone everywhere else.

What I still don't have a clean answer for is how long you keep the old collection warm during dual-write once the corpus is too large to fully re-embed overnight. Retrieval comparison against a frozen eval set tells you the two indexes agree on the queries you thought to ask, not on the ones your users will type down the line. I haven't found a better trigger than a gut feeling for when that eval set itself has gone stale, and I'd rather admit that than pretend the dual-write period has a clean end date.