Price the RAG rebuild before changing chunking

Price the rebuild before you touch the chunking strategy: the parsing pass, the embedding run, the transfer of new vectors to wherever they live, and the validation gate that has to clear before anyone trusts the new index. That's the rule. The rest earns it: migration cost is the line most retrieval experiments skip.

A private chat opened a few times an hour, a coding loop chewing through context most of the day, and a nightly extraction job crawling the whole collection can share one machine and still want opposite things from a rebuild, regardless of whether it improves retrieval for all three equally.

one shard, one baseline, one decision

Capture a baseline first: one named model, one fixed prompt set, the exact server command, written down with the model artifact and prompt template. Those two vanish first, and losing either voids the comparison later. Run the real test on a representative shard, extrapolate full compute and storage from it, and keep the old index serving traffic until the new one clears validation. Retire it early and the experiment becomes an outage you caused.

Track a request end to end: time to first token for loading and prompt handling, steady token rate for decoding, completion time for what the person waiting feels. Add peak memory, queue delay, and wall power when any would change the call. Count completed valid runs per hour for the batch job, and slow waits for the interactive path, since that's what people remember.

One plain entry per run: artifact, runtime, launch flags; workload and fixed input set; cold and warm start at p50 and p95; peak memory and wall energy; failures and abstentions; then decision, owner, retest date.

A benchmark without a decision attached to it is trivia. A setting nobody owns turns into folklore.

soft failures don't throw exceptions

The expensive way to learn nothing is changing chunk size, overlap, and embedding model in one rebuild, then eyeballing the answers and calling it done. It survives because the system still produces plausible text. Most local inference failures stay quiet: a model partly offloads onto the CPU, a cache stops hitting, swap grows, a queue holds requests from a disconnected client, a fallback silently moves a request across a privacy boundary. None of that raises an error.

Watch runtime logs and OS counters while the test runs, not just the final number. Change one variable at a time unless comparing whole configurations. Repeat enough to separate a durable gain from a lucky sample, and read the outputs, because speed isn't proof two indexes agree. If a change makes one task worse, report that cost beside the gain.

Simplicity earns a line too. A fragile optimization that needs hand-repair after every driver or model update isn't free, it's a bill paid in your own time. I'd rather run something boring and reproducible from a service file than a clever setup only I understand, and I'd rather keep hashes and commands than raw artifacts, so a rerun means what mine did. I wouldn't log wall power for an index this small; it's not what bites you at that scale.

the margin is the point

The rule worth protecting above the rest: make reversible experiments cheap before you scale anything past a shard. Less exciting than the newest embedding model or the biggest vector store, but it leaves a stack whose limits are visible. A visible limit gets routed around or priced into next quarter. An invisible one turns into unexplained waiting and an emergency upgrade nobody budgeted for.

Stop tuning once the workload clears its quality and latency target with real headroom, not the bare minimum that passes today's test. That margin isn't idle capacity; it absorbs the longer document someone uploads, the extra user who shows up, a warm afternoon, or whatever the next runtime release changes underneath you.

The tradeoff I take on purpose: I ship the sharper chunking a week later than I could, in exchange for never explaining an outage nobody can reconstruct. What that costs is the day-one win, the chance to say the new index is already live and better. What it buys is a system whose failure modes are ones I chose in advance, not ones the rebuild picked for me.