Merge a LoRA or load it dynamically?
You've got a LoRA adapter that cleared eval last week. It has to serve real traffic tonight. Bake it into the base weights and ship one artifact, or keep base and adapter apart and attach the adapter on request.
one file versus two moving parts
Merging bakes the adapter into the base checkpoint: one file, no attach step, no runtime hot-swap dependency. Dynamic loading keeps the base resident once and applies the adapter's delta at attach time, so several adapters share one base instead of duplicate checkpoints eating VRAM.
It comes down to four things: adapter count, switching frequency, clean hot-swap support, and how release treats a model version. One adapter, rarely rotated: merge. Several an hour, or different customers on different fine-tunes: dynamic, or you'll burn through shared hardware-for-local-llms VRAM.
the provenance question nobody logs
Before benchmarking, pick the number you're moving: first-token latency, jobs per hour, resident adapters, energy per task, fewer corrections. Make it faster doesn't say when to stop.
Pin everything: base revision, adapter rank and target modules, tokenizer, prompt template, runtime build, launch flags, sampling settings. Trained it yourself via lora-fine-tune-at-home? You've got the base revision. Run once cold if users will hit that path, then warm long enough for queueing and memory pressure to show up, thermal throttling included. Break it into phases:
- queue wait
- adapter attach or merge-load
- prefill
- time to first token
- decode to completion
- peak RAM, VRAM, swap
- quality gate: pass, retry, abstain, repair
Raw tokens per second is a diagnostic, not the deliverable: count valid records per hour for batch automation, or track review time and first-token latency for anything interactive.
The failure that bites: merging without recording which base revision and which adapter parameters went in. A merged checkpoint is one opaque blob: nobody can trace or roll it back later. Local runtimes hide it, offloading layers or falling back to a generic kernel while still answering. Change one variable at a time and keep outputs, not just metrics, since quantization or context tricks can shave time while changing the answer, so the quality gate runs on every candidate.
boring is the point
A merged artifact is one thing to version and roll back. Dynamic loading is a base plus adapters plus routing logic: more surface to keep reversible. A five percent gain from an undocumented patch or hand-warming isn't worth it past your own laptop. Leave headroom instead of packing VRAM to the edge: free memory for prompt variance, spare queue capacity for interactive users. Thermal margin keeps the box alive at 2am. None of it's a vendor claim: every layer here is inspectable.
Write it down: workload, date, why, and the retest trigger, new base model, driver update, longer contexts, different traffic.
If you can't reproduce a merge from a log entry, don't merge it.