Separate the embedding server from generation
On one GPU, or one Apple Silicon box, I run retrieval and chat side by side, and the embedding model keeps losing the fight for memory. Every time someone reloads a 14B-class GGUF to try a new sampler, the tiny embedding model gets shoved off the card, and the next ingestion job eats a cold start nobody planned for. That's the whole problem.
Chat wants low latency per token and stays resident all session. Embeddings batch and mostly run unattended overnight. Same silicon, different shapes, only one gets to stay.
Give embeddings a lease chat can't cancel
Before touching a setting, name the actual job: interactive chat, code completion, document extraction, or an overnight batch. You can't tell if a change helped until you know what good means for that job. Give the embedding model its own endpoint and its own lifecycle, isolated from the generation model. Batch the ingestion calls and version the vector index with the model that built it. None of this is exotic; running embeddings through Ollama as a dedicated server gets you most of the way there.
What earns a place in the results file
Use the same inputs every run and save the launch command with the result. One warm sample tells you nothing; include a cold start, and run long enough to expose thermal throttling or a growing queue. Log output quality next to the latency numbers: a run that answers faster but differently has changed the system, not just the speed. Record only what could change your decision:
- time to first token
- prompt-processing and generation speed
- peak memory and wall power
- pass or fail against a real quality gate
Medians describe the typical run. A slow percentile catches the pause that erodes trust. Log the noise and setup friction too.
The reindex nobody notices happening
The costly failure is quieter than a slow query: someone swaps the embedding model, forgets to reindex, and the store ends up holding vectors from two incompatible spaces answering the same searches. "It loaded" is not a result. "The answer looked plausible" is not an evaluation. Check the runtime logs and the real memory footprint, not the flag you passed. Change one variable at a time and say what you expect before you run it, or the exercise turns into anecdotes with graphs attached.
A fragile five-percent win won't survive the next driver update, so don't build a platform to chase it. A short script, a dozen prompts, and a result file get you back to a known state in minutes. Test the smallest change first, keep headroom for what comes next, and stop tuning once the workflow clears its target.
The rule I keep is flat: an embedding model and its index share one version, and nothing else touches either while a query is in flight.