Use synthetic training data with a verification funnel

A single consumer GPU only holds so much at once, so the model generating your synthetic training data is usually on the same card, fighting for the same VRAM, as the model you're about to fine-tune.

Synthetic examples are proposals, not ground truth. Accept them wholesale and you're training a habit, not a skill.

Say you're running a local adapter project off a small set of trusted seed cases. Before touching a launch flag or swapping hardware, write down what needs to improve. Vague targets don't survive contact with a spreadsheet:

• first-token latency, because a fast average hides a slow p95
• accepted jobs per hour, not requests handled
• how many resident models you can keep warm at once
• energy per completed task, if you're paying for power or a fan
• fewer corrections a human has to make downstream

"Make it faster" tells you nothing. It doesn't say when to stop, or what you're allowed to break to get there.

Pin the run before you multiply it

Fix the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings first. Pull inputs from the real workload, not a demo, awkward cases included. If users will hit a cold, unloaded model, run that path once and record it. Then repeat the warm path long enough to surface cache effects, queue buildup, memory pressure, and thermal drift. None of this is exotic. It's the same discipline a lora-fine-tune-at-home setup needs, just one step earlier.

Time the phases, not the vibe

The real experiment comes next: generate diverse candidates, validate what you can check deterministically, route uncertain cases to a human, and keep provenance on every record. Record the phases, not one total number:

time in queue
model load / warm-up
prefill (prompt processing)
time to first token
decode throughput
peak RAM / VRAM / power / swap
verdict: pass, retry, abstain, repair

Tokens-per-second is a diagnostic, not the product. For automation, count valid completed records per hour. For coding, add the review and correction time a human still owes. For chat, look at p50 and p95 time-to-first-token across realistic conversation lengths. A setup that wins a short, warm, single-request benchmark can lose once model swaps and long prompts show up.

The model won't tell you it's degraded

Here's the failure that bites: unreviewed generated answers work their way into an eval set or a training set, and nobody notices until the model has learned to be confidently wrong. Local runtimes are almost too well-behaved for their own good: offload a layer to CPU, page memory, miss a cache, queue a request, fall back to a generic kernel, and they'll still hand back an answer without complaint. That willingness to keep working hides a degraded configuration underneath it. Read the startup log, check device placement, watch the OS-level counters, and confirm the optimization you asked for is active for the tensor shapes and context length you use in production.

Change one knob, save every output

Unless comparing two complete systems end to end, change one variable per run and save the actual outputs, not just the metric. Quantization, context compression, sampling tweaks, and model swaps can all make an answer faster and quietly wrong at once, so the quality gate runs on every candidate, no exceptions. If a response got shorter, check it's still complete. If the extractor got faster, count valid records, not braces that happen to parse.

Operational cost sits next to performance, not below it: startup time, upgrades, observability, rollback, the ability to rebuild the box after a disk dies. A five-percent win that depends on an undocumented patch or manual warming is a bad trade on anything shared with others. Boring configurations age well.

Write the expiration date next to the number

The rule that holds: use synthetic data to widen scenario coverage, and let human or programmatic checks, the kind evals-and-llm-as-judge is built around, protect the labels. Write the conclusion into the result file with the workload, the date, and the reason. Then write what should trigger a retest: a new model family, a driver update, longer contexts than tested, a different user, a shift in traffic mix. Skip that and the numbers turn into folklore.

Leave headroom once you hit the target, then stop. Free memory absorbs prompt-length variance and whatever else runs on the box. Spare queue capacity keeps an interactive user from feeling a batch job land on top. Thermal and power margin lets the thing run for hours, not minutes. Efficiency was never about pinning every resource to a hundred percent, it's finishing the work predictably on the cheapest, least fragile setup that clears the bar.

The one real edge local inference has: every layer is inspectable, so none of this has to stay a vendor's word for it. Measure the workload, keep the quality gate honest, and stop once the system is boring enough to ignore.

All that said, this is more process than most weekend projects need. If you're fine-tuning a toy adapter on a Saturday, skip most of it and just read the outputs yourself. The funnel earns its keep once something you don't check daily depends on what comes out of it.