One hundred clean examples can beat ten thousand scraped ones
Spend the first hours of any local fine-tuning job on the data, not the launch flags. Given a choice between a hundred examples you have read and ten thousand you scraped and never opened, take the hundred. That's the whole rule. Everything below just earns it.
A base model absorbs contradictions during pretraining because it's seen billions of examples and the noise washes out in the average. An adapter trained on a few hundred rows gets no such luxury: every duplicate, every inconsistent label, every shortcut an annotator took on a Friday afternoon gets treated as signal. Try this on a local LoRA setup and a bad row costs you a full retrain.
the annotator's bad day becomes permanent policy
The quiet failure in local fine-tuning is training on model-generated examples without checking they encode the decision you wanted. It's tempting: bootstrap a dataset from a bigger model's output and assume the right shape means the right answer. It doesn't. A fluent, well-formatted example can still teach the wrong lesson if the model that generated it took the same shortcut. Read every example yourself, not a sample of ten. Deduplicate hard: a near-duplicate row just doubles the weight of whatever bias it carries. Balance failure modes on purpose, not by what was easy to collect, and hold out data you never train on, to check the adapter later through an actual eval harness. None of it is glamorous. It's also the one place where effort reliably buys quality, more than another epoch ever will.
a number you can't reproduce isn't a result
Before touching a launch flag, pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, and write it down. Test on real workload inputs, awkward cases included. Run once cold and unloaded if that's a path real users hit, then run warm until cache effects, queueing, memory pressure, and thermal throttling show up. Time the phases: queue wait, load, prefill, time to first token, decode rate, peak memory and power, and retry-or-repair rate. Raw tokens per second is a fine diagnostic and a poor final answer. For automation, count valid completed records per hour. For coding, count the human time spent fixing output, not generating it. For chat, check p50 and p95 time-to-first-token on conversations as long as your real ones. A short, warm, single-turn benchmark win can lose once model swaps and long contexts arrive. Be suspicious of your own harness: local runtimes keep serving requests while quietly offloading layers or falling back to a slower kernel, so confirm the optimization is running.
write the reason down, not just the number
Change one variable at a time unless you're deliberately comparing two complete systems. Keep the actual outputs, not just the metrics: quantization, context compression, sampling, and model changes can all shift the answer while the speed number improves. Run the quality gate on every candidate, not just the favorite. If an answer got shorter, check it's still complete; if an extractor got faster, count valid records, not braces that happened to parse. Operational cost sits beside speed: startup, upgrades, observability, rollback, rebuilding the box after a disk dies. A five percent gain resting on an undocumented patch or a hand-warmed cache is a bad trade for anything you plan to keep running. Boring configurations age well. Write the conclusion into the result file: workload, date, reason, and what should trigger a retest: new model family, driver change, longer contexts, another user, different traffic mix. Skip that and old numbers quietly turn into folklore. Leave headroom after hitting the target: free memory absorbs prompt variance, spare queue capacity keeps a batch from starving interactive users, thermal margin gets you a full day, not a demo.
The tradeoff I'll take, every time, is coverage. A hundred examples I've actually read won't teach the adapter every edge case in your workload, only the ones I checked by hand. Ten thousand scraped rows would cover more ground in theory, and in practice teach the model every formatting quirk and lazy shortcut sitting next to the behavior I wanted. I'd rather ship something narrow and correct than broad and quietly wrong, so I give up breadth on purpose and add the missing cases in small, checked batches.