The LoRA I trained on a weekend (and what it fixed)

the 1,900-token preamble I was tired of paying

My local review assistant (an 8B coder that drafts first-pass review comments before I edit them) needed a 1,900-token preamble to behave. House format: severity tag, file and line, one-line problem, one-line fix, no praise, no hedging. With the preamble it complied on maybe seven drafts out of ten. Without it, essays. Every call paid those 1,900 tokens, every tenth draft got reformatted by hand anyway, and somewhere in mid-June I got tired enough to try the thing I'd been telling other people was usually procrastination.

Fine-tuning, at home, on the old 3090 in the hallway closet.

the dataset I'd been collecting by accident

The training data already existed, which is the only reason this was a weekend and not a quarter. Since winter, my pipeline had been logging every draft the model produced next to the version I actually posted, originally for the reasons in feedback loops, not for training. That log held 2,140 draft-to-final pairs. Saturday went into turning them into a dataset: dedup, strip anything client-identifying, drop the pairs where my edit was a full rewrite (that's not style, that's disagreement), drop a handful where past me was plainly wrong. 1,660 pairs survived: small by the standards of anything you'd read about, plenty for style.

Data work took seven of the weekend's eleven hours, and I had budgeted two.

sunday: three runs on the 3090

Tooling was unsloth-style QLoRA: 4-bit base model, low-rank adapters on top, consumer-card memory budget. The 8B trained at sequence length 2,048 and peaked around 10.8 GB of VRAM, comfortable on a 24 GB card and allegedly possible on 12:

r=16  alpha=32
lr=2e-4  seq_len=2048
batch=2  grad_accum=8
epochs=3   # the mistake

Run one took 2 hours 50 minutes and produced a monster I'll get to. Run two dropped to a single epoch and undershot: better format adherence, still drifting on long diffs. Run three, two epochs with about 15% generic instruction data mixed back in, landed it. The mixing trick is old advice. I'd read it, nodded, and skipped it anyway on Saturday. Total GPU time stayed under eight hours across the weekend, which is less than I've spent tuning prompts for worse outcomes.

what it fixed, and what it refused to

The result is honest and narrow. Format adherence went from about seven in ten to nineteen in twenty across the next fortnight of real reviews. I counted, because counting is the only defense against wishful thinking. The 1,900-token preamble is gone, so every call is cheaper and first-token latency dropped by about a third. House vocabulary stuck: it writes blocker instead of critical issue, it stopped complimenting code, it keeps the fix to one line.

It did not get smarter. The findings themselves (the actual judgment about whether the code is wrong) stayed exactly as mediocre as the base model. It now formats a wrong finding beautifully.

The LoRA made the model sound like me. It didn't make it think like me, and no amount of rank was going to change that.

That's the line I'd draw for anyone tempted. Fine-tune for form: output format, tone, house vocabulary, retiring a giant preamble. Prompt or swap models for competence. If the itch you're scratching is that the model isn't clever enough, a weekend LoRA is procrastination with a progress bar, and a stronger base from the current local coder crop will beat your adapter without a single training step.

the forgetting scare

Run one, the three-epoch one, was the scare. The adapter worked beautifully on review comments and had quietly bulldozed everything else. Casual questions came back wearing severity tags. Tool calling broke: it wrapped the JSON in review format, once inside a markdown table. My standing 20-prompt gauntlet, the one from my model-update routine, fell from 16 to 11. Nothing in the training loss warned me; the loss curve was gorgeous the whole way down. Catastrophic forgetting doesn't announce itself. You have to go looking for it with prompts that have nothing to do with your dataset.

Two epochs plus the general-data mix brought the gauntlet back to 15 and kept the format, and I can live with a one-point tax. I also keep the adapter as an adapter (swappable, deletable) rather than merging it into the weights, because the churn is real: when a stronger base drops, and it will, this LoRA turns into technical debt with my name in the metadata. A format shim, retrained quarterly at most. That's what a weekend buys, and for once it was exactly what I needed.