Fine-tuning on the office H100 pair: what two 96 GB cards buy you

The question arrived about a month after the DL380 went into service: "can we fine-tune on it, or is that a cloud thing?" The answer is yes, with asterisks, and the asterisks are more interesting than the yes.

Two 96 GB cards with an NVLink bridge put you in a specific weight class. You can LoRA anything up to and including 70B-class models comfortably. You can QLoRA the same models on a single card with room left over. You can full fine-tune up to roughly the 8B class. And you cannot pretend you are a training cluster, no matter how good the NVLink numbers look on the datasheet.

where the memory actually goes

The arithmetic that decides everything: a full fine-tune needs weights, gradients, and optimizer state. With bf16 weights and a standard AdamW setup you are paying on the order of 16 bytes per parameter before activations. For an 8B model that is ~130 GB of state (shardable across the pair with FSDP, tight but real). For a 70B model it is north of 1.1 TB. Not on two cards, not with any amount of cleverness, and anyone who tells you otherwise is describing a different technique and calling it full fine-tuning.

LoRA changes the game because the frozen base contributes only its weights. A 70B in bf16 is ~141 GB, which FSDP shards to ~70 GB per card, leaving room for activations with gradient checkpointing on and sequence length at 4k. It fits, but it is the kind of fit where you watch nvidia-smi the first hour. QLoRA is the comfortable path: the 4-bit base is ~39 GB, one card holds everything, and the second card buys you data parallelism instead of survival. On our box the quality delta between bf16-LoRA and QLoRA on a 70B has been within eval noise for every task we've tried, so QLoRA won by default.

the tooling path we standardized on

We evaluated axolotl and torchtune and standardized on axolotl: torchtune is cleaner code and I'd pick it for anything requiring surgery on the training loop, but axolotl's config-file-in, adapter-out workflow is what lets teammates who don't live in PyTorch run their own jobs. A minimal config that matches what we actually run:

base_model: meta-llama/Llama-3.3-70B-Instruct
load_in_4bit: true
adapter: qlora
lora_r: 32
lora_alpha: 64
lora_target_linear: true
sequence_len: 4096
sample_packing: true
micro_batch_size: 1
gradient_accumulation_steps: 8
num_epochs: 2
learning_rate: 0.0002
bf16: true
flash_attention: true
gradient_checkpointing: true
datasets:
  - path: data/support-tickets-chatml.jsonl
    type: chat_template
val_set_size: 0.05
output_dir: ./out/tickets-qlora-r32

Launch it, go to dinner. On our pair, two epochs over ~4,800 instruction samples at 4k sequence length with packing lands in the five-to-seven-hour range. Hours, not days, though I will hedge that number hard because it swings with sequence length, packing efficiency, and rank. An 8B under the same config finishes in well under an hour, which changes how you work: at that price, hyperparameter sweeps become casual.

data prep is still 80% of the job

I want to be honest about where the calendar time went on our first real fine-tune, because it was not GPU time. We spent four days on data and seven hours on training.

Dedupe first, and not just exact-match: near-duplicate detection matters, because production data is full of the same ticket pasted forty times with different greetings. Our first run skipped this and the model learned to end every response with one specific customer's sign-off boilerplate, verbatim, because it appeared 312 times in a 5k-sample set. That model was useless and the failure was entirely self-inflicted.

Then decontaminate: anything resembling your eval prompts gets pulled from training, or your before/after comparison is fiction. Then format everything into the model's actual chat template (not just any chat template, the one the base model was trained with), because template mismatch is the quietest way to torch a fine-tune. axolotl's chat_template dataset type handles this if your JSONL is honest about roles.

The GPU pair determines whether you can fine-tune. The dataset determines whether you should have bothered.

eval before, eval after, no exceptions

Before any training run starts, we freeze a held-out set (around 150 prompts sampled from real traffic) and run the base model against it. After training, same prompts, same sampling settings, and both result sets flow through the inference logging setup so the comparison is diffable rather than vibes. For format-compliance tasks the scoring is mechanical (did it emit valid JSON in our schema: base model 0.71, tuned 0.97 on the last run). For tone and style we use a judge model plus mandatory human eyeballing of twenty transcripts, because judge models are cheap and confidently wrong in correlated ways.

The rule that made this stick: no adapter ships to the shared endpoint without a before/after table in the PR. It takes twenty minutes and it has killed two adapters that felt great in ad-hoc testing.

when not to fine-tune

Most requests that arrive phrased as "we should fine-tune a model on X" are not fine-tuning problems. If the goal is for the model to know things, like product docs, current pricing, or the contents of the wiki, that's retrieval, and RAG will beat a fine-tune on cost, freshness, and honesty about sources every single time. Facts baked into weights go stale the day after training and hallucinate at the edges forever.

Fine-tuning earns its keep for behavior: output format the model won't reliably hold from a prompt, house style, domain vocabulary where the base model keeps reaching for the generic term, and shrinking a 2,000-token system prompt into the weights of a small model you call ten thousand times a day. Our single most valuable adapter to date is an 8B tuned to emit our internal ticket-routing schema, not the 70B anything, and it replaced a prompt so long it was most of our token bill.

Try the system prompt first. Try RAG second. Fine-tune third, and only when the first two demonstrably failed.

the one-box problem

Here is the limitation nobody puts in the tutorial: training and serving want the same silicon. A 70B QLoRA run occupies most of one card; our GPU 1 is carved into MIG slices running standing services, and MIG has to come off entirely before a training job can use the card whole. So real training runs happen at night: MIG geometry torn down after the last CI run, the job queued into the overnight batch window, slices restored before the morning stand-up. It is exactly as glamorous as it sounds, and it works. The alternative, evicting the daytime serving model, was tried once, generated three Slack complaints in eleven minutes, and was not tried again.

If I could change one purchasing decision, it would not be a third GPU for training. It would be accepting earlier that overnight is a fine time to train, that 8B adapters solve more real problems than 70B ones, and that the pair is already more fine-tuning capacity than a team our size can feed with clean data. The bottleneck is the dataset. It was always going to be the dataset.