Budget home hardware for QLoRA honestly

QLoRA turns a 24 GB card into something that can plausibly fine-tune a mid-size model at home. Plausibly is carrying a lot of weight there: the quantized checkpoint you started from tells you almost nothing about what the training step itself will cost.

The file on disk describes frozen weights, and frozen weights are the one part of the budget that quantization actually shrinks. Everything else, adapter gradients, optimizer state, the activations kept around for the backward pass, and the multiplier that sequence length and batch size apply to all three, behaves closer to full precision than to the four-bit file you downloaded. That's why a card that infers a 13B model without complaint can still run dry minutes into an adapter run on that same model, same context length, batch size you assumed was modest. A plain LoRA fine-tune at home on a smaller base might have left you VRAM to spare. That slack doesn't carry over once you swap in a bigger model and lean on quantization to make room.

Five things quantization never touches

Before you touch a launch flag, write down what has to fit. "Make it fit" is not a plan.

frozen weights (quantized)
adapter gradients (full precision)
optimizer state (full precision)
activations retained for the backward pass
sequence length x batch size multiplier

Gradient checkpointing trades some of that activation memory back for recompute time. Turn it on before deciding the card is the problem, not after you've already ordered a bigger one. Run one short training step first, at the sequence length and batch size you actually intend to use, not the friendly example from a README. That single step tells you more about whether 24 GB is enough than any spec sheet or quant file size will. It is the same discipline that matters when sizing hardware for local LLMs, paid twice: once for inference, once for training.

Peak memory happens later than you think

Optimizer state does not exist until the first backward pass calls the update, so a run that loads cleanly and forwards a batch without complaint can still die on step one. Frameworks are stubborn about failing quietly, too: they offload layers, page memory, or drop to a slower kernel rather than throw an honest error. A degraded configuration can limp along until the longest sequence in your dataset finally tips it over.

Change one thing at a time (batch size, rank, context length) so you know which knob moved the peak, and watch VRAM through that first optimizer update, not just the forward pass. Track output quality alongside memory too. A faster run after a tweak isn't obviously better if it's quietly truncating the longer examples.

Size the card to the measured peak plus real headroom, not to the number that got one lucky run across the finish line. Write down the config, the date, and what would make it stale: a new base model, a longer context, a driver update. None of this is worth much if it only works with an undocumented patch or a batch size nobody remembers tuning. The setup that survives is the boring one.

So before deciding the 24 GB card is too small, or buying a bigger one: run one training step at your real context length and batch size, watch peak VRAM past the first optimizer update rather than the forward pass, and find out whether the hardware or your batch size is the optimistic one.