The H100s work nights: our overnight batch queue

Our two H100 NVLs have a day job and a night job. From 07:00 to 19:00 they serve interactive traffic for the team: chat, coding assistance, the RAG stack, all the latency-sensitive stuff I covered in the serving writeup. At 19:00 a systemd timer fires, the serving containers drain and stop, and the same silicon starts grinding through a queue of batch jobs until 06:30 the next morning.

This arrangement came out of a slightly embarrassing observation. I pulled a week of DCGM utilization data last spring and found the pair averaged around 11% SM utilization between 20:00 and 06:00. Two GPUs that cost more than a decent car, doing effectively nothing for eleven hours a day. Depreciation does not sleep. So now the GPUs don't either.

what actually runs at night

The queue is not exotic. It's the pile of work that accumulates during the day because nobody wants to steal capacity from interactive users.

• Whisper large-v3 transcription of the meeting-recording backlog. We record a lot of meetings, and the whisper pipeline can chew through roughly 90 hours of audio per night on one GPU when nobody else wants it.
• Embedding backfills. Every time we change chunking or swap the embedding model, the entire doc corpus needs re-embedding (about 2.3M chunks at last count). The embedding server handles live traffic fine, but a full backfill during the day would starve it.
• Nightly eval sweeps. Every prompt change merged during the day gets run against our eval set overnight, so the morning standup starts with numbers instead of vibes.
• LoRA training runs, which I wrote up separately in the fine-tuning notes. A rank-32 run on our internal data takes 4 to 7 hours. A perfect night-shaped workload.
• Synthetic data generation for those same training runs. Millions of tokens of generated examples, zero urgency.
• The occasional big-model job that needs both GPUs and all 188-ish GB of usable HBM, which simply cannot coexist with daytime serving.

Nothing on that list cares about latency. Everything on that list cares about throughput. That's the whole design insight, and it's not a deep one.

the queue is a directory

I want to be upfront about how unsophisticated this is, because I think the unsophistication is the point.

The queue is a directory. A job is a file: a small TOML with a command, a GPU count (1 or 2), a max runtime, and a priority number. Submitting a job means writing a file into queue/pending/. A runner script, started by the systemd timer at 19:00, sorts pending jobs by priority, moves one to queue/running/, executes it, and moves it to done/ or failed/ depending on exit code. Two-GPU jobs wait until both are free. That's it. Maybe 180 lines of Python, and about 60 of those are logging.

We evaluated Slurm. Slurm is genuinely good software and if we had forty nodes I'd install it tomorrow. We have one node. Standing up slurmctld, slurmd, munge auth, and a partition config to schedule two GPUs in one chassis is putting on a hi-vis vest to cross your own living room. Every layer of scheduler is a layer someone has to debug at 06:40 when the morning handover is broken, and with the directory approach, debugging is ls.

An idle GPU at night is pure depreciation. The batch queue exists to convert dead capital hours into the cheapest tokens we will ever produce.

The one thing I'd defend as non-negotiable even at this scale: jobs declare a max runtime, and the runner enforces it. More on why below.

vLLM offline mode, not the HTTP endpoint

Early on, batch jobs talked to the serving endpoint over HTTP like everyone else. This worked and was also silly. You pay for HTTP framing, client-side concurrency management, request-level scheduling designed to keep p95 latency low. None of that matters when the only user is a script that wants everything done by dawn.

For pure generation jobs we now use vLLM's offline API directly:

from vllm import LLM, SamplingParams

llm = LLM(model="/models/qwen-72b-fp8", tensor_parallel_size=2)
params = SamplingParams(max_tokens=1024, temperature=0.7)
outputs = llm.generate(prompts, params)

You hand it the whole prompt list and it schedules the batch itself, keeping the GPUs saturated without any of the fairness machinery an interactive server needs. On our hardware the difference was roughly 1.6x throughput on the synthetic-data workload versus hammering the OpenAI-compatible endpoint with 64 concurrent requests, approximately. The gap narrows if you tune the HTTP client hard, but why bother when the offline path is less code.

The other win is deterministic memory. One process owns both GPUs for the duration of the job, gpu_memory_utilization set explicitly, no surprises from a serving stack that grew its KV cache while you weren't looking.

checkpoint like the crash is scheduled

The first month taught us this the honest way: a 6-hour synthetic-data run died at hour 5 because a malformed prompt tripped an assertion deep in a tokenizer. We lost the full five hours. Once is an anecdote; it happened again two weeks later with a CUDA ECC hiccup, and twice is a policy.

Now every batch job is required to be resumable. For generation jobs that means writing outputs in shards (one JSONL file per 2,000 prompts) and on startup, scanning for completed shards and skipping them. For the LoRA runs it means save_steps tuned so we never lose more than about 20 minutes of training. For the Whisper backlog it's trivial, since each recording is independently done or not done. The runner's retry logic is dumb on purpose: a failed job goes back to pending with a retry counter, three strikes and it stays in failed/ for a human.

Resumability costs almost nothing to build in from the start and is miserable to retrofit at 09:00 while the team asks where the eval numbers are.

the money argument

Rough math, hedged appropriately. The box was about $60k. Amortize over three years and it's roughly $55 a day whether the GPUs work or not. The 12-hour night window is therefore ~$27 of otherwise-wasted capital, plus electricity: call it another $5 at our tariff with the pair pulling hard.

On our hardware, a 70B-class model in FP8 with tensor parallelism across the pair does very roughly 4,000–5,000 output tokens per second in saturated offline batch. That's heavily dependent on sequence lengths, so treat it as an order-of-magnitude figure. Twelve hours at that rate is on the order of 180–200M tokens. Thirty-two dollars for that many tokens works out to somewhere around $0.15–0.18 per million output tokens, and the capital share of that was being spent anyway.

Compare that against API pricing for comparable open-weight models on /prices and the batch window is the cheapest compute we own, by a comfortable margin. The full accounting (power, cooling, the amortization assumptions I'm hand-waving here) lives in /local-cost. The point survives any reasonable set of assumptions: tokens produced in hours you already paid for are nearly free at the margin.

the 07:00 problem

Here is the thing that has actually bitten us, twice.

The night shift ends when the day shift begins, and the day shift needs clean GPUs. A batch job that hangs (or worse, exits but leaves a zombie child process holding 60-odd GB of VRAM) delays the 07:00 serving start. The first time this happened, vLLM came up at 07:00, found a fraction of the expected free memory, and crashed on startup in a loop until someone SSH'd in at 07:40. The team's morning was noticeably worse and I heard about it.

The fix has three layers. Per-job timeouts, enforced by the runner with SIGTERM then SIGKILL: no job runs past 06:15, period. Resumability makes this painless. Then a hard sweep at 06:30: a script checks nvidia-smi --query-compute-apps=pid,used_gpu_memory --format=csv and kills any surviving compute process outright, no appeals. Then a verification gate at 06:45: if free VRAM per GPU isn't above a threshold, the serving start is blocked and I get paged instead of the team getting a broken morning. In the stubborn zombie case, nvidia-smi --gpu-reset is the last resort, though it has only been needed once.

It's ugly. It's also the difference between a batch queue the team trusts and one they blame.

run your nights

If you own GPUs and they idle overnight, you are paying full price for half a product. You do not need Slurm, Kubernetes, or an orchestration framework with a mascot: you need a timer, a directory, resumable jobs, and a genuinely paranoid handover script. Ours took about two days to build and has produced billions of tokens of transcripts, embeddings, evals, and training data that would otherwise have been daytime contention or an API invoice. The host tuning that makes those nights faster is its own story, covered in the DL380 host notes, but the queue itself is a weekend project. Build the boring version. It will still be running in a year.