How fast should the network be for a local LLM server?
A single chat turn on a local model pushes a few kilobytes over the wire, read back one token at a time. Gigabit Ethernet, or a decent Wi-Fi link, has more headroom than that job will ever touch. The network gets interesting once you add model downloads, multimodal input, or a second machine reading the same weights off shared storage.
Three workloads, one box, different bottlenecks
Picture a home server reached over wired Ethernet and Wi-Fi, running a private chat used a couple times an hour, a coding loop, and a nightly batch extraction job. Three workloads, three different bottlenecks, not one number to optimize. Before touching a cable, write a baseline: one named model, one fixed prompt set, the exact launch command, or your next comparison means nothing.
Measure the pieces separately. For a chat session, latency and reliability matter more than raw link speed; upload time for a real attachment and model-copy time are separate concerns. Time to first token is loading and prompt work; steady token rate is decoding; total completion time is what the person waiting remembers. Background jobs get judged on completed valid runs per hour; interactive ones on the waits a person notices.
Keep the log boring:
- model, runtime, launch flags
- workload, fixed input set
- cold/warm start, p50 and p95
- peak memory and wall power
- quality failures, not just speed
- decision, owner, retest date
The upgrade that fixes the wrong thing
The trap I see most: buying 10 GbE to fix a slowdown that has nothing to do with the wire. A cold model load, a prompt too big for cache, a queue stuck behind a dead connection: they all feel like network problems, same wait, no error thrown. A model partially offloading to CPU because it didn't fit in VRAM looks exactly like a slow link (the offload math explains why). Swap creeping up, a cache miss, a fallback that changes where your data goes: same story, wrong diagnosis.
Watch runtime logs and OS counters while the test runs, change one variable at a time, and repeat enough to rule out a lucky run. Read what came back, not just how fast: a faster wrong answer just means you regressed with better numbers. An optimization that needs babysitting after every driver update isn't free. Keep the hashes and the exact commands even if you toss the raw artifacts, or the next run means something different and nobody notices.
Spend the upgrade where the bytes actually go
The rule that's held up for me: upgrade the network for the traffic that actually crosses it, nothing else. Real bandwidth goes to model downloads, to a homelab syncing weights across boxes, to large multimodal uploads, not a chat session's text stream. A visible limit gets planned around; an invisible one turns into a 2am page and a purchase order nobody can explain.
Stop tuning once the workload hits its latency and quality target with room to spare, room that absorbs the next long document, an extra user, or the runtime update you didn't ask for.
All of which falls apart once you're splitting inference across machines instead of running one box with laptops pointed at it. At that point the network is the architecture, not something you tune last, and most of this stops applying.