A Raspberry Pi cluster is not one large LLM computer

Wire four Raspberry Pi 5 boards into a cluster, point a sharding backend at them, and ask for one big model split across the fleet. People have already tried this and hit the same wall. Every boundary between boards becomes a network hop, so a decode step that costs nanoseconds inside one chip now waits on Ethernet to hand activations to the next node, and the cluster ends up slower than a single board running a model half the size. The extra boards were supposed to add up to a bigger brain. Instead you get four power supplies wired to a laggier one. The boards aren't the problem. A dense decode is starved for memory bandwidth, and extra silicon on the LAN doesn't substitute for bandwidth that only lives inside one chip's own memory bus.

What a stack of small boards is actually good at is running several independent, smaller jobs at once, not one tightly coupled generation stretched across a network. Four Pi 5 nodes sitting around for household chores are plenty. What decides whether it works is everything around the board: storage, power supply, cooling, the OS image, radio or accelerator support, and the service downstream that consumes whatever the model produces. A model that runs once from a shell prompt is still several decisions away from being a feature anyone in the house can rely on.

What the parcel notification actually needs

Write the event, input, output, and deadline in four lines before any board gets racked. "Add AI to the camera" is a wish, not a job. "When a parcel sits inside the porch zone for twenty seconds, produce one local notification with a thumbnail" is testable, and testing it usually shows that half the pipeline doesn't need a model at all.

The first experiment worth running is spreading transcription, embeddings, vision, and automation across separate boards as separate workers, before anyone tries to shard one model across the cluster. Test against the room you actually have: the real microphone, camera angle, sensor, language, network. Edge deployments amplify ordinary variation. Evening light, a television talking in the background, a warm cabinet, a weak cable, or a busy Home Assistant database will move your results more than swapping model versions ever will.

The logbook that catches the boring failures

Keep a short field record for every run, on every board, in the actual room:

• board, OS, power supply, cooling
• model artifact and runtime version
• real input set and the action it should trigger
• cold latency, warm latency, sustained rate
• memory, temperature, wall power
• false action, missed action, recovery path

For anything that fires inside someone's house in real time, accuracy alone tells you almost nothing. Check whether the system duplicates notifications, survives a reboot, reconnects once the network returns, and fails safely when the inference service is down. A feature other people depend on needs visible state and a manual fallback. Nobody should have to know which container or quant is broken just to turn on a light.

Nothing generative gets to hold the lock

The opening failure, boards added to speed up one memory-bound decoder, is a special case of a bigger habit: treating the model as load-bearing where it shouldn't be. Generative output is fine at interpreting a request, summarizing history, ranking options. It has no business being the only thing standing between a person and a lock, a heater, an alarm, a valve, or a battery. Ranges, permissions, timeouts, confirmations, and interlocks belong in deterministic code that doesn't care what the model said. Camera text, calendar entries, webpages, MQTT payloads, and voice transcripts are untrusted data, not instructions.

Privacy needs the same discipline end to end. Local inference kills one upload, but recordings, event logs, thumbnails, backups, and any remote dashboard bolted on afterward can still leave copies scattered around. Decide what gets collected, how long it sits, who can pull it, and what a debugging session exposes. The most private frame is the one the camera never captures. The next best is the one thrown away the moment the decision gets made.

Treat it like plumbing, not a science-fair project

Run the thing under a real service manager, or a small, pinned container definition, not a terminal window someone forgot was open. Keep configuration separate from the model files you'll swap later. A health check should prove the intended function actually happened, not just that a process owns a port. Back up the controller state, record model hashes, and keep a known-good image ready for whichever node ends up somewhere remote.

Power and heat belong in the acceptance test, not an afterthought. Run it long enough for the enclosure to reach a steady temperature, and measure at the wall socket with normal peripherals attached. If a board throttles, swaps, corrupts its own storage, or hogs the machine running your home automation, it isn't ready, even if the one result you screenshotted looked great. Scale across independent household jobs, not transformer layers, and when the model genuinely isn't available, the automation underneath should get dumber, not stop.

That's the tradeoff I make on purpose, and I'd make it again. I give up the raw per-request speed of one machine with real, unified memory bandwidth, the kind no cluster of small boards will match for a single tightly coupled generation. In exchange I get cheap, independently replaceable workers and a project that keeps doing its one small job long after the benchmark numbers stop mattering to anyone.