What a Raspberry Pi 5 can realistically do with a local LLM

An 8 GB Raspberry Pi 5, a compact GGUF model, and llama.cpp compiled with NEON support: that is the entire starting rig for a serious edge-AI project, and it will do far less than the marketing suggests and quite a bit more than the skeptics assume, depending entirely on the job you hand it.

A Pi 5 coordinates, it doesn't crunch

The board is a capable coordinator and a modest CPU inference node. It is not a stand-in for desktop-class memory bandwidth, and pretending otherwise is where most of these projects go wrong before they've written a line of code. The chip itself matters less than people think; what matters is the whole appliance wrapped around it: the storage, the power supply, the cooling, the operating system, whatever radio or accelerator hardware is attached, and the service that actually consumes the model's output. A model that runs once from a shell prompt is still several real decisions away from something a household can depend on. I wouldn't bother trying to serve open-ended chat off this thing, either; the second a person types something and waits for a reply, you feel every bit of that narrow memory bus, and the experience reads as sluggish no matter how you tune it. Narrow, bounded jobs near the sensors are the actual sweet spot: no cloud account required, low power draw, quiet.

Four lines decide the whole project

Four lines, before you buy anything: event, input, output, deadline. "Add AI to the camera" is not a job description, it's a wish. "When a parcel sits inside the porch zone for twenty seconds, generate one local notification with a thumbnail" is testable, and writing it out this way tends to reveal which parts of the pipeline don't need a model at all. Once you have that sentence, benchmark short classification, command parsing, and summarization separately from open-ended chat, because they behave nothing alike on this hardware. Test with data from the actual room: the real microphone, the real camera angle, the real sensor, the real language, the real network. Edge deployments amplify ordinary variation in a way that's easy to underestimate. Evening light, a television talking in the background, a warm equipment cabinet, a slightly loose cable, a busy Home Assistant database under load: any one of those can move the result further than swapping model revisions does. Keep a field record for every configuration you try.

board / OS image / power supply / cooling setup
model file and runtime version
real input sample and the expected action
cold-start latency, warm latency, sustained rate
memory use, board temperature, wall power draw
false action, missed action, recovery path

Accuracy on its own tells you almost nothing about whether this is ready for a four-minute household workflow. Check whether the system throws duplicate notifications, survives a reboot cleanly, reconnects on its own once the network comes back, and fails safely when the inference service isn't running. Anything used by other people in the house needs visible state and a manual fallback; nobody in your family should need to know which container or which quantization broke before they can turn on a light.

The model doesn't get a vote near anything dangerous

The failure mode I see most often is someone loading the biggest model that will boot and calling a thirty-second reply "interactive." Generative output is fine for interpreting a request, summarizing a history, ranking a few options. It should never be the only layer standing between a request and a lock, a heater, an alarm, a valve, a battery, or a person. The model doesn't get a vote. Keep the following in deterministic code, not in anything the model produces:

• ranges and limits on any physical action
• permissions for who or what can trigger it
• timeouts on anything that can be left running
• confirmations before anything irreversible
• hardware interlocks that don't care what the model said

Camera text, calendar entries, scraped webpages, MQTT payloads, voice transcripts: all of that is untrusted data, not instructions, no matter how well-formed it looks. Privacy needs the same full-path thinking. Running inference locally stops exactly one upload; it does nothing about the recordings, event logs, thumbnails, backups, and remote dashboards that can still leave copies of the same information somewhere else. Decide up front what gets collected, how long it survives, who can pull it later, and what happens to it on the day you're troubleshooting the thing in a hurry. The most private frame is the one the camera never captures in the first place. The next best is the one that's gone the instant the bounded decision is made.

This has to survive being ignored for a year

Run it under a real service manager, or a small container definition pinned to a version you chose on purpose, not whatever tag happened to be latest that afternoon. Keep configuration separate from the model file, since the model file is the part you'll replace. Write a health check that proves the intended function actually works, not one that just confirms a process is still holding a port open. Back up the controller's state, record the model file's hash somewhere durable, and keep a known-good storage image ready for any node installed somewhere annoying to reach. Power and heat belong in the acceptance test, not as an afterthought: run the thing long enough for the enclosure to reach a steady temperature, and measure power at the wall with the normal peripherals actually attached. If the board throttles, starts swapping, corrupts its storage, or hogs the machine that's supposed to be running your home automation, it isn't ready, even if the best single inference result you captured looked great.

The rule I actually keep is short: run the smallest model that reliably finishes the one task you wrote down, and when that model isn't available, the automation gets dumber, not dead.