Connect a local LLM to Home Assistant carefully

A Raspberry Pi wired to a porch PIR sensor, running a small local model through Home Assistant's conversation agent, can tell you a package is sitting on the step. It should never be the thing that decides whether to unlock your door.

People show up to this project in one of two moods. Some treat an Orange Pi like a miniature gaming rig and expect it to run anything with LLM in the name. Others write off the whole category because it can't hold a conversation like a hosted model. Both camps miss the actual opportunity: small, quiet jobs that live next to the sensor, keep running when the internet doesn't, and pull almost no power.

Here's the constraint that matters: the model's job is to turn a fuzzy human request into a narrow, pre-approved plan. It doesn't get to own the state of the house. Start with Home Assistant, a local conversation agent, and whatever scripts you already trust, and treat the board underneath as an appliance you're assembling, not a computer you're benchmarking. A model that runs from a shell prompt is still a few decisions away from a household feature.

Write the job on an index card before you shop for boards

Before any hardware decision, write the event, the input, the output, and the deadline, in four lines, no more. "Add AI to the camera" is not a job description, it is a wish. "When a parcel sits inside the porch zone for twenty seconds, send one local notification with a thumbnail" is something you can build against, and fail against.

Once you have that sentence, the first real experiment isn't a benchmark, it's exposing named intents and read-only context to the model, validating arguments, and confirming anything risky before it fires. Test it against the room you have, your microphone, your camera angle, your sensor, your language, your network, not a demo clip. Edge deployments amplify ordinary variation more than people expect: evening light, a television talking in the next room, a warm equipment cabinet, a flaky cable, a Home Assistant database busy doing something else, any one of these will move your result more than swapping model versions will.

Keep a running log for every candidate, and don't trust your memory of it a week later:

• board, OS, power supply, cooling
• model artifact and runtime version
• real input set and the expected action
• cold latency, warm latency, sustained rate
• memory, temperature, wall power
• false action, missed action, recovery path

For a feature that runs inside somebody's four-minute morning routine, accuracy is not the finish line. Check whether it fires the same notification twice, whether it comes back after a power cut, whether it reconnects once the network returns, and what it does when the inference service is down. If another person in the house relies on this, it needs a visible state and a manual fallback. Nobody should have to know which container crashed to turn on a light.

Where the model's authority ends

The failure I run into most often is generosity: someone hands the model every entity and service in the house and trusts the system prompt to supply the judgment. That's backwards. A language model is good at interpreting a request, summarizing what happened overnight, or ranking a few options. It has no business being the only thing standing between a request and a lock, a heater, an alarm, a valve, a battery, or a person. Ranges, permissions, timeouts, confirmations, device interlocks: all of that belongs in ordinary deterministic code, the kind you can read line by line and trust.

A short manifest makes the boundary concrete:

allowed_intents:
  notify.porch_parcel     read: camera_zone, motion_state
  light.living_room.set   confirm: required
  climate.set_temp        range: safe_min-safe_max, confirm: required
  lock.front_door         # never exposed to the model

Everything the model touches outside that list is data, not instruction. Camera captions, calendar entries, a scraped webpage, an MQTT payload, a voice transcript: treat every one of them as something a stranger wrote, because that's what they are. A model that will follow an instruction hidden inside a camera caption will follow one placed there on purpose.

The footage keeps existing after the decision is made

Running the model locally solves exactly one problem: it keeps a frame from leaving the building over the network. It does nothing about the copies that pile up afterward. Recordings, event logs, thumbnails, backups, a remote dashboard you check from your phone: any of these can hold on to the thing you thought you had kept private. Decide, in advance, what gets collected, how long it sits around, who can pull it up later, and what you are willing to look at when debugging a false trigger at midnight.

Write that down too.

The most private frame is the one the camera never captures in the first place. The next best is the one thrown away the instant the bounded decision gets made, thumbnail included. Everything past that point is a retention policy you are choosing, whether you write it down or not.

Power, heat, and backups decide if this survives

Run the thing under a real service manager or a small, pinned container definition, not a terminal window you forget is open. Keep configuration separate from the model file itself, so replacing one doesn't mean rebuilding the other. Add a health check that exercises the function you built, not one that confirms a process still owns a port. Back up the controller's state, write down the hashes of the models you are running, and keep a known-good storage image ready for any node installed somewhere inconvenient.

Power and heat are part of the acceptance test, not an afterthought. Let the enclosure sit until it reaches equilibrium, then measure at the wall with the normal peripherals attached, not the bench setup from the first prototype. If the board throttles, starts swapping, corrupts its own storage, or hogs the machine your home automation runs on, the project isn't done, no matter how good its best answer looked during testing.

The rule that has held up for me: Home Assistant executes the approved primitives, and the model stays outside the boundary that protects people and property. Edge AI earns its keep as a narrow layer sitting above sensors you trust and below a household policy you wrote on purpose. When the model goes down, the automation should get dumber, not stop.

That is the tradeoff I have made on purpose. I give up the version of this project where the assistant notices something subtle and handles it on its own initiative, because I never gave it room to interpret anything that mattered. What I get back is a porch light that turns on every single time, a door that stays locked when the network hiccups, and a board I can leave in a cupboard for a year without wondering what it decided to do while nobody was watching.