Tokens per watt makes sense on small boards
An idle desktop GPU pulls more power than most single-board computers pull flat out, and that fact wrecks half the comparisons people make between them. A small model on a Raspberry Pi is answering a different question than the same model on a workstation: fast versus cheap and unattended for months. Confuse the two and you will spend a weekend chasing tokens-per-second numbers that do not matter for the job.
The parcel test
Before any hardware gets ordered, write the job in four lines: event, input, output, deadline. "Add AI to the camera" fails, it is a mood, not a job. "When a parcel sits inside the porch zone for twenty seconds, generate one local notification with a thumbnail" passes, and usually shows that half of what you imagined needing a model does not.
With that sentence written, meter the wall, not the chip's spec sheet. Plug the board in, include idle draw, run a fixed batch of the real workload, and work out the daily cost. Use footage from the actual room, not a clean test clip: a television in the background or a long cable moves the result more than a better model does.
A short record per run keeps the numbers honest:
| What to log | Why it matters |
|---|---|
| Board, OS image, power supply, cooling | Repro depends on the whole appliance |
| Model artifact and runtime version | A quant bump moves latency and power together |
| Cold, warm and sustained latency | One number hides throttling the others catch |
| False action, missed action, recovery path | Accuracy without recovery is not a finished feature |
None of this replaces an accuracy figure, it sits next to one. A system that mostly gets the classification right but fires duplicate notifications, or forgets everything after a power cut, is not a finished feature just because the accuracy number looks fine. It is unfinished, full stop. I gave up chasing tokens-per-second gains on small boards; that is not the axis this job gets graded on. Anyone else in the house should see its state without a terminal, because the model or the network will be what breaks.
Nothing generative touches the lock
A failure worth naming: treating a chip's rated TDP as the system's measured energy, then acting surprised the real number is worse. That is a spec-sheet error more than a hardware one, fixed by the same wall-meter discipline above.
The bigger boundary is what the model gets to touch. It can read a request, summarize a history, or rank options, and be wrong about any of it without anyone getting hurt. It has no business standing alone between a request and a lock, a heater, an alarm, a valve, or a person. Put the real limits, permission checks, timeouts and interlocks into deterministic code, not into the model. A camera frame, an MQTT payload, a voice transcript: all of it is data, not an instruction, however command-shaped it reads.
Privacy needs the same thinking, since local inference closes only one leak. The frame never leaves as an API call, but the recording, the log, and the backup might still hold a copy. Decide up front what gets kept, how long, and what a debugging session exposes. The most private frame is the one never recorded; the next best is discarded the moment the decision is made.
Wait for the enclosure to get warm
Run the thing under a service manager, or a small pinned container, not a shell you left open. Keep configuration separate from the model artifact, so swapping a quant does not mean rebuilding the deployment. Give it a health check that proves the feature works, not just that a process holds a port open. Back up the controller state, note the model hash, and keep a known-good disk image ready.
Power and heat belong in the acceptance test, not a footnote. Let the enclosure sit under load until temperature stops climbing, and measure at the wall throughout. If the board throttles, swaps, corrupts storage, or ties up your home-automation host, the deployment is not done, however good the best inference result looked in testing.
The rule that holds up: pick the platform that finishes the duty cycle inside a latency and energy budget you can live with, and treat the model as a narrow layer above sensors you trust and below a policy someone wrote down. When the model goes away, the automation should get dumber, not stop. A board that falls back to a plain rule during an outage has done its job; one that locks you out because an inference server crashed has not.
So before ordering a second board, do the boring thing first: run the job for a full day, enclosure closed, meter on the wall, and compare what the idle hours cost against the busy ones. That number, not a TDP figure off a spec sheet, tells you whether the board on your desk is the right one.