Benchmark AI on an SBC without fooling yourself
The rule is short: benchmark the box you are actually going to bolt to a wall or tuck into a cupboard, not a bare board on your desk with every background service killed and a fan blowing straight at the heatsink. Everything else here is just the argument for why that rule holds even when it is inconvenient to follow.
Small-board AI gets talked about in two unhelpful registers. Somebody treats a Raspberry Pi or an Orange Pi like a miniature gaming rig and comes away disappointed. Somebody else decides anything short of a frontier chatbot isn't worth running on a board that size. Both miss the point entirely. The actual opportunity is quiet, bounded jobs that sit near the sensor, keep working without a cloud account, and sip power the whole time.
The reason the rule matters is that small boards are unusually sensitive to conditions a desktop benchmark can shrug off. Start from the Pi or Orange Pi already sitting in its final case, running the services it will actually run in production, not stripped down to chase a good number. The name printed on the board matters less than the whole appliance built around it: the storage, the power supply, the cooling, the operating system, whatever radio or accelerator it leans on, and the service that consumes the result. A model that runs once from a shell prompt is still several decisions away from being a feature somebody's household depends on.
The porch-package test
Write the job in four lines before you shop for hardware: the event, the input, the output, the deadline. "Add AI to the camera" is not a job description, it's a wish. "When a parcel sits inside the porch zone for twenty seconds, raise one local notification with a thumbnail" is something you can actually test, and writing it out that plainly usually shows you which parts don't need a model at all.
Once you have that sentence, the first real experiment is a field record, not a leaderboard run. Capture the OS, the CPU governor, the cooling setup, the power supply, the model hash, thread count, a cold result, and a warm one, where warm means the runtime kept the weights resident the way keep-alive settings do on a workstation, then keep sampling under sustained load. Test against data from the room the thing will actually live in: your microphone, your camera angle, your sensor, your language, your network, not a demo clip. Edge deployments amplify ordinary variation in a way a desktop rarely does; evening light, a television talking in the background, a warm equipment cabinet, a marginal USB cable, or a Home Assistant database under load can move your result more than swapping model revisions does.
board, OS image, power supply, cooling
model artifact + runtime version
real input set, expected action
cold latency, warm latency, sustained rate
memory, temperature, power at the wall
false action, missed action, recovery pathThat discipline is the same one behind honestly measuring tokens per second on local hardware: the number only means something once you've written down everything that could have produced it.
For a feature running unattended in somebody's hallway, accuracy is the least interesting entry on that list. What matters is whether it fires twice for one parcel, whether it comes back cleanly after a power cut, whether it reconnects once the network returns, and what it does the moment the inference service simply isn't there. A feature other people rely on needs a visible state and a manual fallback. Nobody in the house should have to know which container, or which quantization, broke before they can turn on a light.
What the model doesn't get to decide
Draw a hard line around anything that can hurt a person, a pet, or the house, and keep the model outside it. Generative output is fine at interpreting a request, summarizing a history, or ranking a shortlist of options. It has no business being the only thing standing between an unlocked door, an armed alarm, an open valve, a charging battery, and whatever goes wrong next. Put ranges, permissions, timeouts, confirmations, and hardware interlocks in ordinary deterministic code, the kind you could read by flashlight at two in the morning. Treat anything a camera reads as text, anything from a calendar, a webpage, an MQTT payload, or a voice transcript, as data to be inspected, never as an instruction to be obeyed.
Privacy wants the same treatment, start to finish, not just at the inference step. Running the model locally stops one particular upload, but recordings, event logs, thumbnails, backups, and a remote dashboard can all quietly keep their own copies. Decide on purpose what gets kept, for how long, who can pull it, and what leaks out the side door during debugging. The most private frame is the one the camera never records in the first place. The next best is the one thrown away the moment a bounded decision has been made about it.
Run the whole setup as an appliance, not a demo you SSH into when someone asks how it works. A service manager or a small, pinned container definition, configuration kept separate from the model file so you can swap one without touching the other, and a health check that proves the actual function works rather than merely that something is listening on a port. Back up the controller state, write the model hashes down somewhere durable, and keep a known-good storage image ready for any node you installed somewhere annoying to reach.
Power and heat belong in that same acceptance test, not off in a spec-sheet fantasy. Let the enclosure reach thermal equilibrium and measure current draw at the wall with the normal peripherals attached, not on an open bench with a fan aimed at the heatsink. If the board throttles inside its own case, starts swapping, corrupts an SD card, or quietly monopolizes whatever else is running your home automation, it isn't ready, whatever its best single inference number looked like on a spreadsheet. I don't chase that number anymore.
So here is the one rule worth keeping: measure the installed appliance, in its case, under its own load, or you haven't measured anything at all.