Local inference does not eliminate redaction

A 12 GB card running a 14B-class GGUF has no separate compartment for "this text is privileged" versus "this text is public." Model weights, the KV cache, and whatever document you fed it sit in the same memory space, and once that space fills the system starts making its own calls about what gets swapped to disk, cached, or written to a log file you forgot existed. That's the part most local-inference pitches skip. Keeping a contract or an HR record off a vendor's API answers exactly one question about it, and nothing about what your own machine does with the text five minutes later.

Where the contract text actually ends up

Picture a document workflow processing contracts or employee records, on the same box as a private chat someone opens a few times an hour and a coding agent looping through a repo overnight. Three workloads, one machine, each wanting something different. The chat wants low latency, not throughput. The coding loop wants steady decode speed and tolerates a slow first token. The nightly extraction job wants completed, valid jobs per hour and doesn't care what any single request feels like. Treat them as one undifferentiated local-AI problem and you'll tune for the wrong thing, then wonder why the privacy story doesn't match what's on disk.

This is also where a fully offline setup earns its keep or quietly fails: embed those contracts into a vector store and that store holds a retrievable copy of privileged language, subject to whatever retention policy you apply to everything else. See going fully offline with local RAG for what that demands once the model stops being the bottleneck.

The notebook I keep next to the benchmark

Before touching a config file, I capture a baseline: one named model, one fixed prompt set, the exact command that launched the server, the model artifact and the prompt template. Skip that and the comparison stops meaning anything a month later. Then I map every place a copy of the text can land: logs, caches, the vector store, backups, screenshots taken for debugging. I minimize what gets logged, encrypt storage, set a retention window, and keep debugging artifacts isolated from the pipeline touching real documents.

The measurement follows the request through the whole system, not just the model. Time to first token catches loading and prompt processing; steady token rate describes decoding; completion time is what the person or the job downstream gets. I add peak memory, queue delay, and wall power when they're close to the decision. What I write down, every run:

• artifact, runtime, and exact launch flags
• workload and the fixed input set
• cold start, warm start, p50, p95
• peak memory and wall energy
• quality failures and abstentions
• decision, owner, retest date

That last line is the one people skip, and the one that matters: a benchmark with no decision is trivia, and a setting nobody owns turns into folklore within a quarter.

Swap, caches, and other things that don't throw errors

The failure I actually worry about isn't a crash. It's a system called private because inference runs locally, while raw prompts sit on disk indefinitely and nobody notices, because the output still reads as plausible text either way. Local inference fails softly: a model partly offloads to CPU under memory pressure, a cache misses without complaint, swap creeps up over a session, a queue keeps holding requests from a disconnected client, or a fallback path quietly changes which boundary your data crosses. None of that raises an exception.

Catching it means watching runtime logs and OS-level counters while the test runs, changing one variable at a time, and repeating enough runs to tell improvement from a lucky sample. Read the outputs. Speed is not proof two setups behave the same, and none of this replaces real output validation between the model and whatever consumes what it produces, the kind covered under guardrails and output validation. If a change makes a task worse, that cost goes in the record too.

Operational simplicity belongs in the same column. An optimization needing manual repair after every driver or model update isn't free, whatever the benchmark says. I'd rather run something boring, reproducible from a service file or a short script, than something clever only I understand at 2am.

The rule I carry forward: local processing is a foundation for data control, not the finished policy. It gives you a stack whose limits are visible, and visible limits get scheduled around or priced, where invisible ones turn into surprise waiting and an emergency upgrade. I stop tuning once the workload hits its quality and latency target with real headroom left over, and I leave that margin on the table on purpose. It absorbs the longer document, the extra user, the warm afternoon. It's the throughput I give up for a system whose failure modes I can see coming.