A RAG stack with the wifi off
the client who turned the wifi off
The security lead did it himself, in the April kickoff. We were pitching a documentation assistant for maintenance crews (a manufacturing client about an hour out of Brno) and he reached over, toggled wifi off on the demo laptop, and said we could continue whenever we were ready. Their rules were not subtle: 23,000 pages of maintenance manuals, work instructions and audit reports, and none of it leaves the building. No cloud embeddings, no reranker API, no telemetry. Whatever we built had to run with the radio dead.
So that became the acceptance test: the whole pipeline, in airplane mode, on a laptop.
two processes and a file
The stack ended up almost embarrassingly small. Ollama serves bge-m3 for embeddings (half the corpus is Czech, so multilingual wasn't optional) plus an 8B generator, and SQLite holds everything else: sqlite-vec for vectors, FTS5 for keyword search, one database file for the entire corpus. I'd assumed we'd need a proper vector database until I did the math on their scale, which is the argument from do you need a vector database played out in miniature. 118,000 chunks with int8-quantized embeddings is about 120 MB, and a brute-force scan over all of it takes around 70 ms on the M3 Pro MacBook I develop on. Their corpus grows by maybe forty documents a month. Indexes are a problem I'd love to need someday.
Local embeddings were the piece I trusted least going in, and they held up better than expected (field notes in local embeddings via Ollama), with one caveat that shaped everything downstream: a small embedding model bridges vocabulary less bravely than the big hosted ones, so retrieval quality became a chunking and query problem more than a model problem.
chunking cost me the first week
My first pass was the lazy one: fixed 512-token windows with overlap. The demo looked great, because demo questions are softballs. Real technicians got answers that quoted a torque value while the exception right below it had landed in the next chunk: do not apply above 40 °C. In maintenance docs the dangerous sentence is usually the one after the useful sentence. I re-chunked on heading boundaries, kept tables whole, capped chunks near 350 tokens, and prefixed each with its full heading path so the embedding knows a row about seal replacement belongs to pump family 300, not to the manual in general.
Retrieval moved more from that re-chunk than from any model swap I tried. Structure first.
hybrid, because part numbers don't embed
Cosine similarity is charming about prose and hopeless about identifiers. Query E-417 and an embedding model will happily return chunks about E-415, which are similar in every way except being wrong. FTS5 with BM25 catches the exact token; the embeddings catch the feeder-arm question nobody ever phrased that way in the docs. I merge the two lists with reciprocal rank fusion. Boring, no tuning, works. On the 58-question acceptance list we built with their crew leads, hybrid produced 49 answers rated usable against 41 for embeddings alone. That test set is tiny and biased toward their pain points, which is exactly why I trust it more than a public benchmark; RAG that retrieves is the longer version of that argument.
The wifi-off constraint is a quality budget: you pay roughly fifteen percent, and the kickoff is where you say so out loud.
the fifteen percent I couldn't buy back
The honest gap showed up in one query type: technicians asking in their own register. The manual says resonance above 3,200 rpm; the fitter types that the spindle hums when it warms up. A cloud pipeline with a proper reranker bridges that phrasing gap noticeably better: I checked, replaying nine failing queries against sanitized snippets on a hosted stack, and it recovered six of them. Our offline stack had no rescue layer. The best I managed locally was query rewriting, where the generator reformulates the question into manual-speak before retrieval. That clawed back three of the nine, at the cost of about a second of extra latency.
I told the client that number instead of hiding it. They took the trade in about four minutes. The alternative was a fitter spending forty minutes in a PDF viewer.
what it costs in seconds
Latency, honestly: embedding the query runs about 45 ms, the vector scan 70, BM25 under 20, fusion is free. Call it 150 ms to have the right passages in hand. Then generation eats 10 to 16 seconds for a sourced answer on my MacBook, and the small workstation they eventually deployed runs it a touch quicker. No retrieval cleverness hides a local 8B writing slower than a cloud API, and nobody complained once. The wifi stayed off, the audit passed, and the whole thing still fits in two processes and a single file I back up with cp.