Ollama or raw llama.cpp: when the training wheels come off

It took me an embarrassingly long time to internalize what Ollama actually is: llama.cpp under management. The code doing the real work when you type ollama run is the same engine you get by building llama-server from source; Ollama wraps it in a registry, a lifecycle manager, and an API that's listening on 11434 before you've had time to think about it. Once that clicked, the whole which-one debate collapsed into a better question: do I want to operate the engine myself, or pay a small abstraction tax to have it operated for me?

For most of last year the answer was obvious. I ran Ollama on the M2 Ultra, pulled whatever the interesting model of the week was, and never once thought about context allocation or offload layers. That's the product, it works, and I wrote up the daily-driver side in Ollama in practice.

Then, in early March, a client workload broke the arrangement.

the workload that needed the engine, not the wrapper

The job was ticket triage for a logistics client: every inbound support ticket gets classified, and the model emits routing commands in a tiny internal DSL, lines like assign carrier-eu sev2 dedupe. Not JSON. Ollama's structured outputs are genuinely good now, but they speak JSON schema, and this format was never going to become JSON, because the consumer is a Perl script from 2014 that nobody is allowed to touch.

llama.cpp has grammar sampling. You hand llama-server a GBNF file (a small context-free grammar) and constrained decoding masks the logits so the model physically can't produce a token that violates the format. Before the grammar I did what everyone does: prompt harder, regex the output, retry on failure. The retry rate hovered around 7% on a 7B model. With the grammar it went to zero. Not low. Zero, by construction.

A grammar file doesn't ask the model to behave. It removes misbehaving from the vocabulary.

The serving command that's been running since March:

llama-server -m qwen2.5-coder-7b-instruct-q4_k_m.gguf -c 16384 -ngl 99 --grammar-file triage.gbnf --port 8081

Two more flags earned their keep the same week: -c pins the context at exactly what the pipeline needs instead of a guess, and an evening of batch-size tuning bought a few percent of prompt-processing speed on the client's Ubuntu box. There's a whole zoo of these, and I catalogued the ones that matter in llama-server flags.

the speedup I expected and didn't get

Here's the reversal. I assumed dropping the wrapper would make things faster. Closer to the metal, whatever that means on a Tuesday. It didn't. Same engine, same GGUF, same quant; triage throughput moved maybe 3-4% after tuning, which is inside my measurement noise on that box. If you're going raw for speed, save yourself the weekend. The win is control, and control only pays when your problem has the shape of a specific flag.

Speed was never on the table.

what raw costs, in installments

The bill arrived over the following weeks, in small pieces. No registry, so I hunt GGUFs on Hugging Face and pick quants by hand. Choosing a quant is its own afternoon. No lifecycle, so llama-server loads one model and holds it forever; switching means a second process on a second port with its own unit file. And updates are on you: when I rebuilt from source in April, a flag I depended on had been renamed, the systemd unit failed at 2 a.m., and the pipeline sat dead until morning. That one was self-inflicted (pin your builds to a release tag), but notice that Ollama users don't even have this category of problem.

Everything a Modelfile gives you for free (templates, default parameters, keep_alive behavior), you now own as config that only you understand.

where I landed

Daily use came back to Ollama within a month, and honestly it had never fully left. New models on the Mac, quick questions, embeddings, anything interactive: the convenience layer wins there because my attention is the scarce resource, not the flags. The triage pipeline stayed on llama-server, systemd-pinned and grammar-locked, and it's processed somewhere north of 60,000 tickets since March without one malformed line.

Having lived on both sides, my honest gripe with Ollama is that it hides the engine slightly too well. Tags don't always make the quant obvious, engine-level knobs arrive late or arrive renamed, and when the offload guess is wrong you find out from slow generation rather than an error. None of that is disqualifying; it's the price of the wrapper, and most days I pay it gladly.

So my advice runs: start with Ollama, stay with Ollama, and keep llama-server in your back pocket for the day a workload names a flag you can't reach through the wrapper. The training wheels come off one workload at a time.