How to run a local LLM for coding: the complete setup guide

Running a coding model locally sounds like a weekend project. It's actually about fifteen minutes, mostly downloading. Here's the complete path from nothing to a private, zero-cost coding model answering from inside your editor.

Step 1: install a runner

Ollama is the lazy default: one binary, a model library, and an OpenAI-compatible server so your existing tools just work.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# or: brew install ollama   (macOS)

On a Mac, LM Studio (a GUI) and mlx-lm are great alternatives. All three lean on the same engines (MLX / llama.cpp). Stick with Ollama unless you have a reason not to.

Step 2: pull the right model for your hardware

Match the model to your memory (the picks by tier). At int4:

# laptop (16–32 GB):
ollama pull qwen3-coder:14b

# workstation (24–48 GB GPU / 32–64 GB Mac):
ollama pull qwen3-coder:30b

# server / big Mac:
ollama pull glm-5        # or kimi-k2, deepseek-v4

Ollama pulls quantized (GGUF) weights by default, so these fit the memory targets above. If a pull is bigger than your RAM, pick a smaller tag, because otherwise it'll swap to disk and crawl.

Step 3: run it (and get the endpoint)

ollama run qwen3-coder:30b      # interactive, to sanity-check it works

The important part: Ollama is already serving an OpenAI-compatible API at http://localhost:11434/v1. That one fact makes everything downstream trivial: anything that speaks the OpenAI API can point at it.

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "qwen3-coder:30b",
  "messages": [{"role":"user","content":"write a python function to debounce"}]
}'

Step 4: wire it into your editor

Point any OpenAI-compatible client at the local endpoint. A base URL and a dummy key is the whole config:

• Continue (VS Code / JetBrains): add an Ollama provider; it auto-detects local models.
• aider: aider --model ollama/qwen3-coder:30b.
• Cursor / others: set the OpenAI base URL to http://localhost:11434/v1 and any non-empty API key.

# or in your own code, the OpenAI SDK repointed:
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

That's it. You now have a private coding model with zero marginal cost, running on hardware you own.

Step 5: tune and troubleshoot

• Too slow? You're memory-bandwidth-bound; drop to a smaller model or a heavier quant. On a Mac, more unified memory = more headroom (hardware guide).
• Out of memory / it's swapping? The model doesn't fit, so pick a smaller tag or a lower-bit quant (int4 is the sweet spot; avoid sub-4-bit unless desperate).
• Bad output formatting? Constrain it: ask for JSON or use structured outputs. Most "local is unreliable" is formatting noise, not reasoning (squeeze the local tier).
• Long prompts are slow to start? Local prompt-processing lags hosted APIs on big contexts. Keep prompts tight, or escalate those steps.

Where to go from here

A local model is the free base, not the whole answer. The step that matters most is the local-first cascade: let local handle the verifiable 80% for free and auto-route the hard 20% to a paid frontier model. You get private, zero-cost inference for most of your work and frontier capability exactly where it's needed, and that's the whole point of running local in the first place.