Squeeze the local tier: do everything you can before you pay

The local-first cascade lives or dies on one number: what fraction of steps the free local tier clears before anything escalates. Push that from 30% to 70% and your paid bill more than halves. Same quality. The work local handled was work it was capable of. This post looks at the bottom rung: how to make a local model earn its keep on as much as possible.

What local models are genuinely good at

A right-sized local coder (30–70B at int4) is reliable, not just usable, on a specific class of work. Send these to local by default:

• Classification and routing: "is this a bug or a feature?", "which file does this belong in?", "does this need the frontier?" Cheap, high-accuracy, and the local model's verdict often drives the rest of the cascade.
• Summarizing tool output before it hits context (logs, test output, search results). This is the context-compression lever run for free.
• Structured extraction: pull fields out of text, parse a stack trace, turn prose into JSON.
• Boilerplate and mechanical edits: scaffolding, renames, simple refactors, docstrings, commit messages, PR descriptions.
• Drafting: a first pass the paid tier refines rather than authors from scratch.
• Retrieval pre-filtering and reranking: narrow 20 candidates to 3 before a paid model ever reads them (RAG).
• Embeddings: run them locally, free, forever.

Notice the pattern: these are bounded, checkable, high-volume tasks, exactly the ones you do constantly and would hate to meter.

Five ways to push local further

Most "local can't do that" conclusions are really "I didn't set local up to do that." The levers:

1. Right-size the model. Running a 7B on agentic coding and concluding local is hopeless is an under-provisioning error, not a capability ceiling. A 30–70B int4 model on Apple Silicon or a 24–48GB GPU is a different animal. Match the model to the work before you judge it.

2. Constrain the output. A weaker model that rambles becomes a reliable one when you force the shape: structured outputs, JSON schema, or a grammar that makes invalid output impossible. "Local fumbles the format" turns into "local is dependable here" with one config change. This is the single biggest lever for the local tier.

Most of a local model's apparent unreliability is formatting noise, not reasoning failure. Constrain the output and a 30B model becomes trustworthy on tasks you'd assumed needed a frontier call.

3. Narrow the task. Local models do markedly better on small, well-scoped prompts than on sprawling ones. Decompose: have local do ten small pieces and escalate only the one hard synthesis, instead of handing the whole vague thing to local and watching it flail.

4. Give it an oracle. Local plus a verifier (tests, schema, linter, a property check) is trustworthy local, because you catch its misses for free and escalate only on real failure. Without a verifier you're guessing; with one, a fallible local model is safe to lean on.

5. Use it as a preprocessor for the paid tier. The compounding move: local filters and summarizes so that when you do escalate, the paid model gets a small, pre-digested prompt. This raises local's share and cuts the paid call's cost at the same time. (The mechanics of that handoff are their own post: building an autorouter.)

When to stop pushing local

Squeezing local is not a purity contest. Know where the rung ends:

• Deep multi-file reasoning, subtle bugs, novel design: frontier work. Forcing local here wastes time, and a failed local attempt is wasted compute you then pay to redo. Classify these straight to the paid tier.
• Latency-critical interactive steps: local prompt-processing is slower to ingest a big prompt than a hosted API (the hardware tradeoff). Sometimes the paid call is the right default, not the fallback.

The realistic target is local clearing 60–80% of steps (not tasks, steps). That's a huge win for your bill and entirely doable. Chasing 100% is how you end up with a frustrating local-only setup that does the hard 5% badly; the goal is a wide free base, not local purity.

Measure, then tune

Don't decide what goes local by vibes. Decide by the escalation log. Track local's success rate per task type (observability); the tasks where it succeeds 95% of the time should default to local, and the ones where it fails half the time should skip straight to paid. Let the data move the boundary. The local tier is a boundary you tune as you learn what your local model can actually carry, not a fixed config.

The free base is the foundation everything else in the cost stack sits on. Widen it deliberately, and most of your token bill simply never happens.