Stacking it all: ultra token savings at the same quality

Each post in this series pulls one lever. This is the one where we stack them. The thesis: token savings compound, and the quality cost is near zero, because every lever here either removes tokens that carried no information or swaps a model down on tasks that never needed the frontier. Done right, the output the user sees stays the same. Only the bill changes.

The full stack

Five layers, input to output, each covered in its own post:

your code
   │
   ├─ 1. model routing        cheap model unless the task is genuinely hard
   ├─ 2. prompt caching       0.1× on the stable prefix you resend every turn
   ├─ 3. narrow retrieval     fetch 3 relevant chunks, not 20 plausible ones
   ├─ 4. context compression  Headroom: 60–95% fewer input tokens
   │                            ▼ (the model)
   ├─ 5a. terse prose          Caveman: ~65% fewer output tokens
   └─ 5b. lazy code            Ponytail: 80–94% less code, 47–77% cheaper

Layers 1–4 shrink what goes in; 5a/5b shrink what comes out. They're orthogonal (each acts on a different part of the request), which is exactly why they multiply instead of overlapping into nothing.

A concrete setup

Here's the whole thing wired for a coding agent, lazy-first:

# 4 - context compression in front of the agent (zero code change)
headroom proxy --port 8787
export OPENAI_BASE_URL=http://localhost:8787

# 5a + 5b - output shaping, on every session
/caveman full       # terse prose
/ponytail full      # lazy code

# 1 + 2 - routing + caching in your own calls
model = "haiku" if is_simple(task) else "opus"      # route by difficulty
messages = [
    {"role": "system", "content": STABLE_PROMPT,    # cached prefix - never edit mid-session
     "cache_control": {"type": "ephemeral"}},
    *history,                                        # volatile content AFTER the breakpoint
]

# 3 - retrieve narrow: top-3 reranked chunks, not top-20 (see /p/rag-that-retrieves)

That's it. A proxy, two skill commands, and a routing if. No model downgrade on the hard tasks, no quality knob touched.

The honest math

You cannot multiply every fraction and claim 99.9%: the levers overlap (caching and compression both act on input), and not every call hits every layer. But here's a defensible accounting for a cache-heavy agent loop:

Input side: caching and compression together land uncached-input cost well under 0.1× of naive. Output side: ~0.35×. Blend in routing, and a typical agent bill settles around 2–5% of where it started (a 20–50× reduction). Not the literal 99.9% the naive product suggests, but unmistakably an order of magnitude, often two. That's the number the cost-architecture post promised, and it delivered.

The reason quality survives: not one of these levers makes the model dumber. They remove filler tokens and route easy work to cheap models. The frontier still does the hard thinking. It just stops reading and writing things nobody needed.

Why quality holds (the part that matters)

Each lever is quality-neutral by construction:

• Caching is byte-identical content at a lower price. Zero effect on output.
• Routing uses the frontier exactly where it's needed; the cheap model only handles work it was always capable of. Calibrate with the benchmark.
• Compression is reversible. The model pulls full detail back on demand, and accuracy holds on standard evals.
• Terse prose often improves answers (brevity reverses performance hierarchies), just keep it on the working channel, not customer-facing text.
• Lazy code is a better default (less to maintain, fewer 3am incidents), with security and validation explicitly protected.

The harness layer: my actual Claude Code setup

Everything above is per-API-call. But if you run inside an agent harness (Claude Code, in my case), there's a second, orthogonal stack that shrinks the context the model reads on every turn. These levers don't touch the five above; they multiply with them.

• Graph over grep. A knowledge graph of the codebase (the code-review-graph MCP) answers "who calls this, what breaks if I change it, where are its tests" with a handful of structural nodes, instead of grep dumping twenty files into context to find out. My CLAUDE.md makes it a rule: query the graph before Read or Grep. Structure is cheap; raw file text is not.
• Compressed subagents. Hand exploration to a subagent: it reads forty files and hands back a ten-line file:line table. The main thread pays for the conclusion, not the search. That's roughly 60% fewer tokens than scanning inline, and the main context window lasts far longer across a session. My setup uses caveman-compressed cavecrew agents for exactly this.
• Deferred tool schemas. I have hundreds of MCP tools installed. Their JSON schemas don't sit in the prompt. They load on demand when a search pulls the one I need. You stop paying, every single turn, for the definition of a tool you call once a week.
• Compressed memory. CLAUDE.md and the memory files are written in caveman shorthand (caveman-compress): same facts, a fraction of the tokens. Memory is input you resend every turn, so the saving recurs forever.
• Terse working channel. caveman full and ponytail ultra (the 5a/5b levers above) run on by default, so the model's own output stays compressed too.

The principle is identical to the API stack: never spend a token carrying information the task didn't need. The API levers shrink the request; the harness levers shrink the context the agent assembles to make that request. Orthogonal axes, so they stack again. On a long agentic session the harness side is often the bigger win, because context is the thing that grows without anyone deciding it should.

A blunt heuristic from my own config: if the agent is about to read a whole file to learn one fact, that's a token leak. Graph-query it, or send a subagent to fetch the one line. The cheapest token is the file you never pasted.

The build order

Don't install everything on day one. Pull levers in order of payoff-per-effort:

• Turn on prompt caching, audit the cache-read counter. Biggest win, lowest effort.
• Route easy calls to a cheap model. One if.
• Add Ponytail and Caveman. Two commands, output cost drops immediately, code quality improves.
• Add Headroom and tighten retrieval when a bill (not a hunch) says the first three weren't enough.
• If you live in an agent harness, turn on the harness stack: graph-over-grep, compressed subagents, deferred tool schemas, compressed memory. A different axis that stacks on top of the four levers above.

That ordering is itself the ponytail rule: pull the laziest lever that moves the number first, and stop the moment the bill stops hurting. You rarely need all five. But when you do, they're sitting right here, and they stack.

Questions, corrections, or your own numbers? ai@jakubjirak.com.