Prompt caching: the cheapest 90% off your bill (that you're probably getting wrong)

Of all the cost levers, prompt caching is the biggest one and the one most often broken. Done right, it bills your stable prefix at roughly 0.1×: a 90% discount on the bulk of your input, with zero quality change. Done wrong, which is the default, it silently does nothing and you never get an error telling you. Here's how to actually get it.

The win, and the one rule it all follows

Agent and RAG loops re-send a huge, identical prefix every turn: the system prompt, the tool definitions, the codebase, the retrieved docs. Caching charges that prefix at ~0.1× after the first write. On a loop where 90% of the input is stable prefix, you've cut input cost by ~80–90% for free.

It all follows from one rule: caching is a prefix match, and any byte change anywhere in the prefix invalidates everything after it. Render order is tools → system → messages. A breakpoint caches everything before it. Change one byte before the breakpoint and the whole cache is gone.

The silent killers

This is where the money leaks. Each of these quietly zeroes your cache. No error, just a full-price bill:

# every one of these invalidates the cache on every request:
system = f"Current date: {today()}\n{BIG_PROMPT}"   # timestamp in the prefix
system = f"User {user_id}:\n{BIG_PROMPT}"            # per-user id in the prefix
tools  = build_tools(user)                            # tool set varies per request
json.dumps(payload)                                   # unsorted keys → bytes differ
if premium: system += EXTRA                           # conditional prefix sections

The pattern: anything volatile sitting before the breakpoint. A timestamp, a UUID, a per-request ID, a set iterated in nondeterministic order, a tool list that changes. Each makes every request a unique prefix, so nothing ever caches.

The fixes

• Freeze the prefix. No timestamps, no UUIDs, no per-user IDs in the system prompt. Inject dynamic context after the breakpoint (in the messages, not the prefix).
• Serialize deterministically. Sort JSON keys. Order tools deterministically (by name). A set iterated twice is a cache miss.
• Don't change tools or model mid-session. Both invalidate the whole cache. Need "modes"? Pass the mode as message content, don't swap the tool set.
• Put the breakpoint at the end of the stable part, not the end of the whole prompt. If the breakpoint is after the varying question, every request writes a fresh entry and nothing is ever read.

Most "caching isn't helping" complaints are a single volatile byte in the prefix. The system prompt header with datetime.now() in it is the all-time champion cache killer.

Verify it, don't assume

Caching fails silently, so you must check. The response usage object reports it:

print(resp.usage.cache_read_input_tokens)   # served from cache (~0.1×), want this high
print(resp.usage.cache_creation_input_tokens) # written to cache (~1.25×)
print(resp.usage.input_tokens)                # full price, want this low

If cache_read_input_tokens is zero across repeated requests with the same prefix, you have a leak: diff the rendered prompt bytes between two requests and find the byte that changed. This single check, wired into your observability, catches the most common silent cost blowup there is.

The economics

Cache reads cost ~0.1×; cache writes cost ~1.25× (5-minute TTL) or ~2× (1-hour TTL). So you break even in 2–3 requests and win big after. For latency-sensitive apps, you can pre-warm the cache at startup so the first real request is fast. The 1-hour TTL keeps the cache alive across gaps in bursty traffic at the cost of a pricier write.

The architectural view

The deeper point: the cacheable prefix is a design surface, not an afterthought. An advanced agent treats "what is my stable prefix and how do I keep it byte-stable" as a first-class question, because it's the difference between linear and near-flat cost as a conversation grows. Your CLAUDE.md and tool definitions are part of that prefix, so keep them tight and stable. Caching is lever number one for a reason: it's the cheapest, biggest win, and it's sitting in your usage stats waiting for you to check whether it's on.