What Is Prompt Caching and When Does It Actually Save Money?

What Is Prompt Caching?

Prompt caching is a feature where an LLM provider stores the attention state of the unchanging prefix of your prompt — system instructions, retrieved documents, examples — so subsequent calls reuse it instead of paying to recompute it. On Claude 4.7 you pay 10% of the regular input price for cache hits, on GPT-5 you pay 50%, and on Gemini 2.5 you pay 25%. For production apps with stable system prompts, this routinely cuts the LLM bill by 60-90% with zero quality change.

That's the 50-word answer. The rest of this post explains exactly how it works, when it pays back, and the code you need to ship it today.

How Does Prompt Caching Work?

Every LLM call goes through three phases on the provider's GPUs:

1. Tokenization — converting your text into integer tokens
2. Prefill — running attention over every token in the prompt to build the KV cache
3. Decode — generating output tokens one at a time

Prefill is the expensive part. For a 10,000-token prompt, prefill consumes ~95% of the latency and ~98% of the input-side compute cost. The decode phase is comparatively cheap.

Prompt caching exploits a simple fact: if today's prompt starts with the same 8,000 tokens as yesterday's, the prefill work is identical. So providers added the ability to save the KV cache to fast storage (typically host RAM or a distributed cache layer) and replay it on the next matching call, paying only for the new suffix tokens at full price.

The engineering details vary, but the user-facing API contract is consistent across providers: mark a prefix as cacheable, and pay a discounted price when subsequent calls hit it.

The Anthropic prompt caching docs, the OpenAI prompt caching guide, and the Google Gemini context caching docs describe each provider's exact mechanics.

When Does Prompt Caching Save Money?

The break-even math is simple. Prompt caching saves money when:

(cache_write_price - cache_read_price) / (base_input_price - cache_read_price) < expected_hits_per_write

In plain English: you need enough cache reads to amortize the small extra cost of the cache write. On each provider:

• Claude 4.7: Write costs 1.25x base, read costs 0.1x base. Break-even = 1 read. Two reads is pure savings, ten reads is dramatic savings.
• GPT-5: Write is implicit (no extra cost), read costs 0.5x base. Break-even = 1 read. Every cache hit saves money.
• Gemini 2.5: Write is implicit for auto-caching, explicit caching has a small storage fee. Read costs 0.25x base.

The workloads where this math overwhelmingly works:

• RAG with stable system prompts — same instructions and same document chunks reused thousands of times per day
• Multi-turn conversational agents — every turn re-sends the entire conversation history, which becomes the next turn's cached prefix
• Customer support copilots — long product manual or knowledge base loaded once, queried thousands of times
• Codebase-aware coding assistants — entire repository (or large chunks) sent as context across many edits
• Few-shot prompting at scale — same 10-20 example prompts reused for every classification or extraction call

When Is Prompt Caching NOT Worth It?

Three scenarios where caching costs you money or has no effect:

1. One-shot queries with no shared structure — if every call has a totally unique prompt, there's nothing to cache
2. Very short prompts — most providers have a minimum token threshold (Claude requires 1,024 tokens for caching to engage; GPT-5 and Gemini have similar floors). Below that threshold, the cache write overhead doesn't pay back.
3. High-variability prefixes — if your "system prompt" changes per user (e.g. you inject the user's full profile at the top), the cache invalidates on every call and you pay write price with no read benefit. Restructure to put the variable bits at the end, not the start, of the prompt.

Rule of thumb: if your shared prefix is under 1,024 tokens or you reuse it fewer than 2 times per cache TTL, skip caching.

Which Providers Offer Prompt Caching in 2026?

Always check live pricing at the source: Anthropic pricing, OpenAI pricing, Google AI pricing.

The shapes are different but the punchline is the same: every major provider now offers a 50-90% discount on the reused part of your prompt.

How to Implement Prompt Caching

Here's a concrete TypeScript example using the Anthropic SDK with Claude 4.7. The same pattern translates directly to GPT-5 (which auto-detects) and Gemini (which uses similar explicit cache markers).

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

// A long, stable system prompt + RAG context. This is the part we want cached.
const SYSTEM_INSTRUCTIONS = `You are a senior backend engineer helping
debug production issues at Acme Corp. You have access to the codebase,
runbooks, and incident history. Always respond with concrete next steps.
[... 4,000 more tokens of instructions ...]`;

const RAG_CONTEXT = `<runbooks>
[... 6,000 tokens of retrieved runbooks for the current incident ...]
</runbooks>`;

async function answerIncidentQuestion(userQuestion: string) {
  return await anthropic.messages.create({
    model: 'claude-sonnet-4-7-20260506',
    max_tokens: 1024,
    system: [
      {
        type: 'text',
        text: SYSTEM_INSTRUCTIONS,
        cache_control: { type: 'ephemeral' },
      },
      {
        type: 'text',
        text: RAG_CONTEXT,
        cache_control: { type: 'ephemeral' },
      },
    ],
    messages: [
      {
        role: 'user',
        content: userQuestion,
      },
    ],
  });
}

Two things worth noting:

1. The cache_control breakpoints split the prompt into cacheable segments. Anthropic supports up to 4 breakpoints, so you can cache different prefixes that change at different rates (e.g. system prompt that never changes, RAG context that rotates daily, conversation history that grows per turn).
2. The user question is *not* cached — it's the variable part. Put variable content at the end of the prompt, not the start.

For the 1-hour cache variant on Claude 4.7, change { type: 'ephemeral' } to { type: 'ephemeral', ttl: '1h' }.

GPT-5 requires no code changes — caching is automatic when you reuse prompt prefixes. The OpenAI prompt caching docs explain how to monitor cache hit rate via the API response.

For Gemini, the explicit context caching API lets you create a cache once and reuse the cache name across calls, which is more deliberate but gives you fine-grained TTL control.

Worked Example: 1,000 Calls Per Day With 8K Reused Context

Let's calculate the actual savings on a typical production workload:

• 1,000 calls/day
• 8,000 tokens of reused system prompt + RAG context per call
• 200 tokens of unique user content per call
• 500 tokens of output per call
• Cache writes assumed once per 5-minute window (peak case, bottom-line will be higher)

Without caching

• Input: 8,200 tokens × 1,000 calls = 8.2M tokens/day
• Output: 500 tokens × 1,000 calls = 0.5M tokens/day

With caching (assume 95% cache hit rate)

Use our AI cost calculator to plug in your own numbers and see the break-even point on your workload.

Decision Matrix: Should You Cache This Prompt?

Common Pitfalls

A few mistakes we see in production code reviews:

1. Putting timestamps or request IDs in the system prompt. This invalidates the cache on every call. Move them to a tool-call payload or the user message instead.
2. Tool schemas changing between calls. If you reorder or modify tool definitions, the cache breaks. Keep tools stable, or cache only the system prompt without tools.
3. Not monitoring cache hit rate. Both Anthropic and OpenAI return cache statistics in the response object. Log them. A cache hit rate below 70% in production means something upstream is breaking your prefix.
4. Caching tiny prompts. If your prefix is 500 tokens, the write overhead may exceed the read savings. Use caching for prefixes ≥ 1,024 tokens.
5. Forgetting TTL. A cache that expires before the next call gives you all the cost of the write and none of the read benefit. For sparse traffic, prefer Claude's 1-hour cache or Gemini's explicit context caching with a long TTL.

If your agent is also leaking context between calls (a related but distinct failure mode), our guide on why your AI agent loses context covers that side of the same problem.

How Prompt Caching Plays With Model Choice

The frontier-model comparison in our Claude 4.7 vs GPT-5 vs Gemini 2.5 Deep Think head-to-head shows that the three models trade off speed, quality, and price differently. Caching changes the price axis dramatically:

• Claude 4.7 Sonnet with caching lands at roughly $1.50/M effective input price for typical production workloads — cheaper than GPT-5 without caching
• Gemini 2.5 Pro with caching lands at roughly $0.50/M effective input price — by far the cheapest frontier option
• GPT-5 with caching lands at roughly $2.75/M effective — competitive but no longer the obvious pick on cost

If you've been defaulting to GPT-5 because of price perception, re-run the math with caching turned on. Many teams find the picture flips entirely.

Bottom Line

Prompt caching is the most effective single optimization available for production LLM apps in 2026. It is supported by every frontier provider, requires minimal code changes, and routinely cuts bills by 60-90% on workloads with stable prefixes.

The decision is binary: if your prompt has a reused prefix of at least 1,024 tokens that's seen at least 2 times within the cache TTL, turn caching on. Otherwise, don't bother.

For more frontier-model decisions, see our Claude 4.7 vs GPT-5 vs Gemini 2.5 head-to-head and the broader best AI tools for developers in 2026 roundup.

For the bigger picture on production LLM cost optimization, see our pillar guide on [the best AI tools for developers in 2026](/blog/best-ai-tools-for-developers-2026).