Prompt Caching & Batching

I What Is Prompt Caching?

Part 2 introduced prompt caching as one of five optimization levers. Time to open the hood.

Here's the problem: every API request includes a system prompt. Your product's personality, instructions, few-shot examples, retrieval context — all of it gets sent on every single call. And every single time, the model processes it from scratch. That's like reciting the company handbook before every conversation.

Prompt caching stores the model's processed representation of your static contextso it doesn't have to recompute it. The technical term is "KV-cache" — the key-value pairs the model generates internally when it reads your input. On the first request, the model processes your system prompt and saves the result. On subsequent requests with the same prefix, it skips straight to the new part.

The payoff is immediate and measurable: 40-90% savings on input token costs and dramatically reduced Time to First Token (TTFT). For a product sending thousands of requests with a 2,000-token system prompt, this is the single highest-ROI optimization you can make.

II How It Actually Works

The mechanics are straightforward, which is part of what makes this so powerful — there's almost no reason not to use it if your prompts have a stable prefix.

Step 1: First request.You send your full prompt — system instructions, few-shot examples, retrieval context, then the user's message. The model processes everything, generating an internal representation (the KV-cache) as it reads. The provider stores this representation, keyed to your exact prefix.

Step 2: Subsequent requests. You send the same prefix plus a new user message. The provider recognizes the prefix, loads the cached KV-cache, and the model only has to process the new tokens. The static portion is essentially free.

Step 3: Billing.Cached input tokens are billed at a steep discount — typically 75-90% off. So a 2,000-token system prompt that normally costs $3/million tokens might cost $0.30-$0.75/million when cached. The non-cached portion (the user's actual message) is billed at the normal rate.

There are constraints. The cache has a TTL (time to live)— typically 5-10 minutes of inactivity before it's evicted. The prefix must match exactly, byte for byte. Change a single character in your system prompt and you get a cache miss. And there's usually a minimum token threshold (e.g., 1,024 tokens) before caching kicks in.

III When to Cache (and When Not To)

Prompt caching isn't a universal win. It's a powerful one, but like every optimization, it has a sweet spot.

Cache when:

• Your system prompt is large and stable. A 2,000-token instruction set that changes once a month? Cache it. The bigger the static portion relative to the dynamic portion, the bigger the savings.
• You use few-shot examples.Those five carefully crafted input-output pairs you send with every request are perfect cache candidates. They're substantial, they're identical across calls, and they're expensive to reprocess.
• RAG context is shared across a session. If a user is asking multiple questions about the same retrieved document, the document stays in cache between turns. The second question about the same context is dramatically cheaper than the first.
• You have high request volume. Caching has the most impact when many requests share the same prefix. A product handling 10,000 requests/hour with the same system prompt sees enormous savings. A product handling 10 requests/hour with unique prompts sees almost none.

Don't cache when:

• Every request is unique.If the entire prompt changes with each call — no shared system prompt, no repeated examples — there's nothing to cache.
• Context changes frequently.If your system prompt updates every few minutes (real-time data feeds, frequently rotated instructions), the cache will miss more than it hits and you'll pay the write cost without the read benefit.
• Prompts are short.A 200-token system prompt saves so little when cached that it's not worth thinking about. Focus your optimization energy elsewhere.

IV Batching: The Other Lever

Caching optimizes the per-request cost. Batching optimizes the throughput cost — and it works on a completely different principle.

Batch APIs let you send hundreds or thousands of requests in a single API call and get results back asynchronously — usually within 24 hours. In exchange for giving up real-time responses, you get a massive discount: typically 50% off the standard per-token price.

Why the discount? Because the provider can schedule your batch during off-peak hours, fill GPU capacity gaps, and optimize throughput without latency constraints. You're trading urgency for efficiency — and the provider passes the savings through.

Best for:

• Eval harnesses.Running 500 test cases against your prompt? That's a perfect batch job. You don't need results in milliseconds — you need them by morning.
• Bulk processing.Classifying 10,000 support tickets, summarizing a quarter's worth of call transcripts, generating product descriptions for a catalog. High volume, low urgency.
• Offline analysis.Weekly reports, content moderation backlogs, data enrichment pipelines. Anything where the consumer of the output is a system or a human who'll look at it later.
• Dataset generation.Creating synthetic training data, generating test fixtures, building evaluation sets. Volume matters, speed doesn't.

Bad for:

• Anything user-facing and real-time.If a human is waiting for the response, batch is not the answer. A 24-hour SLA on a chatbot response is called "email."
• Interactive workflows.If the next step depends on the model's output, you can't batch because you need the result to continue.

See which approach fits your use case:

V The Cost Math

Theory is nice. Let's do the arithmetic.

Scenario:You're running a customer support product. 50,000 requests per day. Each request includes a 2,000-token system prompt (instructions + few-shot examples) and a 300-token user message. Average output: 200 tokens. You're using a Sonnet-class model ($3/M input, $15/M output).

Without caching:

• Input cost: 50,000 requests x 2,300 tokens x $3/M = $345/day
• Output cost: 50,000 x 200 tokens x $15/M = $150/day
• Total: $495/day ($14,850/month)

With prompt caching (2,000 tokens cached at 90% discount):

• Cached input: 50,000 x 2,000 tokens x $0.30/M = $30/day
• Non-cached input: 50,000 x 300 tokens x $3/M = $45/day
• Output cost: unchanged at $150/day
• Total: $225/day ($6,750/month)

That's a ~78% reduction on the input side (about 55% off the total bill) and $8,100/month saved — from a configuration change that takes an afternoon to implement.

Now layer on batching for your offline workloads. Say 30% of your traffic is non-urgent (ticket classification, weekly summaries, eval runs). Move those to batch at 50% off:

• Real-time (70%): $225/day x 0.7 = $157.50/day
• Batch (30%): $225/day x 0.3 x 0.5 = $33.75/day
• Combined: $191.25/day ($5,738/month)

From $495/day to $191/day. A 61% total reduction.And you haven't changed a single word of your prompts or switched models.

Play with your own numbers:

VI Implementation Patterns

Understanding the theory is step one. Shipping it well requires attention to a few practical patterns.

1. Structure prompts for cache-friendliness. The golden rule: static content first, dynamic content last. Your prompt should be layered like this:

1. System instructions — personality, rules, output format. Changes rarely.
2. Few-shot examples — input-output pairs. Changes occasionally.
3. Shared context — retrieved documents, session context. Changes per session.
4. User message — the actual request. Changes every call.

Everything above the user message is your cacheable prefix. The longer and more stable that prefix is, the more you save.

2. Cache warming.The first request always pays full price (no cache exists yet). For high-volume products, send a "warming" request at startup or after deploys to prime the cache before real traffic hits. This is especially important if your system prompt is large — you don't want your first 100 users paying the full processing cost when a single warm-up call would have covered everyone.

3. TTL management. Cache entries expire after a period of inactivity (typically 5-10 minutes). If your traffic is bursty — heavy during business hours, dead overnight — the cache will go cold every night. Consider a lightweight keep-alive request during low-traffic periods if the cache warming cost is significant. But do the math first: if your nightly traffic is 10 requests, warming the cache costs more than just paying the full price.

4. Monitor cache hit rates.You can't optimize what you don't measure. Track:

• Cache hit rate — what percentage of requests use the cache? Below 80% means something is wrong with your prefix stability.
• Cost per request (cached vs. uncached) — to quantify actual savings.
• TTFT reduction— cache hits should reduce time to first token by 30-80%. If they don't, verify the cache is working.

5. Version your prompts carefully.A single character change in your system prompt invalidates the cache. Use a deployment process that changes prompts deliberately, not accidentally. A stray newline in a config file shouldn't double your API costs overnight.