Caching for LLM Apps

Most production LLM features are 10x more expensive than they need to be. The fix is not switching to a smaller model. The fix is caching, and most teams have not turned it on.

This article walks through three layers of LLM caching, when each one wins, and the real cost savings you can expect.

The economics that make caching worth it

A typical AI feature looks like:

• System prompt: 1,500 tokens (instructions, examples)
• Retrieved context: 4,000 tokens (RAG chunks)
• User query: 50 tokens
• Output: 300 tokens

That is 5,850 tokens per call. At Claude Sonnet pricing ($3 input / $15 output per million tokens):

• Input cost: $0.0167
• Output cost: $0.0045
• Total: $0.021 per call

Looks cheap. But at 10,000 calls per day:

• Daily cost: $210
• Monthly cost: $6,300

Now multiply by 10 customers, 50 customers, more queries per customer. The bill scales fast.

Caching changes this from “we need to switch models” to “we already had the answer”.

The three caches

flowchart TB
    REQ[User query]
    REQ --> SC{Semantic cache?}
    SC -->|Hit| RET1[Return cached answer]
    SC -->|Miss| EC{Embedding cache?}
    EC -->|Hit| RAG[Use cached embedding for retrieval]
    EC -->|Miss| GEN[Generate embedding]
    GEN --> RAG
    RAG --> PC{Prompt cache?}
    PC -->|Hit| CHEAP[90% cheaper input]
    PC -->|Miss| FULL[Full price input]
    CHEAP --> LLM[LLM call]
    FULL --> LLM
    LLM --> RES[Response]

Three layers, three different things to cache:

1. Semantic cache: “I have already answered this exact question for someone.”
2. Embedding cache: “I have already computed this query embedding.”
3. Prompt cache: “The provider has my long system prompt in fast memory.”

Each cuts cost in a different way. Use them together.

Layer 1: Prompt caching (provider-side)

Both OpenAI and Anthropic let you mark parts of your prompt as cacheable. When you reuse them in the next call, you get 50% to 90% off the input cost for those tokens.

This is the easiest win in production. One line of config.

Anthropic

response = client.messages.create(
    model="claude-sonnet-4-5",
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,  # 1500 tokens
            "cache_control": {"type": "ephemeral"},  # cache this
        }
    ],
    messages=[{"role": "user", "content": query}],
)

After the first call, the next 5 minutes of calls to this same prompt cost about 10% of what they used to (for the cached portion).

OpenAI

OpenAI caches automatically when the same prefix is sent. No config required. You see the cache hit in the usage object:

response = client.chat.completions.create(...)
print(response.usage.prompt_tokens_details.cached_tokens)
# 1500  # cached

When prompt caching wins

• You have a long, stable system prompt (1k+ tokens) reused across calls
• You have many short user queries hitting the same prompt
• Your RAG retrieval is cached (same chunks for similar queries)

When it does not help

• Every call has a unique long prompt
• System prompt changes per user
• Tiny prompts (overhead is not worth it)

Real numbers

A team with a 2,000-token system prompt + 1,000-token RAG context, 50k calls per day:

• Before: 3,000 tokens × $3/M × 50k = $450/day input cost
• After: 300 tokens × $3/M × 50k (cache hit) = $45/day

That is $400/day saved on one config change. Anthropic gives this away. Just turn it on.

Layer 2: Semantic caching

Semantic caching catches the case where two different users ask the same thing in different words.

flowchart LR
    Q1["'How do I reset my password?'"] --> EMB1[Embed]
    Q2["'I forgot my login'"] --> EMB2[Embed]
    EMB1 -.cosine 0.91.-> EMB2
    EMB2 --> CACHE{Check cache:<br/>any embedding<br/>within 0.97?}
    CACHE -->|Yes| HIT[Return previous answer]
    CACHE -->|No| CALL[Full LLM call + cache the result]

The implementation:

import numpy as np

class SemanticCache:
    def __init__(self, threshold=0.97, ttl_seconds=3600):
        self.entries = []  # list of (embedding, query, answer, expires_at)
        self.threshold = threshold
        self.ttl = ttl_seconds

    def get(self, query_embedding):
        now = time.time()
        # Clear expired
        self.entries = [e for e in self.entries if e[3] > now]
        # Find similar
        for emb, query, answer, _ in self.entries:
            sim = cosine_similarity(query_embedding, emb)
            if sim > self.threshold:
                return answer
        return None

    def set(self, query_embedding, query, answer):
        self.entries.append(
            (query_embedding, query, answer, time.time() + self.ttl)
        )

In production you store this in Redis or a vector DB, not memory.

Choosing the similarity threshold

• 0.99: almost identical queries. Few false positives, fewer hits.
• 0.97: very similar. Good default for FAQ-style apps.

• 0.95: similar topics. Risk of returning wrong answer.
• 0.90: too broad. Hits start to lie.

Tune this with real data. Pull queries from logs, sample pairs with various scores, check whether their correct answers were the same.

When semantic cache wins

• High-traffic features with repetitive user intent (customer support FAQs)
• Public-facing AI tools (lots of similar questions)
• Where freshness is not critical (TTL of 1 hour or more)

When it loses

• Personalized output (“recommend a movie for ME”)
• Time-sensitive answers (“what is today’s weather?”)
• Low traffic (cache hit rate stays at 0% if no two queries are similar)

Real numbers

A customer support bot, 10k queries per day, 20% semantic similarity hit rate:

• 2,000 calls per day skip the full LLM pipeline
• At $0.021 per call → $42 per day saved
• Plus latency drops to about 50ms for cached responses

For a higher-traffic FAQ bot, 40% hit rates are achievable.

Layer 3: Embedding caching

Generating embeddings is much cheaper than an LLM call but adds up at scale.

The pattern: cache the embedding of any query you have seen exactly before.

import hashlib

class EmbeddingCache:
    def __init__(self, redis_client):
        self.redis = redis_client

    def get_or_compute(self, text):
        key = "emb:" + hashlib.sha256(text.encode()).hexdigest()
        cached = self.redis.get(key)
        if cached:
            return np.frombuffer(cached, dtype=np.float32)
        emb = compute_embedding(text)
        self.redis.set(key, emb.tobytes(), ex=86400)  # 24h TTL
        return emb

When embedding cache wins

• Same queries repeat (a popular FAQ, common search terms)
• You embed the same documents for multiple downstream uses
• High traffic, low diversity

Real numbers

At OpenAI text-embedding-3-large ($0.13 / 1M tokens), a 100-token query costs $0.000013. Trivial per call. But at 1M queries per day with 50% cache hit rate, you save 500k embedding calls, which is also free latency and one less point of provider dependency.

The bigger win: document embeddings in a RAG pipeline. Re-embedding the corpus is much more expensive. Cache aggressively by content hash; only re-embed when content changes.

Cache invalidation, the second-hardest problem in CS

What invalidates a cached LLM answer?

Time-based (TTL): simplest. After N minutes, drop it. Use this when freshness matters at the scale of minutes (news, prices).

Event-based: when a source document changes, invalidate any answer derived from it. Hard to implement cleanly. Practical pattern: tag cached entries with the document IDs they used, and bulk-invalidate on document update.

Version-based: tag entries with prompt version, model version, and RAG corpus version. Bump the version when any of these change.

cache_key = f"{prompt_version}:{model}:{corpus_version}:{query_hash}"

When you deploy a new prompt, you do not need to manually invalidate; the new key never matches old entries.

Putting it together: a real cost-cut recipe

A team I worked with had a customer support AI feature spending $8k/month. They turned on, in this order:

1. Prompt caching (Anthropic). System prompt + RAG common chunks: down to $4k/mo (50% off).
2. Semantic caching (Redis, 0.97 threshold, 1h TTL). Hit rate 22%: down to $3.2k/mo.
3. Smaller model for simple queries (Haiku for classification, Sonnet for synthesis): down to $1.8k/mo.

End state: same product, 77% cost reduction, faster average latency (semantic hits are 50ms instead of 4s).

None of this required smarter prompting or a new model.

Common beginner mistakes

No caching at all. The default. Turn on prompt caching today. Free win.

Threshold too low. 0.90 semantic similarity often returns wrong answers. Stay at 0.95+ until you have data.

No TTL. Stale answers in customer support are a real PR problem. Always set a TTL appropriate to your domain.

Caching personalized output. If the answer depends on user ID, the cache key must include it. Forgetting this leaks data across users.

Caching across prompt versions. Always include prompt version in the cache key. Otherwise you serve old answers from old prompts forever.

Skipping cache hit metrics. Track hit rate per cache layer. Without it, you do not know if caching is even helping.

Questions you will face in production

“What is the cache hit rate?” Always know this number. Tag every request with cache_hit=true/false. Track per layer.

“How do you invalidate on a content change?” Tag cache entries with the source document IDs. On document update, bulk-invalidate entries with that tag.

“How do you handle user-specific data?” Include user ID (or tenant ID) in the cache key. Or skip caching entirely for personalized responses.

“Could the cache return stale or wrong answers?” Yes. Mitigations: short TTL, similarity threshold, version-aware keys, audit-trail logging. For high-stakes content (legal, medical), reduce caching aggressiveness.

“What is the right balance between cache layers?” Prompt caching is free, always on. Semantic caching at 0.97 threshold catches most repeats. Embedding cache for query embeddings is almost free. Layer all three.

What to remember

• Prompt caching is the free win. Turn it on today.
• Semantic caching catches paraphrased repeats. 0.97 threshold default.
• Embedding caching is small per-call savings but big at scale.
• Always include prompt version + model in the cache key.
• TTL based on how stale answers are acceptable in your domain.
• Track hit rate per layer. Without that, you are flying blind.

What to study next

Now that you know what to cache, you need to know what is actually happening in production. Move to article 11, Production AI Observability.

Further reading

• Anthropic — Prompt caching. Official docs: how cache_control works, TTL behavior, pricing impact.
• OpenAI — Prompt caching. OpenAI’s automatic caching, what prefixes get cached, how to verify cache hits.
• GPTCache — Semantic caching library. Open-source semantic cache implementation, useful to read even if you build your own.
• AWS — Bedrock prompt caching. Bedrock’s take on prompt caching for cross-provider context.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.