Cost Optimization in Production AI

Q: How would you cut our AI bill in half?

Walk through the breakdown: input vs output, model mix, cache hit rate, context length, top_k. Find the biggest single line item, attack that first. Show that any change passes the eval suite before shipping.

Q: What is the cost per request for this feature?

You should know. Tag every request with feature. Average and P95 cost per request, per feature. This is the most useful single metric.

Q: How do you decide which model to use?

Eval on each model. Pick the cheapest that passes. Re-evaluate every 6 months as model prices drop and new models come out.

Q: How do you avoid surprise bills?

Daily cost summaries to the team. Per-feature budget alerts. Hard caps via provider settings. Aggressive max_tokens.

Q: What is the ROI of optimization vs new feature work?

$5k/month in cuts = $60k/year. One engineer-week of optimization usually pays back forever. Spend it.

The first month your AI bill stays at “weirdly expensive but okay”. The third month you get an angry Slack message from finance. The CTO asks why your AI feature costs more than the database. You did not pick a more expensive model. Usage just grew.

This article is the playbook. Where the money actually goes, the five highest-leverage cuts, and the real numbers you can expect.

Where the money actually goes

Pull last month’s invoice from Anthropic or OpenAI. Then break it down:

flowchart LR
    BILL[Total LLM bill]
    BILL --> INPUT[Input tokens<br/>usually 70-90% of cost]
    BILL --> OUTPUT[Output tokens<br/>usually 10-25% of cost]
    BILL --> EMB[Embeddings<br/>usually 1-5% of cost]

Most teams find input tokens dominate. The fix is to send fewer of them, or get them cached.

Now compare to your overall AI infra bill which also includes:

Vector DB (Pinecone, Qdrant Cloud, pgvector)
Reranker API (Cohere, Voyage)
Embedding API (could be same provider or different)
Compute (if you self-host a model)
Observability (LangSmith, Helicone, etc.)

Get the full picture before optimizing. Make a single dashboard showing all five.

Lever 1: Prompt caching

Free, immediate, biggest single lever. We covered this in detail in article 10, so the short version:

# Anthropic
system=[{"type": "text", "text": LONG_PROMPT, "cache_control": {"type": "ephemeral"}}]

# OpenAI: automatic when you reuse the same prefix

After 5 minutes of usage, cached input tokens cost about 10% of normal.

Real impact: a stable system prompt + RAG context that is reused often → 50-80% reduction in input cost.

Turn this on today. It is not even close to the next lever in effort/payoff.

Lever 2: Model routing

Not every query needs the smartest model. A classifier query can run on the cheap model. A complex synthesis query needs the smart one.

Why does model routing not hurt quality?

The naive assumption: “GPT-4o is always better than GPT-4o-mini, so using the cheaper model must be a quality compromise.” This is wrong for most production tasks.

LLMs are designed to handle a huge range of difficulty: from “summarize this paragraph” to “write a research paper”. Most of your traffic is on the easy end. A smaller model is easily capable of “is this user query about billing or about technical support?” Using GPT-4o for that question is like using a Ferrari to deliver pizza.

The mistake teams make is routing without eval. They assume hard queries need the big model and easy queries can use the small one, but they never measure. Sometimes the small model is fine for all of it. Sometimes you need the big model more often than you thought. You only know by running an eval set against both models, then routing only the cases where the small model fails.

Done right: 30-60% cost reduction with no measurable quality drop. Done wrong: silent quality degradation that you only catch when users complain.

flowchart TB
    Q[Incoming query]
    Q --> CLASS[Tiny classifier<br/>or rule-based check]
    CLASS -->|Simple| CHEAP[GPT-4o-mini or Haiku<br/>$0.15 per 1M input]
    CLASS -->|Complex| SMART[GPT-4o or Sonnet<br/>$3 per 1M input]
    CHEAP --> ANS[Response]
    SMART --> ANS

Three ways to route:

Rule-based

Easiest. Classify by simple heuristics: query length, presence of certain words, conversation depth.

def pick_model(query):
    if len(query.split()) < 6:
        return "claude-haiku-4-5"  # tiny queries
    if any(word in query.lower() for word in ["analyze", "explain", "why", "how"]):
        return "claude-sonnet-4-5"  # likely needs reasoning
    return "claude-haiku-4-5"

Classifier-based

Use a fast cheap LLM to classify difficulty, then route.

classification = haiku_call(
    f"Classify the difficulty of this query: easy/medium/hard\n{query}"
)
model = {"easy": "claude-haiku-4-5", "medium": "claude-haiku-4-5", "hard": "claude-sonnet-4-5"}[classification]

The classification call itself is cheap (under a cent). The savings on the main call are 10-20x.

Tiered fallback

Try the cheap model. If the output looks bad (parsing fails, confidence is low, output is too short), retry with the smart model.

Real impact: 30-60% cost reduction when 60-80% of queries can run on the cheap model.

Validation step: build an eval set, score both models on it, confirm the cheap model passes for the queries you intend to route to it. Without this you will trade cost for quality silently.

Lever 3: Output token reduction

Output tokens cost 4-5x more than input tokens. Limiting them helps a lot.

Tactics:

Set max_tokens aggressively. If the right answer is 100 tokens, set max_tokens=200, not 4096. The model often pads to fill the budget.

Ask for shorter outputs. System prompt: “Answer in 2 sentences max.” Has measurable effect.

Use structured outputs. Forces JSON, which is tighter than prose. No flowery preambles.

Skip “explain your reasoning” unless you need it. Chain-of-thought triples output tokens. Use it when accuracy demands it, not by default.

flowchart LR
    BEFORE["Answer:<br/>'I'd be happy to help! After<br/>analyzing your question, I think<br/>the answer is 42 because...'<br/>(80 tokens)"]
    AFTER["Answer:<br/>'42.'<br/>(5 tokens)"]
    BEFORE --> COST1[Cost: 16x more]
    AFTER --> COST2[Cost: 1x]

Real impact: 20-50% output cost reduction without quality loss for most extraction and Q&A tasks.

Lever 4: Semantic caching

Covered in article 10. The short version: cache by query semantic similarity, return cached answers for paraphrased repeats.

Real impact: 10-40% additional cost reduction depending on query repetition.

Higher for customer support FAQ bots. Lower for personalized advisory tools.

Lever 5: Context-length discipline

Long contexts are expensive even with caching. Each call still pays for the cache miss the first time and for any uncached new tokens (the user query, recent retrieval).

Tactics:

Cap the RAG context. Common bug: setting top_k=20 because “more is better” then sending all 20 chunks. Set top_k=5. Cap total context tokens.

Trim the system prompt. Many system prompts grow over months: “also handle case X, also case Y”. Audit and prune.

Drop conversation history. Keep only the last N turns in multi-turn chats, or summarize older history.

flowchart LR
    BLOAT["3000-token system prompt<br/>+ 6000-token retrieved context<br/>+ 12 turns of history<br/>= ~15k tokens per call"]
    LEAN["1500-token system prompt<br/>+ 2500-token retrieved context<br/>+ summarized older history<br/>= ~5k tokens per call"]
    BLOAT --> C1[$0.045 per call]
    LEAN --> C2[$0.015 per call]

Real impact: 30-60% input cost reduction.

Beyond LLM costs

The LLM bill is usually the biggest line, but not the only one.

Vector DB: most billing is per-vector-stored or per-search. Quantize embeddings (float32 → int8) for 4x storage savings. Skip the dedicated vector DB if you have <10M vectors; pgvector is fine and almost free.

Reranker API: costs add up. For low-stakes queries, skip rerank entirely. For high-stakes, use it. Rule of thumb: rerank cost should be under 20% of LLM cost.

Embedding regen: if you re-embed your corpus on every prompt change, that is wasted money. Re-embed only when the embedding model itself changes or content changes. Tag embeddings with their model version.

Observability tools: usage-based pricing on LangSmith/Helicone scales with you. At high volume, switch to sampling (log 10% of traces) or self-hosted (Phoenix).

A real cost cut walk-through

A team I worked with was at $14k/month. Here is the order of cuts:

Turn on prompt caching (1 day): $14k → $7k
Reduce top_k from 20 to 5 in RAG (1 day): $7k → $5k
Set max_tokens=400 instead of 2048 (15 minutes): $5k → $3.8k
Route simple queries to Haiku (1 week of dev + eval): $3.8k → $2.4k
Semantic cache at 0.97 threshold (1 week of dev): $2.4k → $1.6k

End: $1.6k/month. Same product, no quality regression (measured by their eval pipeline).

The discipline that mattered most: eval-driven cost cuts. After each change they ran the eval suite to confirm no quality regression. Without that, optimizations bleed quality silently and you cannot tell the difference between “cheap and still good” and “cheap and broken”.

When NOT to optimize cost

Premature optimization in AI is real. Skip these levers when:

Volume is tiny. $200/month is not a problem. Spend your time on product.
You have not measured. Optimizing the wrong line item is wasted effort. Pull the breakdown first.
Quality is not yet good enough. You will trade away quality you do not have to spare. Get to acceptable first, then optimize.

The order: build → measure quality → ship → measure costs → optimize → re-measure quality.

Common beginner mistakes

Defaulting to the biggest model “to be safe”. GPT-4 for a classifier is wasteful. Smaller models pass eval for most narrow tasks.

Caching off because “we want freshness”. Most queries are not time-sensitive. Set a 5-minute TTL and stop worrying.

Counting prompt tokens by hand. Use the provider’s tokenizer or tiktoken. Off-by-1000 mistakes are common.

Not tracking cost per request type. Aggregate billing hides the truth. Tag every request with feature/intent; alert on cost-per-request anomalies.

Optimizing in isolation. Cutting cost without running evals is how you ship a cheap broken product. Always eval after.

Questions you will face in production

“How would you cut our AI bill in half?” Walk through the breakdown: input vs output, model mix, cache hit rate, context length, top_k. Find the biggest single line item, attack that first. Show that any change passes the eval suite before shipping.

“What is the cost per request for this feature?” You should know. Tag every request with feature. Average and P95 cost per request, per feature. This is the most useful single metric.

“How do you decide which model to use?” Eval on each model. Pick the cheapest that passes. Re-evaluate every 6 months as model prices drop and new models come out.

“How do you avoid surprise bills?” Daily cost summaries to the team. Per-feature budget alerts. Hard caps via provider settings. Aggressive max_tokens.

“What is the ROI of optimization vs new feature work?” $5k/month in cuts = $60k/year. One engineer-week of optimization usually pays back forever. Spend it.

What to remember

Input tokens dominate the bill. Cut them first.
Prompt caching is free and immediate. Always on.
Model routing typically cuts 30-60% with no quality loss when done with eval.
Output tokens are 4-5x more expensive than input; aggressively cap max_tokens.
Context discipline (top_k, system prompt size, history) is a 30%+ lever.
Always eval after a cost cut. Optimization without eval is quality regression.

What to study next

You finished Track 04. One article left: the capstone.

Article 13: Build It: A Small RAG Service That Ships wires every concept from the curriculum (chunking, hybrid retrieval, structured generation, evals, caching, observability, cost tracking) into one working service you can deploy on a $5 VPS. It is where the twelve isolated pieces become one system.