Production AI Observability

You shipped an AI feature. It works. Two weeks later something gets weird. Answers feel off. You have no idea where to look.

This is the moment teams discover they have no AI observability. Standard tools like Datadog and Sentry tell you that your service is up and your API is fast. They cannot tell you that your AI started hallucinating.

This article shows you what AI-specific observability actually looks like, and the three layers you need.

Why AI is different

Traditional services are deterministic. Same input, same output. Errors are exceptions. You log them and move on.

AI services are probabilistic. Same input might give different outputs. There is no “exception” when the model produces a bad-but-not-broken answer. Quality drifts silently.

flowchart LR
    subgraph trad["Traditional service"]
        T1[Request] --> T2[Logic]
        T2 --> T3{Error?}
        T3 -->|Yes| TE[Log exception, alert]
        T3 -->|No| TS[Return success]
    end

flowchart LR
    subgraph ai["AI service"]
        A1[Request] --> A2[LLM]
        A2 --> A3[Response]
        A3 --> A4{Bad?}
        A4 -.no way to tell.-> A4
        A4 --> A5[Return whatever]
    end

You need observability that catches the silent quality drop, not just the loud crash.

The three layers

flowchart TB
    L1[Layer 1: Traces<br/>What happened on this call?]
    L2[Layer 2: Evals in CI<br/>Did the change make things worse?]
    L3[Layer 3: Production sampling<br/>Is quality dropping over time?]
    L1 --> L2 --> L3

Each catches a different class of problem.

Layer 1: Traces

A trace is a structured log of one call through your AI pipeline. For a RAG system, a trace looks like:

{
  "request_id": "req_abc123",
  "user_id": "u_456",
  "timestamp": "2026-05-25T14:23:01Z",
  "query": "How do I reset my password?",
  "steps": [
    {
      "name": "query_embedding",
      "model": "text-embedding-3-large",
      "duration_ms": 87,
      "tokens": 8
    },
    {
      "name": "vector_search",
      "duration_ms": 124,
      "candidates_returned": 50,
      "top_score": 0.84
    },
    {
      "name": "rerank",
      "model": "cohere-rerank-3",
      "duration_ms": 211,
      "top_score": 0.91,
      "selected_chunks": ["doc_789:chunk_2", "doc_122:chunk_8"]
    },
    {
      "name": "llm_generation",
      "model": "claude-sonnet-4-5",
      "duration_ms": 2340,
      "input_tokens": 1820,
      "output_tokens": 180,
      "cache_hit": false,
      "cost_usd": 0.0084
    }
  ],
  "final_response": "To reset your password...",
  "total_duration_ms": 2762,
  "total_cost_usd": 0.0084
}

This is the gold standard. With one trace you can debug almost anything.

Why a structured trace and not just regular logs?

Regular logs are flat strings: “got query”, “ran search”, “called LLM”. When something breaks, you grep through thousands of lines trying to reconstruct which steps happened to which request and in what order.

A structured trace is a single JSON object per request that contains every step as a nested entry. One line of jq extracts everything: which query, which model, which chunks were retrieved, what each step cost, total latency. You can answer “what was the slowest step for queries that contained the word ‘invoice’ yesterday” in 30 seconds.

Two practical wins:

Debugging: a user complains the answer was wrong. Pull their trace, walk through it: did retrieval surface the right chunk? Did the re-ranker rank it correctly? Did the LLM ignore it? In one screen, you know.
Aggregation: average latency per step across millions of requests. Spot a regression in just the re-ranker without re-deploying anything.

Treat traces as a first-class data type, not “logs in JSON”.

What to capture per step

Latency: every step, every time
Tokens: input and output for LLM calls
Model used: critical if you route between models
Cache hit: per layer (prompt cache, semantic cache, etc.)
Cost: computed from tokens + model
Intermediate outputs: retrieved chunks, reranker scores, tool calls

What to capture per request

Request ID (link to logs)
User ID or tenant ID
Timestamp
Final response (or hash if it contains PII)
Quality signals from the user (thumbs up/down, regenerate, copy)

Tooling

You have two paths:

Hosted: LangSmith, Helicone, Phoenix, Langfuse. Drop in their SDK, get a UI for traces, eval runs, comparisons. Best for getting started.
DIY: structured JSON logs to your existing stack (Datadog, Honeycomb, S3, ClickHouse). More work but no vendor lock-in.

Either works. Pick based on whether you already have a logging stack you trust. The format matters more than the tool. As long as every request has a structured trace, you can switch tools later.

Layer 2: Evals in CI

You read the evals article. Now: where do evals actually run in your workflow?

flowchart LR
    PR[Developer opens PR] --> CI[CI runs eval suite]
    CI --> COMPARE{Worse than main?}
    COMPARE -->|Yes| BLOCK[Block merge<br/>show regressions]
    COMPARE -->|No| ALLOW[Allow merge<br/>green check]

Eval pipeline runs on every PR that touches:

The system prompt
Any RAG component (chunking, embeddings, reranker)
The model configuration

The CI job:

Pull the eval dataset (versioned in git)
Run each case through the new code
Score with rules + LLM-judge
Compare against the baseline (main branch scores)
Post a GitHub comment with the diff
Fail the build if any metric regresses beyond threshold

Real example of a PR comment:

Eval Results vs main

Overall:           87.2% → 88.4%  (+1.2%)
Relevance:         91.3% → 92.0%  (+0.7%)
Faithfulness:      85.1% → 88.2%  (+3.1%)  ✓
Tone match:        88.0% → 86.5%  (-1.5%)  ⚠️ within tolerance
Latency P95:       2.4s  → 2.8s   (+0.4s)  ⚠️ within tolerance

5 of 50 cases changed verdict:
  +3 improved
  -2 regressed (see report.html)

This is the single most useful CI integration for AI features. It catches accidental regressions before they reach users.

Layer 3: Production sampling

Pre-deploy evals catch regressions in your eval set. They do not catch:

New user queries you never anticipated
Model provider changes (yes, providers silently update model weights)
Data drift (your corpus changes over time)

The fix is sampling production traffic.

flowchart LR
    PROD[Production traffic] --> SAMPLE[Sample 1-5% of requests]
    SAMPLE --> JUDGE[Run LLM judge offline]
    JUDGE --> TRACK[Track score per day]
    TRACK --> ALERT{Score dropped<br/>>5% week over week?}
    ALERT -->|Yes| PAGE[Page on-call]
    ALERT -->|No| OK[Continue]

How to do it:

Sample 1-5% of production requests (more if low volume, less if high volume)
Score each sample asynchronously with your LLM judge
Track the rolling average over time
Alert when it drops

Add user signals on top:

Thumbs up/down rate
Regeneration rate (high = users are unsatisfied with first answer)
Time-to-first-action (longer = confused)
Copy/paste rate (higher = useful answer)

Drift detection

Beyond quality scores, monitor:

Query distribution drift: are users asking new kinds of questions? Embedding-cluster your queries weekly and watch for new clusters.
Cost per call: is it creeping up? Caching breaking? Model dropdown not happening?
Latency P95: regressed? Provider issue? Caching cold?
Refusal rate: model refusing more often? Possibly a prompt regression.
Token usage: did average input/output tokens change? Could indicate a leak or a prompt change someone forgot to revert.

flowchart TB
    METRICS[Production metrics, per day]
    METRICS --> M1[Quality score]
    METRICS --> M2[Cost per call]
    METRICS --> M3[Latency P95]
    METRICS --> M4[Refusal rate]
    METRICS --> M5[Token usage]
    M1 --> DASH[Single dashboard,<br/>last 30 days]
    M2 --> DASH
    M3 --> DASH
    M4 --> DASH
    M5 --> DASH
    DASH --> EYES[Reviewed weekly]

What about PII

The hard tension in AI observability: you want to log everything to debug. You cannot store user PII forever.

Pragmatic patterns:

Hash sensitive fields before logging: emails, names, IDs → SHA-256
Truncate or summarize prompts in long-term storage: keep the trace structure, drop the full text after 7 days
Sample at storage time: keep 100% of traces for 24 hours, 1% for 90 days, 0.01% forever
Separate stores: full traces in a short-retention bucket; sanitized metrics in long-retention

Build this in from day one. Retrofitting after a data review request is much harder.

A minimal stack for a startup

You do not need the full enterprise setup. Here is what a 5-engineer team needs:

Traces: structured JSON logs sent to your existing observability tool (Datadog/Honeycomb). One log line per LLM call with the structure above.
Evals in CI: a Python script + 100-case dataset in git. Runs on PR, posts results.
Production sampling: a cron job that pulls 1% of yesterday’s prod requests, scores them with a judge, writes a daily summary.
Dashboard: one page in your existing tool showing the 5 metrics from earlier, last 30 days.

Total effort: 1 to 2 weeks. Saves you from many production fires.

Common beginner mistakes

Treating LLM errors like HTTP errors. A 200 OK with bad content is the silent killer. Log quality signals, not just status codes.

Not capturing intermediate steps. If you only log “request in, response out”, you cannot debug WHICH step in your RAG pipeline failed.

No PII strategy. “We will figure that out later” leads to a panic deletion run when someone asks “what about GDPR”.

Skipping eval in CI. “We will run evals before each release.” You will not. Automate it on every PR.

One huge dashboard. Engineers do not look at dashboards with 50 metrics. Make it 5-7 numbers that tell you the story.

Missing the model version. Provider updates a model behind the same name. You see quality drop, do not know why. Pin model versions in your config and log them.

Questions you will face in production

“How would you debug a sudden quality drop?” Pull recent traces. Compare them to a week ago. Look at retrieval scores (did retrieval get worse?), output token counts (model rambling?), model version (provider update?). If still unclear, sample 50 recent low-rated responses and read them.

“How do you know your evals are correct?” Calibrate the judge against humans every quarter. Track judge-human agreement. If it drops below 70%, rewrite the rubric.

“What is your alerting strategy?” Page on: error rate above N%, quality score drop above X% week-over-week, cost per call up above Y%. Daily summary email of metrics for the team. No noisy alerts.

“How do you handle a model provider outage?” Multi-provider fallback with a shared interface. Health-check both providers continuously. Alert when one is degraded but keep serving from the other.

“How do you detect prompt injection?” Log every prompt + completion. Sample some, run a classifier (or a separate LLM judge) for injection patterns. Alert on hits.

What to remember

Three layers: traces (per call), evals in CI (per PR), production sampling (per day).
The format of traces matters more than the tool you use.
Eval-in-CI catches regressions before they ship. Always block merge on regression.
Production sampling catches drift after deploy. Always page on quality drop.
Always pin model versions in config; provider-side updates can silently change behavior.
Hash PII before logging. Plan retention from day one.

What to study next

Last article in this track: now optimize what you can see. Move to article 12, Cost Optimization in Production AI.