How to Know If Your AI Is Actually Working

You built an AI feature. It works on your test prompts. You ship it. Two weeks later, a customer screenshots a wild answer and asks “is this normal?”

This is the problem evaluation solves. In AI engineering, we shorten it to evals.

Evals are how you measure whether your AI system is good, before you ship a change, and while it runs in production.

If RAG is the most common AI pattern, evals is what separates someone who has built AI products from someone who has only played with them. This guide will get you up to speed.

Why evals are so important

Traditional software is easy to test. You write a unit test. The function returns 5. You expect 5. Pass or fail.

AI is different. An LLM can answer the same question in 50 different correct ways. There is no single “right” output.

flowchart LR
    subgraph trad ["Traditional code"]
        direction LR
        T1[Input: 2 + 3] --> T2[Function]
        T2 --> T3[Output: 5]
        T3 --> T4{Equals 5?}
        T4 -->|Yes| T5[Pass]
    end

flowchart LR
    subgraph ai ["AI output"]
        direction LR
        A1[Input: question] --> A2[LLM]
        A2 --> A3[Output: 200 words]
        A3 --> A4{Good?}
        A4 -.???.-> A5[How do we<br/>even check?]
    end

So “is this output good?” becomes a real problem. Evals are the answer.

What an eval actually is

At its simplest, an eval is three things:

A list of test cases (questions, prompts, or tasks)
A way to run each test through your AI
A way to score the output

flowchart LR
    DATA[Eval dataset:<br/>200 test cases] --> RUN[Run each case<br/>through your AI]
    RUN --> SCORE[Score each output]
    SCORE --> REPORT[Total score:<br/>87 out of 100]

That is it. The hard part is each piece. Take them in turn.

Building your eval dataset

This is 80% of the work. People skip it. Don’t.

The best test cases come from real users.

flowchart TB
    SRC1[Real user questions<br/>from production logs] --> DATASET[(Your eval dataset<br/>200-500 cases)]
    SRC2[Bug reports<br/>things that broke] --> DATASET
    SRC3[Edge cases<br/>weird inputs] --> DATASET
    SRC4[Adversarial prompts<br/>jailbreak attempts] --> DATASET

Where to get test cases, in order of importance:

Real production logs. Sample 200-500 questions actual users asked. For each, write down what the ideal answer would look like.
Things that broke. Every bug report becomes a permanent test case. Never delete one once added.
Edge cases. Weird inputs, very short questions, very long ones, ambiguous ones.
Adversarial cases. People trying to get the AI to do something harmful or break out of its instructions.

Why 200-500 cases? Why not 50 or 5,000?

Below ~50 cases, statistical noise dominates: a regression of 2 cases (out of 50) looks like a 4% drop, but it could just be random variation. You cannot trust the signal.

At 200-500 cases, the signal stabilizes. A 1-2% change is meaningful. You can split into slices (by feature, by difficulty) and still have enough cases per slice to draw conclusions.

Above ~2,000 cases, you hit diminishing returns. The marginal information from each new case drops, but the cost of maintaining and re-labeling them grows. Worse, large eval sets get stale: the queries from 12 months ago no longer reflect what users are asking.

Pick 200-500 high-quality cases over 5,000 mediocre ones. Treat the set as living: add new bug-derived cases, prune obsolete ones, re-label when behavior changes.

Start with 20 cases you have carefully chosen and labeled. Grow to 200 only once those 20 are stable. 50 well-chosen cases beat 5,000 random ones; the bad ones add noise and hide regressions.

Scoring outputs: 3 ways to do it

Now you have your test cases. You run them through your AI. You get 200 outputs. How do you score them?

There are three main approaches.

Method 1: Rules (deterministic checks)

You write a function that checks the output. Simple, fast, free.

Examples:

Is the output valid JSON?
Does it contain the right product ID?
Is it under 200 words?
Does it avoid certain forbidden words?

flowchart LR
    OUT[AI output] --> RULE[Check: Is it<br/>valid JSON?]
    RULE -->|Yes| PASS[Pass]
    RULE -->|No| FAIL[Fail]

Use this when: the task has a clearly checkable correct format.

Method 2: LLM-as-judge

You ask another LLM to grade the output. This is the most common method for open-ended tasks.

flowchart LR
    Q[Original question] --> JUDGE[Judge LLM]
    OUT[AI output] --> JUDGE
    RUBRIC[Scoring rubric] --> JUDGE
    JUDGE --> SCORE[Score: 4 out of 5<br/>Critique: Missed step 3]

You give the judge:

The original question
The AI’s answer
A rubric (what good looks like)

The judge writes a short critique and gives a score (often 1-5).

Use this when: the task is open-ended, like answering questions or writing summaries.

Method 3: Human review

A real person reads the output and scores it. Expensive but high quality.

Use this when: the task is creative, subjective, or high-stakes. Also: use a few human-scored cases to check that your LLM-as-judge agrees with humans. If the judge says “good” and humans say “bad”, your judge needs fixing.

A simple LLM-as-judge prompt

Here is what a basic judge prompt looks like:

You are reviewing an AI assistant's answer to a customer question.

QUESTION:
How do I reset my password?

AI ANSWER:
Go to Settings > Security > Reset Password. You will get an
email with a reset link.

Rate the answer on these three criteria, 1-5 each:
- Relevance: does it answer the actual question?
- Accuracy: is the information correct?
- Clarity: is it easy to understand?

First write a short critique (1-2 sentences).
Then output JSON: {"critique": "...", "relevance": N, "accuracy": N, "clarity": N}

The “critique first, then score” pattern is important. It forces the judge to think before grading. Without it, you get noisy scores.

Three biases to know about LLM judges

LLM judges have known weaknesses. Here are the ones that matter most.

Position bias: LLMs tend to prefer the first option in a comparison. Fix: when comparing A vs B, run it twice with the order flipped, and average.

Verbosity bias: LLMs tend to prefer longer answers. Fix: include length in the rubric.

Self-preference bias: GPT-4 thinks GPT-4’s answers are great. Fix: use a different model as the judge than the one you are evaluating.

The full eval pipeline

Now you have all the pieces. Here is what a real eval pipeline looks like:

flowchart TB
    PR[Developer<br/>opens a PR] --> RUN[Run eval suite]
    RUN --> LOAD[Load 200 test cases]
    LOAD --> EXEC[Run each through<br/>the new AI version]
    EXEC --> JUDGE[Score with rules<br/>and LLM judge]
    JUDGE --> AGG[Add up all scores]
    AGG --> COMPARE{Better than<br/>current version?}
    COMPARE -->|Yes| ALLOW[Allow merge]
    COMPARE -->|No| BLOCK[Block merge<br/>show what regressed]

This runs automatically on every code change. If your new version scores lower than the current one, the system blocks the merge. This is how you keep AI quality from silently dropping over time.

Watching production (after you ship)

Pre-deploy evals are not enough. Things change in the real world:

Users start asking new kinds of questions
The LLM provider quietly updates their model
A prompt change has unintended effects

So you also need to watch production.

flowchart LR
    PROD[Production<br/>traffic] --> SAMPLE[Sample 1% of<br/>real questions]
    SAMPLE --> JUDGE[Score with<br/>LLM judge]
    JUDGE --> TRACK[Track score<br/>over time]
    TRACK --> ALERT{Score dropped<br/>5% this week?}
    ALERT -->|Yes| PAGE[Alert the team]
    ALERT -->|No| OK[All good]

You also collect direct user signals:

Thumbs up / down buttons
Did the user copy the answer?
Did the user click “regenerate”? (If yes, the first answer was probably bad.)

Common beginner mistakes

Mistake 1: Skipping evals entirely. “I’ll add them later” usually means never. Add evals from day one, even if they are tiny.

Mistake 2: Treating the eval set as static. Real users do unexpected things. Grow the eval set continuously by adding cases from production every week.

Mistake 3: Using one big score for everything. A single number hides problems. Score by category. The overall score might be 85% while one important category silently regressed to 60%.

Mistake 4: Not measuring the judge. Your LLM judge is also a model that can be wrong. Periodically have a human label 50 cases, then check how often the judge agrees.

Mistake 5: Using BLEU or ROUGE scores. These are old metrics from before LLMs. They do not work well for modern AI output. Ignore them.

Questions you will face in production

“How do you build the eval set in the first place?” Start with real user questions from production logs (or, if pre-launch, your own team’s questions). Label them with what a good answer would look like. Add edge cases as you find them.

“What if your eval set leaks into the model’s training data?” This is a real risk. Solution: keep a small private test set that you never publish or share. Re-create it periodically.

“How do you evaluate a multi-step AI agent?” Score the final output AND each step. Did the agent pick a sensible plan? Did each tool call have correct arguments?

“How do you eval things like creative writing where there is no right answer?” Pairwise comparison: show two outputs and ask “which is better?” Track the win rate over time.

“What is your cost per eval run?” Each run is maybe 200 cases × 2-3 LLM calls = ~600 API calls. At a penny per call, $6 per run. Cache aggressively. Use cheaper models for the judge when possible.

What to remember

Evals = test cases + a way to run them + a way to score them
The dataset is most of the work
Three scoring methods: rules, LLM judge, human review
LLM judges have known biases, write the rubric carefully
Run evals before every deploy AND monitor production
A growing eval set is more valuable than the model itself

What to study next

If you want to go deeper:

Hamel Husain’s blog (hamel.dev), the best practical resource on production evals
“Judging LLM-as-a-Judge” paper, explains judge biases in detail
OpenAI evals or Inspect AI, open source frameworks you can borrow patterns from

The best way to learn is to build one. Pick a small AI feature you have shipped. Write 20 test cases. Score them. You will learn more in one weekend than in a month of reading.