LLM-as-Judge: When It Breaks and How to Fix It

In the evals article we introduced LLM-as-judge: using one LLM to score the output of another. It is the most useful technique in modern eval pipelines because it works for open-ended tasks where rules-based scoring cannot.

It is also full of subtle failure modes that look like bugs in your AI, when really the judge is wrong.

This article goes deep on the biases, how to detect them, and how to build a judge you can trust.

The basic setup, recap

flowchart LR
    Q[Original question] --> JUDGE[Judge LLM]
    OUT[AI output to grade] --> JUDGE
    RUBRIC[Rubric: how to score] --> JUDGE
    JUDGE --> SCORE[Score + critique]

You give the judge:

• The original question or task
• The AI’s output
• A rubric explaining what good and bad look like

The judge returns a score and a short critique. You aggregate scores across many test cases to get a quality signal.

This works. It also has at least four well-documented biases.

Bias 1: Position bias

When you ask a judge to compare two outputs side by side (“Which is better, A or B?”), the judge picks the first one more often than chance. This has been measured at 60% to 70% in some studies.

flowchart LR
    SAME["Identical outputs A and B<br/>given to judge"]
    SAME --> J1[Judge picks A in 65% of runs]
    SAME --> J2[Judge picks B in 35% of runs]

Why it happens: the model gives extra weight to the first option because of how attention works during generation.

Fixes:

• Randomize the order before sending to the judge
• Better: run twice, swap order, average the two
• Even better: train a calibration with both orderings on your data

def pairwise_judge(query, output_a, output_b):
    score_ab = judge_score(query, output_a, output_b)  # A first
    score_ba = judge_score(query, output_b, output_a)  # B first
    # Average the verdict; tie-break with confidence
    return (score_ab + (1 - score_ba)) / 2

The first time you measure position bias with the swap-and-average trick, you will be unsettled by how much your single-order scores were lying to you.

Bias 2: Verbosity bias

Longer outputs are judged as higher quality, even when they say the same thing.

flowchart LR
    A["A: 'Yes, the answer is 42.'<br/>(10 tokens)"]
    B["B: 'Certainly! After careful<br/>consideration of all factors,<br/>I can confidently say the<br/>answer to your question is 42.'<br/>(40 tokens)"]
    A --> JUDGE[Judge]
    B --> JUDGE
    JUDGE --> WIN["Judge picks B<br/>'more thorough'"]

Why it happens: the model has learned that lengthy academic-style text correlates with quality in its training data. Even though both answers are equally correct.

Fixes:

• Add a length-aware criterion to the rubric: “concise is good; do not reward unnecessary elaboration”
• Use the rubric explicitly: “rate conciseness (1-5)” alongside quality
• For pairwise, length-normalize before comparing
• For high-stakes scoring, drop the judge and do absolute scoring with explicit criteria

Bias 3: Self-preference bias

GPT-4 thinks GPT-4 outputs are better than Claude outputs. Claude thinks Claude is better. Each major model has been shown to prefer its own outputs in blind comparison.

flowchart LR
    OUTGPT["Output from GPT-4o"]
    OUTCLAUDE["Output from Claude"]
    OUTGPT --> JG[GPT-4o judge]
    OUTCLAUDE --> JG
    JG --> WG["GPT-4o picks GPT-4o<br/>more often than chance"]
    OUTGPT --> JC[Claude judge]
    OUTCLAUDE --> JC
    JC --> WC["Claude picks Claude<br/>more often than chance"]

Why it happens: models prefer outputs that match their own stylistic distribution. They were trained on slightly different data.

Fixes:

• Use a different model as the judge than the one being evaluated
• Use an ensemble of judges (GPT + Claude + Gemini, take majority)
• Calibrate judge scores against human labels regularly

If you can only use one judge model, just be honest about it in your eval reports.

Bias 4: Format and style bias

Markdown headers. Bulleted lists. Emojis. The judge often rewards outputs that look polished, regardless of content quality.

Fixes:

• Add a “content over presentation” criterion explicitly
• Strip Markdown from outputs before sending to the judge if format does not matter for your use case
• Pair the judge with a rules-based check for the actual facts (e.g., “did the output mention X?”)

Calibrate against humans

This is the technique that separates serious eval pipelines from theater. Periodically have humans label a small set of cases, then measure how often the judge agrees.

flowchart LR
    SET[50 to 100<br/>labeled cases] --> HUMAN[Human score]
    SET --> JUDGE[Judge score]
    HUMAN --> COMP[Agreement rate]
    JUDGE --> COMP
    COMP --> ALERT{Agreement<br/>below 70%?}
    ALERT -->|Yes| FIX[Rewrite rubric;<br/>try a different judge model]
    ALERT -->|No| KEEP[Trust the judge]

Use Cohen’s kappa or simple percent agreement. Aim for:

• 80%+ agreement on objective questions (correctness)
• 70%+ on subjective ones (helpfulness, tone)

If your judge agrees with humans 50% of the time, your judge is barely better than chance.

Rubric design that survives reality

A good rubric is the difference between a judge that returns useful scores and one that returns random ones. The pattern that works:

You are evaluating an AI assistant's email reply.

ORIGINAL EMAIL:
{email}

AI REPLY:
{reply}

Score the reply on three independent dimensions, 1 to 5:

1. RELEVANCE: does it answer what was actually asked?
   1 = completely off-topic
   3 = partially addresses the question
   5 = directly answers the question

2. FAITHFULNESS: does it stick to what is in the original email?
   1 = invents facts not present
   3 = adds reasonable inferences
   5 = strictly grounded in the email

3. TONE MATCH: does it match the original's tone (formal/casual/urgent)?
   1 = completely mismatched
   3 = mostly matches
   5 = matches well

Write a one-sentence critique. Then output JSON:
{"critique": "...", "relevance": N, "faithfulness": N, "tone": N}

Key principles:

• Independent dimensions. Score each separately. Aggregate later. Otherwise the judge anchors on one bad thing.
• Critique before score. Forcing the judge to articulate reasoning before scoring reduces noise.
• Explicit anchors. Tell the judge what 1, 3, and 5 mean. Otherwise the scale drifts.
• Output format. Always JSON. Always parse it. Validate.

Pairwise vs absolute scoring

You have two options for getting a quality signal:

Absolute scoring

Each output gets a number. Easy to aggregate. Easy to track over time.

Pairwise comparison

Show two outputs side by side. Judge picks the better one. Less noisy than absolute. Better for subtle quality differences.

flowchart LR
    subgraph abs["Absolute scoring"]
        A1[Output] --> A2[Judge] --> A3[Score 1-5]
    end
    subgraph pair["Pairwise"]
        P1[Output A] --> PJ[Judge]
        P2[Output B] --> PJ
        PJ --> PR[Win/Loss/Tie]
    end

Use absolute when you want to track a quality trend over many releases.

Use pairwise when comparing two specific versions (A/B testing prompt v3 against v4). Track win rate: percent of cases where the new version wins.

A typical eval pipeline uses both: absolute scoring as the deploy gate (block if regression), pairwise for promotion decisions.

Cost considerations

Judges are LLM calls. They cost money.

For a 500-case eval:

• 1 generation call + 1 judge call per case = 1000 calls
• At $0.01 per call → $10 per eval run

Cost optimizations:

• Use a cheap judge (Haiku, GPT-4o-mini) for first-pass scoring
• Escalate to a strong judge (Sonnet, GPT-4) for cases where cheap judge is uncertain
• Skip judging on cases where rule-based checks pass (saves 30-50% of calls)
• Cache: if the input/output pair is unchanged from last run, reuse last score

Common beginner mistakes

One score, all criteria. Combining “is it correct AND well-written AND on-topic” into a single 1-5 hides what is failing. Score each independently.

No human calibration. You will not catch when the judge starts drifting or has a bias for your specific task. Label 50 cases quarterly.

Same model for generation and judging. Self-preference bias. Use different models.

Skipping the critique. The critique is forcing the judge to think. Without it, scores are noisier by 30%+ in published studies.

Trusting absolute scores blindly. If your judge says “4 out of 5”, but agrees with humans 60% of the time, your 4 is meaningless. Calibrate first, trust second.

Questions you will face in production

“How do you know your judge is correct?” Calibrate against humans. Track agreement over time. Publish the numbers internally so the team knows what to trust.

“What if the judge has a bias your humans share?” Possible. Get evals from multiple humans with different backgrounds. Or use multiple judge models.

“How do you handle a judge that says everything is great?” Score-distribution check: if your judge gives 5/5 for 80% of cases, the rubric is too lenient or the anchors are vague. Rewrite anchors with stricter criteria.

“Can you trust the judge for safety evals?” With more caution. Safety failures are rare and high-cost. Use multiple judges plus human review for any case the judges flag, even if their confidence is low.

“How do you build a rubric for a new feature?” Start with 5-10 cases you score yourself. Notice what made you give different scores. Turn that into the rubric. Iterate.

What to remember

• LLM-as-judge has known biases: position, verbosity, self-preference, format.
• Mitigate position bias with order-swap and average.
• Mitigate verbosity with length-aware rubrics.
• Mitigate self-preference by using a different model than what you are evaluating.
• Score independent dimensions separately. Critique before score.
• Calibrate against humans. Agreement < 70% means rewrite the rubric.
• Pairwise comparisons are less noisy than absolute scoring for close-call decisions.

What to study next

You can now build evals you trust. Next, optimize the production system you are evaluating. Move to article 10, Caching for LLM Apps.

Further reading

• Zheng et al. (2023) — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The paper that defines the position bias / verbosity bias / self-preference findings cited here.
• Hamel Husain — LLM-as-judge in production. How a working ML engineer calibrates judges against humans.
• LMSYS Chatbot Arena. Real-world pairwise judging at massive scale; methodology page is instructive.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.