Guide · AI agent evaluation

LLM as a judge: grading AI output with another model.

A judge model reads an answer against written criteria and returns a verdict. It scales where people cannot — and it has biases you must measure before you trust it.

When an output is free text — a summary, an answer, a reply to a customer — no exact rule can grade it. Using an LLM as a judge means asking a second model to grade it against a rubric. This guide covers the three ways to set one up, what the research found about how often judges agree with people, the biases that skew them, and a step-by-step method for building a judge whose verdicts you can rely on.

Updated October 2026 · 8 sources

Read the guide How Botminds does it

What goes into one judge call

Criteriaa written rubric: what counts as correct, complete, grounded

rubric

The inputthe question or task the system was given

task

The outputthe answer being graded — or two answers to compare

candidate

Referencea known good answer or the source passages, when you have them

optional

Reasoningthe judge explains its view against each criterion first

before the verdict

Verdictpass or fail, A or B, or a score on a defined scale

structured

What is LLM as a judge?

LLM as a judge is an evaluation method in which a language model grades the output of another AI system against written criteria — scoring one answer, picking the better of two, or comparing an answer with a reference. It replaces slow human grading for open-ended text at scale. Research shows strong judges agree with human preferences about as often as people agree with each other, but they also favour the first answer shown, longer answers and text like their own, so a judge must be checked against human labels before it is trusted.

How it works

Three ways to set up LLM as a judge: single grading, pairwise comparison, reference-guided.

The MT-Bench paper that made the method mainstream used three set-ups. Each answers a different question and fails in a different way.

01one output

Single answer grading

The judge scores one answer against the rubric, alone.

Works for monitoring live traffic
Scores drift without anchors
Prefer pass or fail to 1–10

02two outputs

Pairwise comparison

The judge sees two answers to the same input and picks the better one, or a tie.

Best for comparing two versions
Sensitive to which answer comes first
Run both orders

03output + key

Reference-guided grading

The judge compares the answer with a known good one or with source passages.

A reference cut maths grading failures from 14 of 20 to 3 of 20 in MT-Bench
Needs a golden dataset
Checks key facts and grounding

Which set-up fits the question you are asking?

Is this answer acceptable?Single grading with binary criteria — and a reference if you have one.

Is version B better than version A?Pairwise comparison, run in both orders, counting ties.

Did it get the facts right?Reference-guided: list the key facts and ask, for each, stated or not.

The evidence

LLM as a judge reliability: expert-level agreement, with measured biases.

Agreement with people is high on average and uneven in the details. How ties are counted moves the headline figure by twenty points, and the same papers that showed judges work also catalogued how they go wrong.

85%

GPT-4’s agreement with expert votes on MT-Bench when ties are left out; experts agreed with each other 81%. Counting ties, the two figures are 66% and 63%.

Zheng et al. (2023)

66 of 80

Queries on which Vicuna-13B beat ChatGPT, as judged by ChatGPT, after only the order of the answers was changed.

Wang et al. (2023)

7×+

Lower cost of a panel of smaller judges from different model families than one large GPT-4 judge — which the panel also outperformed, with less intra-model bias.

Verga et al. (2024)

Mar 2023G-EvalA judge reasons through criteria, then fills in a form; better agreement with people on summaries than older metrics.
May 2023Position bias measuredSwapping the order of two answers flips verdicts; calibration by running both orders.
Jun 2023MT-Bench and Chatbot ArenaThree judge set-ups, 80%+ agreement on non-tie votes, and the biases named: position, verbosity, self-enhancement.
Apr 2024Self-preference shownJudges rate their own outputs above others’ that people judge equal, and can recognise their own text.
Apr 2024Juries over judgesSeveral smaller judges from different families beat one large one.
Oct 2024Agent as a judgeAn agent that can inspect files and run checks grades agent work as reliably as human evaluators did.

Known failure modes

LLM judge biases and the countermeasure for each.

The first four are measured in published work; the fifth is the property every model grader shares. None is a reason to stop using judges; each is a reason to design the judge call carefully and keep measuring it.

Bias	What happens	Countermeasure
Position	The answer shown first, or second, wins more often regardless of quality. In MT-Bench, GPT-4 gave the same verdict after a swap in 65% of cases.	Run pairwise comparisons in both orders; count a split as a tie.
Verbosity	Longer answers score higher, even when the extra text adds nothing. Padding a list with rephrased items fooled GPT-3.5 and Claude-v1 91% of the time, GPT-4 9%.	Put length in the rubric; compare answers of similar length; grade claims, not prose.
Self-preference	A judge rates its own outputs above others’ that people judge equal; GPT-4 and Llama 2 can tell their own text apart.	Use a judge from a different family than the system under test, or a panel of several.
Weak reasoning	Judges miss errors in maths and reasoning — GPT-4 passed wrong answers to problems it could solve alone, misled by the answer it was shown.	Grade those with code: run the tests, check the sums. Use a reference answer.
Non-determinism	The same output can get a different verdict on another run, and verdicts shift when the judge model or prompt changes.	Binary verdicts; re-check against human labels after any change to the judge.

A judge is a model, so it is evaluated like one. Anthropic’s guidance lists the weakness of model-based graders plainly: they are non-deterministic and need calibration against human graders. Treat the judge as one more system with its own golden dataset.

Step by step

How to build an LLM as a judge you can trust, in five steps.

The steps are the same whether the judge grades summaries, support replies or extracted clauses. The order matters: a rubric nobody agreed on cannot be calibrated.

Write binary criteria
Split “good” into yes-or-no checks: states each key fact, makes no claim the source does not back, asks before acting.
rubric
Choose the judge
The most capable model you can afford, from a different family than the system under test.
independent
Reason, then decide
The judge explains against each criterion, then returns a structured verdict.
structured
Label a sample by hand
People grade a sample with the same rubric — 50 to 100 outputs is our rule of thumb, not a published figure — and you count how often the judge agrees.
calibrate
Keep checking
Re-sample after any change to the rubric or the judge model, and read the disagreements.
monitor

A binary, reference-guided judge prompt (abridged)

You are grading an answer to a customer question.

Question:   {question}
Source:     {retrieved passages}
Key facts:  {facts the answer must state}
Answer:     {answer under test}

For each key fact: is it stated in the answer? (yes/no)
For each claim in the answer: is it supported
by the source? (yes/no)

Explain briefly, then return JSON:
{ "facts_stated": [...], "claims_without_support": [...],
  "verdict": "pass" | "fail" }

Why each choice

Binary checksYes or no holds steadier across runs than a score out of ten.
Key facts listedThe judge checks a list instead of judging “quality” in the open.
Source attachedGrounding is graded against the passages, not the judge’s own knowledge.
JSON verdictThe result can be counted, compared and gated on.

Choosing a grader

When an LLM judge is the right grader — and when code or a person should grade.

Graders sit on a scale from cheap and rigid to costly and nuanced. A good evaluation uses the cheapest grader that can see the failure, and reserves people for calibrating the rest.

Exact and cheapNuanced and costly

12345

Code gradersString match, numeric tolerance, unit tests, checks on the end state. Fast, repeatable, brittle to valid variation.
Reference-guided judgeA model checks an answer against known facts or sources. Flexible, needs a golden dataset.
Open rubric judgeA model grades tone, completeness or helpfulness. Scales well; must be calibrated.
Agent as a judgeA judge that can open files and run checks, for grading an agent’s multi-step work.
Domain expertsThe reference for everything else. Slow and expensive; use them to calibrate.

On Botminds

How Botminds uses LLM as a judge inside firm rules.

Botminds uses a judge where rules cannot read — and keeps it honest: rules score everything they can, the judge answers yes or no, and the scorecard names the judge.

Rules first
Every field a rule can score is scored by the rule.
Yes or no
The judge returns a binary verdict — supported or not.
Your choice of judge
Pick one from a different model family than the agent.
Named on the card
The scorecard names the judge, and shows a fallback as a fallback.
Compared with a baseline
Verdicts feed a case-by-case comparison with a pinned baseline when an agent is published.

Key facts

Does the answer state each fact the case expects?

Grounding

Is each claim supported by the source documents?

Conversations

Checked by rules on every turn: said what it must, avoided what it must not, asked before changing anything.

Read the full design in AI agent evaluation, and set up an evaluation in the Academy lesson crew and evaluations.

Questions people ask

LLM as a judge: common questions.

The technology behind itEvaluation & observabilityEvery change measured, every run watched

Is LLM as a judge reliable?

On average, strong judges agree with human preferences over 80% of the time — about as often as people agree with each other. That figure leaves out ties; counting them, GPT-4 and experts agreed 66% of the time and experts with each other 63%. Judges are weaker on maths and reasoning, and biased by answer order, length and their own outputs, so validate a judge against human labels for your task before trusting it.

What is LLM as a judge used for?

Grading open-ended output that no exact rule can score: whether a summary is faithful, whether an answer covers the key facts, whether a reply follows tone and policy, and which of two versions of a system is better.

What is a rubric in LLM as a judge?

The written criteria the judge applies. Good rubrics break quality into specific yes-or-no checks — states each key fact, makes no claim the source does not back — rather than asking for an overall score.

What is position bias in LLM judges?

The tendency to prefer the answer in a particular position when comparing two. In one study, changing only the order let a weaker model beat a stronger one on 66 of 80 questions. Running each comparison in both orders cancels it.

Should the judge be a different model from the one being tested?

Yes, where you can. Judges rate their own outputs above others of equal quality; a judge from a different family, or a panel of several from different families, reduces that bias.

What is the difference between LLM as a judge and agent as a judge?

An LLM judge reads the output and returns a verdict. An agent judge can act: open files, run code and inspect intermediate steps, which suits grading an agent’s multi-step work. Research in 2024 found it as reliable as human evaluators on a benchmark of 55 AI development tasks.

How do you evaluate an LLM judge?

Have people grade a sample of outputs with the same rubric, measure how often the judge agrees, read the disagreements, and repeat after any change to the rubric or the judge model.

Can an LLM judge replace human review?

For scoring at volume, largely yes — once calibrated. People remain the reference: they write the rubric, label the calibration sample and settle the cases where the judge and the rules disagree.

Sources

Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)arxiv.org
Wang et al.: Large Language Models are not Fair Evaluators (2023)arxiv.org
Liu et al.: G-Eval (2023)arxiv.org
Verga et al.: Replacing Judges with Juries (2024)arxiv.org
Panickssery et al.: LLM Evaluators Recognize and Favor Their Own Generations (2024)arxiv.org
Zhuge et al.: Agent-as-a-Judge (2024)arxiv.org
Anthropic: Demystifying evals for AI agents (2026)anthropic.com
OpenAI: Evaluation best practicesdevelopers.openai.com

See it on
your own work.

Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.

Book a demo Evaluation & observability