Guide · AI agent evaluation

LLM evaluation metrics: what each one measures, and what it misses.

Q: What are the most important LLM evaluation metrics?

The ones that match your output. For structured fields: field accuracy and critical-field accuracy. For RAG: context recall and faithfulness. For code: tests passed and pass@k. For agents: outcome checks and pass^k. Always add cost and time per case, and calibration if you route on confidence.

Q: Is BLEU a good metric for evaluating LLMs?

Rarely on its own. BLEU counts shared word sequences with a reference, so it penalises correct paraphrases and can reward a fluent answer with one wrong word. Use it as a tripwire for sudden drops, with task checks or a judge for correctness.

Q: What is the difference between ROUGE and BLEU?

BLEU measures precision of shared n-grams and was designed for translation. ROUGE measures recall of the reference’s n-grams or longest common subsequence and was designed for summaries. Both measure overlap of words, not correctness of meaning.

Q: What is the difference between pass@k and pass^k?

pass@k, the standard code metric since HumanEval, is the chance that at least one of k attempts succeeds; pass^k, introduced with τ-bench, is the chance that all k succeed. With 75% per attempt, pass@3 is about 98% while pass^3 is about 42%. Use pass^k when every run reaches a user.

Q: What is faithfulness in RAG evaluation?

The share of claims in an answer that are supported by the retrieved passages. Low faithfulness means the model added things the sources do not say, even when retrieval was correct.

Q: How do you measure LLM accuracy?

Run the model on a golden dataset of real cases with verified answers, score each output with a rule suited to it — exact, fuzzy, numeric tolerance, or a judge for free text — repeat runs to measure consistency, and report the margin of error with the score.

Q: What is G-Eval?

A method from 2023 in which a strong model reasons through evaluation criteria and then fills in a score form. It correlated better with human ratings than earlier automatic metrics, and its authors noted a bias toward text written by language models.

Q: What metrics does HELM use?

HELM reports seven: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, across many scenarios, so models are compared on the same cases and on more than accuracy.

Every metric answers one narrow question. Picking the right ones means knowing the question your users actually care about.

There are dozens of LLM evaluation metrics, from word-overlap scores built for machine translation to judges that read an answer against a rubric. This guide sorts them into families, works through what each one rewards and where it misleads, and ends with a decision guide that maps your kind of output to the metrics worth reporting.

Updated October 2026 · 15 sources

Read the guide How Botminds does it

Six families of metric, from strings to systems

Exact and field matchis this value right, under a rule for the field?

exact matchF1field accuracy

Word overlaphow many words and phrases match a reference?

BLEUROUGEMETEOR

Meaning similarityhow close is the meaning, by embeddings?

BERTScorecosine

Model-gradeddoes it meet the rubric, as read by a judge model?

G-Evalrubricpairwise

Retrieval and groundingdid it fetch the right passages and stay within them?

faithfulnesscontext recall

Reliability and costis it right every time, fast enough and affordable?

pass^kcalibrationlatencycost

What are LLM evaluation metrics?

LLM evaluation metrics are the measures used to score a language model’s output: exact match and F1 for answers with one right value, word-overlap scores such as BLEU and ROUGE, embedding similarity such as BERTScore, model-graded scores from an LLM judge, grounding metrics such as faithfulness for RAG, and reliability measures such as pass^k, calibration, latency and cost. No single metric is enough; you pick the ones that match what your users would call a correct answer.

Two questions sort them

A map of LLM evaluation metrics: does it need a reference, and who grades?

Two questions place every metric. Do you need a known right answer to compute it? And is the grade computed by code, or read by a model? The answer decides what the metric costs to run and how far you can trust it.

Graded by a model

Reference-guided judgeA model compares the output with a known answer: “does it state each key fact?”judge + key

Open judgingA model scores against a rubric or picks the better of two; RAG faithfulness checks claims against the retrieved text.G-Eval · faithfulness

Computed by code

Matching the keyExact match, F1, field accuracy, BLEU, ROUGE, BERTScore. Cheap and repeatable — and only as good as the reference.exact · overlap

Properties of the outputValid JSON, follows the schema, unit tests pass, latency, cost per answer.checks · ops

Needs a reference answerWorks without one

Metrics HELM reports for every model and scenario: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency.

Liang et al., HELM (2022)

17.9%

Share of HELM’s core scenarios a model had been evaluated on, on average, before HELM raised it to 96%.

Liang et al., HELM (2022)

Word-overlap metrics

BLEU, ROUGE and BERTScore: what overlap and similarity metrics reward.

BLEU was built in 2002 to score machine translation by shared word sequences; ROUGE, in 2004, to score summaries by recall of a reference’s words. Both reward saying it the same way as the reference. A language model often words a right answer differently, so both can mark it down — and can score a wrong answer well when it reuses the reference’s words.

scored too low

Right answer, low overlap

Reference: “The invoice is due on 14 March.” Output: “Payment deadline: March 14.” Almost no shared words; a person marks it correct.

Different words
Same meaning
Same date

scored too high

Wrong answer, high overlap

Reference: “The invoice is due on 14 March.” Output: “The invoice is not due on 14 March.” Nearly every word matches; the meaning is reversed.

Same words
Opposite meaning
One word added

LowBLEU and ROUGEHigh

Higher — the meanings are closeBERTScoreStill high

CorrectField rule or judgeWrong

What each one computes

BLEUPrecision of shared word sequences against one or more references. Built for translation.
ROUGERecall of the reference’s word sequences, or its longest common subsequence. Built for summaries.
BERTScoreSimilarity of contextual embeddings, token by token. Tolerates paraphrase; but metrics built on language-model embeddings are insensitive to negation (Anschütz et al., 2023), so a flipped fact can still score high.

Use them as a tripwire. OpenAI’s evaluation guidance lists relying solely on academic metrics like perplexity or BLEU as an anti-pattern. A sudden drop in overlap is worth a look; a stable score proves little.

Correctness and consistency

Task metrics for LLM evaluation: exact match, F1, pass@k and pass^k.

When an answer can be checked — a field, a label, code that must run — check it directly. Then measure the thing average accuracy hides: a model can be right on a case some of the time.

Metrics that check the answer itself

Exact matchOne right value: a code, a label, a yes or no. Normalise case and spacing first.
Precision, recall, F1Lists of items — entities, line items, citations: how many found were right, and how many right ones were found.
Field accuracyStructured extraction: a match rule per field, with the fields that must never be wrong reported on their own.
Unit tests passedGenerated code: run it. HumanEval scores functional correctness this way.
Outcome checksAgents: inspect the end state — was the booking made, the record updated — not what the agent said.

optimistic

pass@k — at least once in k tries

The HumanEval paper (Chen et al., 2021) made it the standard code metric. Useful when one success is enough, such as a developer picking one of several suggestions. With 75% per try, pass@3 is 1 − 0.25³ ≈ 98%.

what users feel

pass^k — every time in k tries

Introduced with τ-bench (Yao et al., 2024). Useful when a customer sees each run. With 75% per try, pass^3 is 0.75³ ≈ 42% — Anthropic’s worked example.

28.8% → 70.2%

HumanEval problems solved by Codex with one sample, and with 100 samples per problem — the gap pass@k measures.

Chen et al. (2021)

<25%

pass^8 on τ-bench retail tasks for state-of-the-art function-calling agents such as gpt-4o, which succeeded on under half of tasks even in a single trial.

Yao et al., τ-bench (2024)

Retrieval and grounding

RAG evaluation metrics: retrieval and generation, scored separately.

A retrieval-augmented answer can fail in two places: the wrong passages were fetched, or the right ones were ignored. Scoring them separately tells you which half to fix. The Ragas paper (Es et al., 2023) defined three reference-free scores — faithfulness, answer relevance and context relevance; the Ragas library now ships context precision and context recall for the retrieval side.

Question
What the person asked.
Context precision
Were the relevant passages ranked above the irrelevant ones? The Ragas library’s name; the paper’s nearest score, context relevance, penalises retrieved text the answer does not need.
retrieval
Context recall
Of the facts the answer needs, how many appear in what was retrieved? Always needs a reference to compare against.
retrieval
Faithfulness
Of the claims in the answer, how many are supported by the retrieved passages?
generation
Answer relevance
Does the answer address the question that was asked, without padding?
generation

The RAG answer was wrong. Which score dropped?

Context recallRetrieval missed the facts. Fix chunking, the index or the query before touching the prompt.

FaithfulnessThe facts were there and the model went beyond them. Tighten instructions, require citations, check claims.

NeitherFaithful to the right passages, yet wrong: the source itself is out of date or the question was ambiguous.

Model-graded metrics

Model-graded metrics: when to use an LLM judge, and how far to trust it.

Tone, completeness, whether a summary is faithful — no rule computes these. A judge model can, with a written rubric. G-Eval, which has the judge reason through criteria before filling in a score, reached a Spearman correlation of 0.514 with human ratings on summarization — better than earlier automatic metrics, and still far from perfect.

Narrow verdicts

Ask “is every claim supported — yes or no?” rather than for an open opinion. Anthropic’s grading advice is to have the judge output only “correct” or “incorrect”, or a short fixed scale, and to reason before it scores.

Reference when you have one

In a small test on 10 MT-Bench math questions, giving GPT-4 a reference answer cut how often it passed a wrong answer from 70% to 15% (Zheng et al., 2023).

Check the judge

Score a sample by hand and measure agreement before trusting the judge at scale. See LLM as a judge.

0.514

Spearman correlation of G-Eval (GPT-4) with human judgements on summarization; the authors also flag a bias toward LLM-written text.

Liu et al., G-Eval (2023)

Decision guide

How to choose LLM evaluation metrics for your kind of output.

Start from what a correct answer looks like to the person who receives it. Then add the metrics that say whether it stays correct — and what it costs.

What does your system produce?

Fields or labelsField accuracy with a match rule per field, critical-field accuracy, precision and recall per label, and calibration of confidence.

CodeUnit tests passed, pass@k when a person picks from options, pass^k when code runs unattended.

Answers from documentsContext recall and precision for retrieval, faithfulness and key-fact coverage for the answer.

Actions by an agentOutcome checks on the end state, pass^k over repeated runs, tool errors, and cost and time per task.

Report alongside any quality score

The dataset versionA score is only comparable with scores on the same cases.
The spreadHow many cases, and the margin of error that implies.
Consistencypass^k or the share of cases that flip between runs.
Cost and timePer case — a better score at triple the cost is a trade-off, and someone must choose it.

On Botminds

The LLM evaluation metrics Botminds reports on every scorecard.

Botminds scores agents, workflows and meshes on your golden datasets, with the metrics that show whether an agent is right, right every time and still right after a change — and puts cost beside quality.

Field accuracy

Per field and overall, with precision, recall and F1 — so you know which field is weak.

Critical-field accuracy

The fields that must never be wrong, on their own line.

pass^k

A case counts when every repeated run passes — the number a customer feels.

Shape and spend

Schema breaks, time per case, and model cost per case for agents.

Judges where text must be readYes-or-no verdicts on key facts and grounding, with the judge named on the scorecard.
Calibrated confidenceEach field’s confidence fitted to reviewed outcomes, so a stated 90% tracks how often it is right.
Compared with a baselineEach run compared case by case with a pinned baseline before a change ships.

Read the full design in AI agent evaluation, and try it in the Academy lesson crew and evaluations.

Questions people ask

LLM evaluation metrics: common questions.

The technology behind itEvaluation & observabilityEvery change measured, every run watched

What are the most important LLM evaluation metrics?