01
Same input, different output
Run an agent twice and you may get two answers. Repeated runs show whether it is reliably right.
Evaluation & observability
AI agent evaluation: measured before it ships, watched after.
An agent you have not measured is a guess. One you stop watching drifts.
Botminds tests any agent, workflow or mesh against golden datasets of thousands of cases, runs them side by side with repeats to catch inconsistency, flags a change that makes things worse, fits each field’s confidence to reviewed outcomes so a stated 90% tracks how often it is right, watches every live run for cost and failures, tracks audited outcomes continuously, and reports input drift against a reference window.
1Golden datasetdocuments with known answers, versioned
2Publish an agentthe change is saved; its evaluations start
3Run, k timesthe agent, workflow or mesh
4Scorefield rules first; a judge where text must be read
5Comparedocument by document, against the baseline
6Gatea worse change is flagged, with the reason
7Watchtraces, cost, exceptions, audited outcomes
8Learnadd each failure found live to the dataset
Every change, measuredwhen it ships, and after
What is AI agent evaluation?
AI agent evaluation is measuring whether an agent does its job correctly and consistently — before a change goes live and while it runs. It means running the agent on a golden dataset of cases with known answers, scoring each output with exact rules and, where text must be read, an LLM judge, repeating runs to test consistency, and comparing with a baseline. Live observability then shows what tests cannot: real traffic, real cost, real failures.
Scales out on KubernetesServices and workers add capacity as work arrives, and give it back when it is done.
Built for millionsMillions of documents, records and agent runs — the indexes and workers grow with them.
Any cloud, any regionAzure, AWS, Google Cloud or on-premises, in the region your data must stay in.
Always onRedundant services, rolling releases and continuous backups, for mission-critical work.
Secure and governedSingle sign-on, encryption, least privilege and a sealed audit trail.
Why it is different
Why AI agent evaluation is not like testing software.
A unit test asks one question with one right answer and gets the same result every time. An agent breaks all three assumptions, so evaluation is built for that.
02
Many right answers
“Acme Corp.” and “ACME Corporation” are one supplier. Scoring needs a rule per field, not string equality.
03
Noise looks like change
A five-point drop and run-to-run noise look alike. A verdict needs a confidence interval.
04
The world moves
Suppliers change layouts; model versions change. An agent that passed in March can drift by June.
LLM evaluation vs AI agent evaluation. LLM evaluation scores a model on general benchmarks. AI agent evaluation scores your agent — instructions, tools, data and model together — on your cases. The first helps you pick a model; the second tells you if your process works.
Golden datasets
Golden datasets: the answer key for your agents.
A golden dataset is a set of real cases with the answers a careful person would give. Everything else depends on it, so it is built like an asset: versioned, reviewed, and shaped to the job.
record: invoice-0142 review: verified
documents: [invoice-0142.pdf]
expected:
supplier "Northwind Traders" fuzzy ≥ 0.85
invoice_date 2026-03-14 date
total 18,240.50 numeric ± 0.5% critical
currency "EUR" exact critical
po_number "PO-77310" exact
line_items [12 rows] listFour kinds of case
Documents with expected fields, packages with cross-document rules, questions with answers, and conversation scenarios.
A rule per field
Exact, fuzzy, numeric with a tolerance, date or list — a right answer written differently still scores right.
Critical fields
The fields that must never be wrong weigh double and get their own accuracy.
Versioned and reviewed
Every edit is a new version; each case is verified before it becomes the answer key.
Evaluation metrics
Evaluation metrics that predict how an agent behaves in production.
An average accuracy hides the two things customers notice: the field that is wrong when it matters, and the case that is right only some of the time. The scorecard reports both.
Run 1Run 2Run 3All three?
Invoice 1✓✓✓yes
Invoice 2✓✕✓sometimes
Invoice 3✓✓✓yes
Invoice 4✕✕✕no
Invoice 5✓✓✓yes
Average pass rate73%11 passes out of 15 runs — looks fine
pass^360%3 of 5 invoices right every time — the number a customer feels
Field accuracy
Per field and overall, with precision, recall and F1 — so you know which field is weak.
Critical-field accuracy
Accuracy on the fields that must never be wrong, on its own line.
pass^k
A case counts when every repeated run passes — the number a customer feels.
Shape and spend
Schema breaks, plus time per case, and model cost per case for agents: a better score at triple the cost is a trade-off.
LLM as a judge
LLM as a judge — where rules cannot read, and kept honest.
Is this summary faithful, does this answer cover the key facts, is this reply grounded in the source? An LLM judge can read them. It can also be lenient and inconsistent, so it works inside firm rules.
- Rules first
Every field a rule can score is scored by the rule.
- Yes or no
The judge returns a binary verdict — supported or not — which holds steadier than a score out of ten.
- Your choice of judge
Pick one from a different model family than the agent, so it never marks its own homework.
- Named on the card
The scorecard says which judge was used, and shows a fallback as a fallback.
Key facts
Does the answer state each fact the case expects?
Grounding
Is each claim supported by the source documents?
Conversations
For assistants, checked by rules on every turn: did it say what it must, avoid what it must not, and ask before changing anything?
Baselines and publish gates
Baselines and publish gates: flag real drops, not noise.
A gate that fails a change on every dip gets switched off within a week; one that never fails a change is decoration. The Botminds gate compares a change with a pinned baseline, case by case, and flags what the evidence supports.
Change against the baselineworse0betterVerdict
A real dropthe whole interval sits below zero
regressionmarked failing, with the reason, when the gate is set to enforce
Too close to callthe interval crosses zero — more cases or repeats sharpen it
inconclusivethe floor decides
Holds or improvesthe interval is at or above zero
passallowed
And the floor. A comparison says “no worse than before”. The floor says “good enough”: a score under the pass mark you set fails on its own, so an agent that was always weak is caught too.
Pin a baseline
Pick a good run as the baseline. Every later run is compared with it, case by case.
Three gate settings
Off; advisory, which reports; and enforce, which marks a worse change as failing and says why.
Know your precision
Each run reports the smallest change it could detect, so you know when more cases will sharpen it.
An agent evaluation framework
An agent evaluation framework that runs itself.
An automation ties a target, a dataset, a judge, repeats and a gate together, and runs on every publish or on a schedule — so evaluation happens without anyone remembering to start it.
target agent: invoice-extractor
dataset invoices-golden (version 7)
judge your choice — a different model family
repeats k = 3
schedule on every publish
gate enforce · floor 0.90Any target
An agent, a workflow or a whole mesh, scored on the same dataset shapes.
On publish
Publishing an agent saves the change and starts its evaluations; an enforced gate flags a worse change with the reason.
On a schedule
Nightly or weekly runs catch drift when nothing in your project changed but the world did.
Complete runs, compared
A run that stopped early is marked, so it is compared as what it is.
Calibrated confidence
Calibrated confidence: a stated 90% tracks how often it is right.
Every extracted field carries a confidence. Raw model confidence is usually too high. Botminds fits a calibration curve per field from reviewed outcomes, and then lets confidence send work straight through at a precision you set.
The field saysHow often it was right
90–100%
80–90%
70–80%
60–70%
50–60%
raw confidencecalibratedwhere it should bean illustration, not a measurement
A curve per field
A simple curve while outcomes are few; a finer one as they grow.
Kept when it helps
A curve is accepted when, on outcomes it was not fitted on, calibration error falls and overall error holds.
- Declare
Set the precision you need: this field may skip review at 99%.
- Sample
A neutral audit sample of straight-through work is still reviewed.
- Certify
The platform finds the confidence bar whose precision clears 99% with 95% confidence.
- Go straight through
Above the bar, the field skips review; below it, a person checks.
- Trip on drift
If audited errors climb, the field returns to review on its own.
AI agent observability
AI agent observability: traces, cost, failures and drift, live.
Tests cover the cases you thought of; production brings the rest. LLM observability on Botminds records every request, every model call’s cost and every failure — and the Action Center turns them into a short list of things a person should do.
- Tracesevery request end to end, grouped by session
- Exceptionsgrouped by cause, with samples
- Model spendtokens and cost by day, workspace and model
- Database workthe slowest and busiest queries, by caller
- Page budgetseach page’s speed against its budget
- Jobs and runsscheduled work, agent runs, meshes
- Qualityevaluation scores over time, and drift
Action Center60+ checksturn signals into items: what is wrong, how bad, who should act, the next stepnew checks learn quietly for a week, so a noisy rule is tuned before it alerts
- Operatorsan item that says what is wrong and what to do next
- Project ownersfailures that are their input, code or settings
- Piper“what needs me today?” — answered, then acted on with a yes
Asking Piper in plain words
You
What needs me today?
Agent
Three items. Invoice intake has 41 documents waiting at extraction since 10:20 — the model provider’s rate limit, not your settings. Model spend for the claims project is 2.3× its usual Tuesday. And a reviewer queue in onboarding has had nobody claim it for six hours.
You
Why is claims spend up?
Agent
One agent, Claims summariser, moved to a larger model yesterday at 16:05; each call costs about four times more. Volume is normal. Shall I show the change and who made it?
Evaluation at scale
AI agent evaluation at scale: every change measured, every run watched.
Evaluation on Botminds runs at enterprise scale: golden datasets of thousands of cases, run side by side and repeated, on every publish of an agent and nightly, weekly or on demand for workflows and meshes — and continuous monitoring of production on the same scaled-out platform, on Azure, AWS, Google Cloud or on-premises.
Golden datasetsThousands of casesversioned and reviewed; add each failure found live
Run side by side · each repeatedcases fan out in parallel; every case is run k times, so consistency is measured, not assumed
Score and compare
- field rules first
- a judge where text must be read
- pass^k, case by case
- against the baseline
Every change, measured
- Agents: on every publish
- Nightly
- Weekly
- On demand
In production
- Traces
- Cost
- Exceptions
- Audited outcomes
- Quality over time
- Input drift vs a reference window
Thousands of cases
Add each failure found live to the dataset, so the answer key keeps pace with the work.
Parallel and repeated
Cases run side by side; each is repeated to measure consistency.
Every change measured
Set an agent’s evaluations to run on every publish; workflows and meshes are measured nightly, weekly or on demand against their baselines.
Production, all the time
Traces, cost, exceptions and audited outcomes are watched continuously, and input drift is reported against a reference window.
Side by side
AI agent evaluation vs LLM benchmarks vs observability tools.
Three kinds of tool get called “evaluation”. They answer different questions, and an enterprise running agents needs all three answers, joined up.
| LLM benchmark | Observability tool | Botminds evaluation & observability | |
|---|---|---|---|
| Question it answers | Which model is smarter in general? | What happened in production? | Is my agent right, consistently, and still right today? |
| Test cases | Public datasets | None | Your golden datasets, versioned and reviewed |
| Consistency | One score | Outside its scope | pass^k over repeated runs |
| Is a change worse? | Outside its scope | After the fact | Case-by-case comparison with a baseline: on publish for agents, on a schedule for workflows and meshes |
| Confidence | Outside its scope | Outside its scope | Calibrated per field; straight-through at a certified precision |
| Live failures and cost | — | Yes | Yes, with causes, owners and Piper |
| Setup | Download a leaderboard | Instrument your code | Built into every agent, workflow and mesh |
Measured, or it did not happen. Every number on these pages marked measured names the record it came from — see the scale results in agent mesh. The same rule applies to your agents.
Questions people ask
Evaluation & observability, answered.
More from the platform: the full architecture, and how security works at Botminds.
What is AI agent evaluation?
AI agent evaluation measures whether an agent does its job correctly and consistently, before a change goes live and while it runs. It uses golden datasets with known answers, rule-based scoring and LLM judges, repeated runs, and comparison against a baseline.
How do you evaluate an AI agent?
Build a golden dataset of real cases with verified answers; run the agent on it several times; score each field with the right match rule and use a judge where text must be read; compare with a pinned baseline; and set a gate that flags a real drop or a score under your floor on every publish.
What is a golden dataset for LLM evaluation?
A golden dataset is a versioned set of real inputs with the answers a careful person would give, used as the answer key. Good ones cover common formats and known edge cases, mark the fields that must never be wrong, and are reviewed before use.
What is the difference between a golden dataset and ground truth?
Ground truth is the true answer for one case. A golden dataset is a curated, frozen collection of such cases, chosen to test what matters and versioned so that two runs are compared on exactly the same questions.
Is LLM as a judge reliable?
It is far steadier inside firm rules: Botminds scores everything a rule can score with the rule, asks the judge for yes-or-no verdicts, lets you choose a judge from a different model family than the agent, and names the judge on every scorecard.
What is pass^k?
pass^k is the share of test cases an agent gets right on every one of k repeated runs. Unlike average accuracy, it exposes cases that are right only some of the time — the ones behind “it worked yesterday” complaints.
What are the most important AI agent evaluation metrics?
Field accuracy with precision and recall, accuracy on critical fields, pass^k for consistency, schema breaks, time per case, and model cost per case for agents. For assistants: whether replies cover key facts, stay grounded and ask before changing anything.
How does AI agent evaluation scale?
On Botminds, golden datasets grow to thousands of cases, cases run in parallel with repeats, and you set an agent’s evaluations to run on every publish; workflows and meshes are measured nightly, weekly or on demand against their baselines — while production runs are monitored continuously for cost, failures and audited outcomes.
What is AI agent observability?
Seeing what agents do in production: a trace of each request, the cost of each model call, failures grouped by cause, queues and jobs, and drift in inputs and quality. On Botminds it feeds an Action Center that says who should act, and Piper answers questions about it in plain words.
How do you detect drift in an AI agent?
Run evaluations on a schedule even when nothing changed, compare input distributions with a reference window, and watch audited outcomes on fields that skip review. On Botminds, a climb in audited errors sends a straight-through field back to review automatically.
Bring your hardest documents. We’ll show you the score.
Give us real cases. We’ll build the golden dataset, run your agent on it with repeats, and show you the scorecard, the weak fields and the calibrated confidence.