botminds.aiRequest a demo
Guide · AI agent evaluation

AI agent observability vs LLM observability.

LLM observability watches each call to a model. Agent observability watches the whole run — the plan, the tools, the hand-offs, and whether the job got done.

A single agent request can make a dozen model calls, run five tools, consult another agent and wait for a person. Monitoring each model call tells you what was spent and how fast it came back. It cannot tell you why the task failed. This guide shows what each kind of observability records, how an agent trace is built from OpenTelemetry’s GenAI spans, which failures only show up at the agent level, and the signals worth an alert.

Updated · 6 sources

Read the guideHow Botminds does it
one model call

LLM observability

Records each request to a model and its response.

  • Prompt and completion
  • Tokens, cost, latency
  • Errors and rate limits
  • Model and version
one whole task

AI agent observability

Records the run an agent makes to finish a task, as a tree of steps.

  • Plan and each decision
  • Tool calls and their results
  • Hand-offs to other agents and people
  • Outcome, cost and time per task

What is the difference between AI agent observability and LLM observability?

LLM observability records individual calls to a language model — the prompt, the response, tokens, cost, latency and errors. AI agent observability records the whole run an agent makes to complete a task: its plan, every model call and tool call nested under it, retrievals, hand-offs to other agents or people, and the final outcome. LLM observability answers “what did this call cost and return?”; AI agent observability answers “why did this task succeed or fail, and what did it cost end to end?”

Anatomy

Anatomy of an AI agent trace, span by span.

A trace is a tree. The root is the task; under it sit the agent’s plan, each model call, each tool call and any agent it called in turn. Model calls are the leaves — which is why watching them alone misses the shape of the run.

One agent run as a trace (illustrative)
invoke_workflow  claims-intake                 41.2 s
├ invoke_agent   claims-agent                  39.8 s
│ ├ plan                                        2.1 s
│ │ └ chat       model A    1,840 → 212 tok    2.0 s
│ ├ execute_tool read_attachments               6.4 s
│ ├ chat         model A    9,310 → 640 tok    4.9 s
│ ├ execute_tool policy_lookup        ERROR    30.0 s
│ ├ execute_tool policy_lookup                  1.2 s
│ ├ retrieval    claims-guidelines   8 docs     0.4 s
│ └ chat         model A   11,020 → 388 tok    3.8 s
└ invoke_agent   notify-agent                   1.3 s

What the tree shows that calls alone cannot

  • The slow stepA tool timed out for 30 seconds; the model calls were quick.
  • The retryThe same tool was called twice — once failing, once succeeding.
  • The context growthInput tokens climb from 1,840 to 11,020 as results pile up.
  • The cost of the taskEvery call summed under one root: the number a business case uses.
The span types, from task down to call
01
Workflowa coordinated process — a graph or orchestrator — that runs several agents or model calls
invoke_workflow
02
Agentone agent working on one task, with its name and conversation
invoke_agentcreate_agent
03
Planthe agent decides how to proceed; the model call that plans sits inside it
plan
04
Tool and retrievaleach tool run and each search, with arguments and results
execute_toolretrieval
05
Model callone request to a model: model, tokens, finish reason
chatembeddings
The standard

OpenTelemetry’s GenAI conventions give AI agent observability shared span, attribute and metric names.

OpenTelemetry’s generative AI semantic conventions give every tool the same names for these spans, attributes and metrics, so a trace from one framework reads the same in any backend. They are still marked as in development, and they already cover agents, tools and workflows alongside model calls.

Attributes on the spans

  • Operationgen_ai.operation.name — what the span is: chat, execute_tool, invoke_agent…
  • Modelgen_ai.request.model — the model asked for; the provider in gen_ai.provider.name.
  • Tokensgen_ai.usage.input_tokens and output_tokens — the basis of cost.
  • Agentgen_ai.agent.name — which agent ran; gen_ai.conversation.id joins turns.
  • Toolgen_ai.tool.name — which tool ran, with its call id, arguments and result.

Metrics

  • Call timegen_ai.client.operation.duration — how long each model operation took.
  • First tokengen_ai.server.time_to_first_token — the wait before output starts.
  • Agent run timegen_ai.invoke_agent.duration — how long a whole agent run took.
  • Calls per rungen_ai.invoke_agent.tool_calls and inference_calls — tool and model calls per run.
  • Tool timegen_ai.execute_tool.duration — how long each tool took.
  • Workflow timegen_ai.invoke_workflow.duration — how long a whole multi-agent workflow took.

Content is opt-in, for a reason. Under the conventions, prompts, responses, system instructions and tool definitions are opt-in attributes, flagged as likely to hold sensitive or personal data. Decide what to capture, where it is stored and who can read it before you switch it on.

Signals

LLM observability vs AI agent observability, signal by signal.

The two overlap on cost and latency. They part ways on the questions a business asks: did the task finish, was it right, and what did it cost to get there?

SignalLLM observabilityAI agent observability
Unit of recordOne model requestOne task, as a tree of spans
CostPer call, per modelPer task, summed across every call and tool
LatencyTime to first token, call durationTime to finish the task, and the slowest step in it
ErrorsTimeouts, rate limits, refusalsTool failures, retries, loops, abandoned plans
QualityOutput scored per responseOutcome checked: was the record updated, the case closed?
PeopleOutside its viewWaits for approval, hand-offs, overrides
Typical questionWhy is this model slow today?Why did 6% of claims stall at the policy check?
Failure modes

Six agent failures that only AI agent observability can see.

Each of these leaves every individual model call looking healthy. They show up only when calls are read as one run.

01tool calls per run

Loops

The agent calls the same tool with the same arguments again and again.

02tool arguments

Wrong tool, right answer shape

A plausible tool with wrong arguments returns valid but irrelevant data.

03error then success

Swallowed errors

A tool fails, the agent carries on, and the answer is built on nothing.

04input tokens per step

Context bloat

Tool results pile into the prompt until cost climbs and quality falls.

05hand-off spans

Lost hand-off

A sub-agent or a person receives the task without what the first agent learned.

06outcome check

Said done, was not

The agent reports success but the system of record never changed — why Anthropic’s eval guidance grades the final state, not the claim.

Alerts worth setting first
  • Tool calls per runFar above the usual count for that agent signals a loop.
  • Cost per taskAgainst the same day last week, per agent — not per model.
  • Task completion rateRuns that ended without an outcome, by step.
  • Tool error rateBy tool, so one failing system is visible at once.
  • Wait for peopleItems waiting for approval longer than the service level.
From traces to tests

Traces feed evaluation: each production failure becomes a test case.

OpenTelemetry’s own guidance notes that, because agents are non-deterministic, their telemetry also serves as a feedback loop. Traces show what happened in production; evaluation decides whether it was right; each failure becomes a test.

Production to test and back
  1. TraceRecord every run as a tree of spans, with cost and outcome.
  2. Sample and scoreScore a sample of live runs with rules and judges.
  3. Read the failuresRead the traces of failed runs, step by step.
  4. Add to the datasetEach real failure becomes a case in the golden dataset.
  5. Fix and compareChange the agent, and compare with the baseline before release.

every fix ships against the grown dataset, and production is watched again

Which do you need first?

One model behind a chat boxLLM observability: tokens, cost, latency and errors per call will answer most questions.
Agents that call toolsAI agent observability: traces per task, tool spans, outcome checks and cost per task.
Workflows across teamsBoth, joined: workflow and agent spans on top, model calls underneath, and an owner for each alert.
On Botminds

AI agent observability on Botminds: every request traced, every failure routed to someone who can act.

On Botminds every request is recorded, every model call carries its cost, and every failure is kept — then turned into a short list of things a person should do.

Observability on Botmindsevery run watched, in plain words
Every request recordedtraced end to end, grouped by session
Cost per model callevery call carries what it cost
Failures by causeexceptions grouped, with samples
Action Centermore than sixty checks turn signals into items: what is wrong, who should act
Ask Piper“what needs me today?” answered in plain words
Queues and driftjobs, queues, and inputs compared with a reference window
Audited outcomesfields that skip review keep being sampled
Joined to evaluationscheduled runs against golden datasets
Asking Piper in plain words
You

Why is claims spend up?

Agent

One agent, Claims summariser, moved to a larger model yesterday at 16:05; each call costs about four times more. Volume is normal. Shall I show the change and who made it?

Read the full design in AI agent evaluation and observability, and the Academy lesson on observability.

Questions people ask

AI agent vs LLM observability: common questions.

The technology behind itEvaluation & observabilityEvery change measured, every run watched
What is AI agent observability?

AI agent observability is recording and analysing what an AI agent does to complete a task — its plan, each model call, each tool call, retrievals, hand-offs to other agents or people, and the final outcome — as one trace, so you can see why a task succeeded or failed and what it cost.

What is LLM observability?

Monitoring individual calls to a language model: the prompt and response, tokens, cost, latency, errors and the model version, often with a quality score on each response.

What is the difference between LLM observability and agent observability?

LLM observability is per call; agent observability is per task. An agent run contains many model calls plus tools and hand-offs, and most agent failures — loops, swallowed tool errors, reporting success without doing the work — are invisible when calls are viewed one at a time.

Does OpenTelemetry support AI agents?

Yes. Its generative AI semantic conventions define spans for creating and invoking agents, running tools, planning and invoking workflows, alongside model inference, embeddings, retrieval and memory operations. The conventions are marked as in development.

What should you monitor in an AI agent?

Task completion rate, cost and time per task, tool calls per run, tool error rates, input tokens per step, waits for human approval, and an outcome check on the system of record — plus sampled quality scores.

Should prompts and responses be stored in traces?

Only by decision. The OpenTelemetry conventions make message content opt-in because it can hold sensitive or personal data. Decide what is captured, where it is stored and who may read it.

How is observability different from evaluation?

Evaluation tests a system against known cases before and after changes. Observability watches real traffic. They work as one loop: failures seen in traces become test cases, and evaluations check the fix.

Sources
  1. OpenTelemetry: GenAI agent and framework spansgithub.com
  2. OpenTelemetry: GenAI client spansgithub.com
  3. OpenTelemetry: GenAI metricsgithub.com
  4. OpenTelemetry blog: AI Agent Observability – Evolving Standards and Best Practicesopentelemetry.io
  5. Anthropic: Demystifying evals for AI agents (2026)anthropic.com
  6. Anthropic: Building effective agents (2024)anthropic.com
Read next
LLM as a judgeHow model graders work, the biases research found, and how to calibrate oneWhat is a golden dataset?The answer key for testing AI agents: what goes in it and how to build itLLM evaluation metricsBLEU to pass^k: what each metric measures and which one fits your taskAI audit trailWhat to record for every AI decision, how to seal it, how long to keep itAll guidesEvery explainer, by topic

See it on
your own work.

Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.

Book a demoEvaluation & observability