Crew & evaluations

In one line. Watch your agents work in real time on the collection’s Crew tab, and prove they still work — against curated ground truth, on a schedule — with Evaluations. You’ll be able to. Read the live crew view during a run, build a golden dataset, run an evaluation against any crew (agent, XFlow or mesh), and use baselines to catch regressions before your users do.

Crew — the live view

A crew is the platform’s umbrella word for whatever workforce is attached to a collection: a single agent, an XFlow, or a mesh. The Crew tab on the collection workspace merges “what is this collection?” and “who works it, right now?” into one screen:

Identity strip — the collection’s title, a live Ready / Crew working status pill, and one row of vitals: documents, fields, stages, intake. A Manage in Studio link jumps to configuration.
Pipeline rail — the collection’s stages as numbered nodes, tinting while work is in flight.
The live crew — KPI tiles, the crew stage with each member lighting up as it works, recent decisions, and recent runs.

The Crew tab ships behind the crewTab release flag, default on; when off, the previous Overview and Agents tabs render unchanged.

This is the screen to keep open during a demo or a go-live: documents land, lanes light left to right, decisions accumulate — and when something stalls, you can see which member stalled instead of inferring it from a silent queue.

The Crew tab also hosts the per-document step-log timeline: click a document row and every ingestion step appears in order — including each retry attempt as its own row, attempt-numbered and carrying the full per-attempt error, so an admin can read “failed 1/3 → retrying → failed after 3” and tell a transient blip from a broken model config. A Knowledge collection’s Index Health View logs link jumps straight here.

Evaluations — the proof

The live view tells you the crew is working. Evaluations tell you it is working correctly — and still working correctly after every prompt tweak, model upgrade, or upstream change.

Studio ▸ Agent Builder ▸ Evaluations

The system-under-test is any crew: a single agent, an XFlow, or a mesh. The loop:

Golden dataset. Curate documents whose correct outputs you know — drawn from human-verified annotations and reviewed corrections, not generated by a model. This is the asset that makes every later step honest.
Evaluation run. The crew re-processes the golden documents as if they were new; outputs are scored against the ground truth with hybrid metrics — deterministic checks (field accuracy, classification metrics, set precision/recall) plus LLM-judge scoring for narrative outputs (faithfulness, coverage), with the judge running on a different model family than the crew it judges.
Baseline & regression. A run can be pinned as the baseline; later runs compare against it, so “did last week’s prompt change hurt extraction?” has a measured answer instead of an argument.
Schedule it. Evaluations run on a schedule, which turns them from a release ritual into a canary: quality drops show up in the report, not in a customer escalation.

What each word means here

Word	Meaning	Caught by
Accuracy	Output matches curated ground truth	Field and document scoring, classification metrics
Inconsistency	Same input produces different outputs run to run	Stability scoring over repeated runs
Drift	Quality or data distributions move over time on live traffic	Scheduled runs and monitoring over live telemetry

The habit that makes this pay

Treat the golden dataset like code: it grows with every incident. When a reviewer catches a wrong extraction, that document — with its corrected values — belongs in the dataset, so the mistake can never come back silently. Twenty well-chosen documents that cover your ugly cases beat two hundred easy ones.

Run an evaluation before every meaningful change ships: persona edits, model swaps, schema changes, new pipeline steps. The comparison against baseline is your regression gate.

Where to go next

The workforce being watched: A11 · Teams & mesh.
Where reviewer corrections come from: G2 · Human-in-the-loop.
The runs and logs underneath: G4 · Observability.