You wouldn't hire a team off one interview question

The AI industry has trained buyers to ask one question: how accurate is the model? It’s a fine question — for a demo. In production, it’s the wrong unit of measurement entirely, and I want to explain why we spent an engineering cycle building the machinery that measures the right one.

Here’s the thing about real document operations: no single agent runs them. A borrower file moves through a crew — an extraction agent reads the statements, a normalization agent maps them to your template, a drafting agent writes the memo, an approval flow routes exceptions to a human. Four specialists, three handoffs. And after watching thousands of these runs, I can tell you where production failures actually live: in the handoffs. The extractor was right. The normalizer was right, given what it received. And the final answer was still wrong — because “right, given what it received” is exactly how teams fail.

Single-agent metrics are blind to this. It’s like hiring a whole team by asking each person one interview question, separately, and never watching them work together.

So we built evaluations for crews

Crew Evaluations shipped this cycle, and the design principle was simple: measure the unit you actually deploy.

You define golden cases — real documents with known-correct outcomes, your hardest ones, the scanned tax return with the creative Excel sidecar. Then you run the whole crew against them: individual agents, orchestrated workflows, meshes of cooperating agents. The platform scores the end-to-end result and every intermediate step. So the finding is never just “the memo was wrong.” It’s “extraction was right, normalization dropped the minority-interest line on multi-entity statements.” That’s not a report card — that’s a repair manual.

And evaluations stream live. You watch the crew work, agent by agent, artifact by artifact, every number appearing with its citation. I’ve sat in rooms where this changed the conversation in real time: an operations leader stops asking “do we trust the AI?” — which is unanswerable — and starts asking “which step do we tune next?” — which is engineering.

Promotion is earned, not assumed

The part I care most about as CTO: the gate. Crews that clear their evaluation bar get promoted into production. Changes that regress get stopped — before a customer, before an underwriter, before an auditor. Software engineering learned this discipline decades ago; nobody ships to production on “the demo looked good.” Your AI workforce deserves the same rigor, and now it has it. It’s also the only honest way to keep saying “battle-proven” release after release — proof has to be re-earned every time something changes.

The question to take away

If you’re deploying AI on work that matters, stop asking vendors how accurate their model is. Ask them: how do you evaluate the team, where do failures between agents show up, and what stands between a regression and my production queue? If the answer is a benchmark table, keep looking.

If you want to see a crew evaluated against your own documents — messy ones welcome, that’s the point — request a demo. Crew Evaluations run under the same governance core as everything else we ship: citations, approvals, audit trail, all the way down to financial spreading.