How to measure the accuracy of an AI funding review, field by field

How accurate is AI document review, really, when it is reading loan documents for an equipment-finance funding package? The honest answer is that "accurate" is not one number, and a single percentage borrowed from a general study of a different kind of document tells a funding desk almost nothing about the package it is about to release money against. Equipment Finance Pre-Funding Review's six agents read a complete package and test it against 22 named conditions, and the useful question is not "how accurate is it" but four narrower ones: can it read the page, does what it reads agree across documents, does it ever use a fact it cannot point to, and does it ever state something confidently that the source does not actually support.

The figures below come from a fictional sample package — Tamarack Earthworks LLC financing $1,207,523 for two excavators, a wheel loader and a titled trailer — used here as a demonstration, not a customer result. The published research cited alongside it studies different documents entirely, and the gap between the two is as much the point of this guide as either number is.

"Accurate" is not one number

A vendor who answers "how accurate is your AI" with a single percentage is usually quoting a number measured on a different task, a different document type, or a different definition of "correct" than the one a funding desk needs. A model that is 95% accurate at classifying a document's type and a model that is 95% accurate at reading a serial number off a schedule have nothing in common except the number. The four measures below are deliberately narrow, each testable on a desk's own files rather than trusted on a vendor's word.

Read rate: can it read the page at all

The most basic measure, and the one most often skipped, is whether the system can read the document in front of it. A scanned schedule with a faded stamp, a photographed plate at an angle, a signature page split across two images — any of these can make a field unreadable through no fault of the extraction logic. The package reader in Equipment Finance Pre-Funding Review is built to leave a field blank rather than guess at it, so a low read rate shows up as missing fields on the checklist, not as wrong ones.

In the sample package, the package reader read all 214 fields and all eleven signatures across sixteen documents — nothing unreadable. That is a read rate of 100% on one package, not a general claim about every scan a desk will ever receive; a desk evaluating a vendor should ask for the read rate on its own worst scans, not its cleanest ones.

Match rate: does the serial on the schedule match the plate

Match rate is where a funding package actually gets caught doing something wrong. It asks, field by field, whether the same fact reads the same way everywhere it appears — the clearest version being a machine's serial number, which should read identically on the plate photo, the invoice, the schedule, the delivery certificate, the title and the UCC-1. In the sample package, the equipment matcher made 25 reads of four serial numbers across seven documents and found three breaks: all three traced to one machine's serial, transposed on the equipment schedule and nowhere else.

Equipment match: one unit's serial, across every document that names it

A match rate measured this way answers a sharper question than "did the AI get it right": it answers "when two documents disagreed, did the system notice." A system that never flags a disagreement either has an unusually clean set of documents, or it is not actually comparing them.

Confabulation rate: a federal definition for the failure that matters most

The National Institute of Standards and Technology's Generative AI Profile names the failure that makes raw accuracy numbers dangerous on their own: confabulation, "a phenomenon in which GAI systems generate and confidently present erroneous or false content in response to prompts" (NIST AI 600-1). The same document explains why it keeps happening: "confabulations are a natural result of the way generative models are designed" — they predict plausible text, and plausible text is not the same thing as text grounded in a specific document.

Confabulation rate asks how often a system states something with confidence that the cited source does not actually support — the worst outcome on the list, because it looks identical to a correct answer until someone checks the citation. In Equipment Finance Pre-Funding Review the design answer is structural rather than statistical: every value the package reader relies on carries a page and position citation, so "confident but uncited" is not a state the checklist can be in. The measure to ask a vendor for is not "what is your confabulation rate," which most will not have measured, but "show me the citation behind this field" — on the specific package in front of you, not a demo.

What published LLM extraction benchmarks show, and don't show, about this work

Published research on LLM document extraction gives a sense of scale, and a clear warning about transferring it. A 2025 study extracting structured data from government fiscal documents — not loan files — reported that across seven volumes of a state budget, the pipeline's numerical-consistency checks passed 1,952 of 2,311 comparisons, about 84%, and that its structural-soundness measure, Tree Edit Distance Similarity, showed the extracted hierarchy "achieve[d] an accuracy of 73% to 96%" depending on the volume (arXiv 2511.10659, not peer reviewed). A separate 2026 benchmark built specifically to test whether LLMs can find errors in financial documents, using more than 900 documents reported in 2025, across three levels of difficulty, found that "current LLMs still struggle with this task, especially in high-complexity cases" (arXiv 2608.12342, not peer reviewed).

Neither study is about loan or lease documents, and neither should be read as "this is how accurate an equipment-finance review is." What they show is that even a narrow, well-scoped extraction task on structured government filings lands well short of 100%, and that error-detection specifically — not just extraction — is still a task language models visibly struggle with. That is the case for measuring a specific review on a specific desk's own documents, rather than borrowing a number from an unrelated study.

Why bank model-risk guidance doesn't yet reach agentic AI

A funding desk's model-risk or QA function may reach for bank model-risk guidance to frame the question, and should know where it stops. The Federal Reserve's current letter is explicit that its guidance "is expected to be most relevant to banking organizations with over $30 billion in total assets" (SR 26-2), and separately, that "generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance" (SR 26-2). A funding desk cannot point to this letter and say an agentic review has been validated against it, because the letter says plainly that it does not yet cover agentic systems.

That gap does not mean no standard applies — it means the standard has to come from the desk's own measurement, using the four rates above, rather than from a framework built for traditional statistical models.

Outcomes analysis: back-testing on your own closed packages

The part of bank model-risk guidance that does transfer cleanly, even though the guidance itself does not yet cover agentic AI, is the idea of outcomes analysis: "compar[ing] model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use" (SR 26-2). Applied to a funding review, that means running the six-agent review against packages that are already closed, already signed, and whose real exceptions are already known — and comparing what the review finds against what the desk's own people already found.

Serial lookup: checking a machine against the book and the public record, not just the package

The same Federal Reserve letter notes that "validation generally occurs prior to a model's first use" (SR 26-2) — the reason a pilot on already-signed packages belongs before go-live, not after. A desk that replays last year's closed deals and finds the review agrees with its own people on the overwhelming majority, and disagrees only on cases the desk's own file notes already flagged as judgment calls, has a real outcomes-analysis result. A desk that has only seen a vendor's demo has not.

Building your own gold set from already-signed packages

A gold set, for this purpose, is a stack of 20 to 30 already-funded, already-signed packages where the desk already knows the answer — which serials matched, which lien filings were correct, which insurance certificates were right the first time and which needed a correction. Replaying the review against that stack, and scoring read rate, match rate, unsourced rate and confabulation rate against the desk's own known-correct answers, is a measurement a vendor's marketing cannot substitute for, because it is specific to the documents that desk actually funds.

No public, equipment-finance-specific accuracy benchmark exists for this kind of review, and a desk should treat that absence as the reason to measure its own result rather than as a gap someone else will eventually fill. The four rates above, run against a desk's own already-signed packages, are a more honest number than any single accuracy figure a vendor can quote.

Questions people ask

How accurate is AI document review for loan documents?

There is no single trustworthy number, because accuracy on loan documents has not been independently benchmarked the way other document types have. The honest approach replaces one score with four measured rates — read rate, match rate, unsourced rate and confabulation rate — tested on a desk's own closed packages rather than taken from an unrelated published study.

How do you test AI loan-document accuracy?

Assemble 20 to 30 already-funded, already-signed packages whose real exceptions the desk already knows, replay the review against them, and compare its findings field by field against the known-correct answers. That gives a read rate, a match rate and a confabulation rate specific to the desk's own documents, not a vendor's demo set.

Can AI be trusted to check loan documents?

Only to the extent its claims are checkable. A review that cites the page and position behind every value it relies on, and that drafts a fix rather than silently correcting a document, gives a reviewer something to verify. A review that returns a single pass/fail with no citation does not, regardless of its claimed accuracy.

What is AI confabulation?

NIST's Generative AI Profile defines it as the production of confidently stated but erroneous or false content — language popularly called hallucination. It is the central risk in document review because a confabulated field looks exactly as confident as a correctly read one until someone checks the citation behind it.

How do you validate an AI underwriting or funding-review tool?

Bank model-risk guidance points at the right shape of an answer even though it does not yet cover agentic AI by name: validate before first use, then run outcomes analysis on an ongoing basis, comparing the tool's outputs to real, already-known results. For a funding review, that means a pilot on closed packages before go-live and a periodic back-test afterward.

What is a typical AI document-extraction error rate?

Published figures vary widely by task and document type — one 2025 study of government fiscal-document extraction reported 73% to 96% structural accuracy and about 84% numerical consistency, while a 2026 benchmark found current language models still struggle to detect errors in financial documents, especially complex ones. None of these studies measured loan or lease funding packages, so none should be quoted as this review's own error rate.