Backtesting an AI model means running it against data a lender already has the right answer for, and checking whether the model's output matches what actually happened. For a field-exam tool, the right data already exists in every exam a lender has closed and signed: each one has a final set of findings, confirmed on site and approved by a credit officer, that a new tool's output can be measured against before that tool ever touches a live file. This is the same discipline bank model-risk functions already apply to every other model they rely on, applied to the newest kind of model a lending desk is likely to adopt.
The process below follows how a fictional sample lender would backtest Field Exam Automation, Botminds' software for asset-based lending field exams, against its own closed exams before using it live. The lender and its files are invented for this walkthrough; the regulatory language is quoted from the federal banking agencies and NIST, linked where it appears.
What backtesting means for an AI field-exam tool
Backtesting comes from bank model-risk guidance, which was rewritten this year. The revised interagency guidance of 17 April 2026, which "supersedes and replaces SR letter 11-7", says that "outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use", and names "back-testing or outlier analysis" among its forms (Federal Reserve, SR 26-2). Applied to a field-exam tool, the model outputs are the findings it raises on a replayed exam, and the real-world outcomes are the findings the examiner and manager actually signed off on when that exam ran for real. The period has to be one the tool never saw during its own development or tuning, or the comparison tests memory rather than judgment.
One caveat belongs at the top of any model-risk file on this subject: the same guidance states that "generative AI and agentic AI models are novel and rapidly evolving" and "are not within the scope of this guidance", leaving their governance to each bank's own "risk management and governance practices". A backtest of an agentic field-exam tool is therefore not something a regulation prescribes. It is the bank borrowing the most transferable discipline in the rulebook because nothing more specific exists yet.
This is a different exercise from a vendor demonstration, which typically runs a tool on a file chosen to show it in its best light. A backtest runs it on files a lender's own team already closed, picked by the lender, with a result the team can independently check against workpapers it already has — raising the bar from "does this look convincing" to "does this match what we actually found."
Outcomes analysis: comparing model output to what actually happened
The revised guidance treats outcomes analysis as a component of model validation, alongside conceptual soundness, and adds that "when a model’s design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment" (SR 26-2), which describes a field exam well. For a field-exam tool, that comparison runs finding by finding: did the tool flag the same ineligible receivables, the same dilution pattern, the same inventory exclusion gap the original exam found, with the same dollar effect on availability — or did it miss something, flag something that was not really there, or reach a different number by a different route.
NIST's framework for AI systems describes the same discipline from a measurement angle: "accuracy measurements should always be paired with clearly defined and realistic test sets — that are representative of conditions of expected use — and details about test methodology" (NIST, Artificial Intelligence Risk Management Framework, NIST AI 100-1). A lender's own closed exams, by definition, are a realistic test set drawn from conditions of expected use — they are the actual borrowers, actual data formats and actual findings the tool will face once it is live.
The backtest as four steps, two of them a person's
Agent
ReplayThe tool runs the closed exam's original data download as if it were new, producing its own findings.
Agent
ScoreEach tool finding is matched against the exam's original, signed findings: agree, miss, or a new flag the original exam did not raise.
A person decides
AdjudicateA field exam manager reads every disagreement and decides which side — the tool or the original exam — was right.
Never: The tool never adjudicates its own disagreements.
A person decides
DecideModel risk or credit leadership compares the scored results against the pass line set before the replay ran, and decides whether to pilot.
Building a gold set from exams you have already closed
A gold set, for this purpose, is a collection of already-closed exams whose original findings are treated as the answer key the new tool is measured against. The files do not need to be special or curated — in fact a representative set is a better test than a hand-picked one, because it includes the ordinary exams where nothing unusual happened alongside the ones with real findings. A reasonable starting gold set draws from several recent quarters, across a range of borrower sizes and industries, and includes at least some exams with confirmed findings and some with none, so the backtest can check both whether the tool catches real problems and whether it stays quiet on clean files.
What a backtest needs before it can run
- The original data download for each exam in the gold set
The same files the examiner worked from — the general ledger, agings, perpetual inventory, shipping and returns logs, the certificate.
- The exam's final, signed findings
What the field exam manager actually approved, including the dollar effect on availability for each finding.
- The workpapers behind each finding
So a disagreement can be checked against the original evidence, not just the original conclusion.
- A period the tool was not tuned on
Exams used to configure or train the tool are not a fair test of how it performs on new ones.
- A spread of outcomes
Some exams with real findings, some clean, across a few different borrowers — not one impressive file repeated.
What counts as agreement, and what counts as a miss
Three outcomes cover every comparison in a backtest. An agreement is a finding the tool raised that matches the original exam's signed finding, in substance and in dollar effect, even if the wording differs. A miss is a finding the original exam raised that the tool did not — the more serious of the two error types, because a missed finding is exactly the risk backtesting exists to catch before it reaches a live file. A new flag is something the tool raised that the original exam did not; it is not automatically wrong, because the original examiner, working under time pressure on one pass through the file, can simply have missed something the tool's full-file read catches.
The dollar size of a miss or a new flag matters as much as the count. A tool that misses one $400 finding across ten exams is a different result from one that misses a single $200,000 ineligible-receivables finding once — a simple agreement percentage across all findings can hide the second case inside a reassuring-looking overall number, so a backtest should weigh misses by their effect on availability, not just count them.
Reading a disagreement: sometimes the old file was wrong
A disagreement is not automatically the tool's fault. Field exams are themselves produced under time pressure, by people sampling a download rather than reading every line of it, and a backtest sometimes surfaces a pattern the original examiner had no realistic way to catch — a credit-and-rebill outside the sample that was tested, an inventory exclusion gap in SKUs nobody counted that quarter. When that happens, the right read is not "the tool found something extra" but "the tool found something the original exam, working by sample, was always going to have some chance of missing."
This is why adjudication belongs to a person who can go back to the original workpapers and the underlying files, not to a simple score. A field exam manager reading a disagreement needs to ask whether the tool's new flag holds up against the actual documents — does the invoice genuinely show a rebill pattern, does the SKU genuinely have no sale in the relevant window — rather than assuming either side is automatically right. A disagreement that holds up on inspection is evidence the tool is working as intended, even though it technically did not "agree" with the original exam.
Setting the pass line before you see the results
The order matters more than any specific number: a model-risk function or credit leadership should set what counts as a pass before the replay runs, not after looking at how the tool performed. The revised guidance describes outcomes analysis identifying "material errors or persistent deviations outside of the banking organization’s established performance thresholds", in which case "model adjustment, recalibration, or redevelopment may be warranted" (SR 26-2). Established is the operative word: a threshold chosen after seeing a disappointing result is not a threshold; it is a rationalization.
A pass/fail line, set before the replay runs
No missed finding above
$25,000 of availability effect
Any larger miss fails the backtest outright, regardless of overall agreement.
Overall agreement, by dollar
At least 90%
Weighted by each finding's effect on availability, not a simple count of findings.
New flags, on inspection
Hold up against the workpapers
A new flag that does not survive a documents check counts against the tool.
Whatever the specific numbers, writing them down first keeps the question clean: did the tool clear the line the lender itself set, or not. The backtest's job stops there — what to do about a tool that falls just short, run a second gold set, adjust scope, or decline, is a decision for the people who set the line, not a conclusion the backtest draws on its own.
From backtest to a live pilot, in weeks not months
A backtest that clears its line does not mean going live the same week — it means moving to a pilot, where the tool runs alongside a live exam in parallel with the existing process rather than replacing it. The pilot answers a question the backtest cannot: how the tool performs on a file nobody has already seen the answer to, confirmed by the examiner on site in real time rather than checked against workpapers afterward.
A backtest-then-pilot schedule, in weeks
- Assemble the gold set1 wk
- Set the pass line1 wk
- Replay and score1 wk
- Adjudicate disagreements2 wk
- Parallel pilot, live exams4 wk
Field Exam Automation is built and deployed from a lender's own requirements — its exam program, its loan agreement definitions, its report format, and a sample of its own past data downloads — and does not go live until it has passed every check in that process, starting with exactly the replay this guide describes, run on a lender's own file before any commitment is made.
"We will walk you through it on one of your own past exams first."
Field Exam Automation
What a model-risk file should hold afterward
Whatever the backtest decides, the record of it belongs in the same kind of file a bank already keeps for every other model it relies on. That includes the gold set used, with its date range and why it was chosen; the pass line set in advance and who approved it; the scored results, including every disagreement and how it was adjudicated; and, if the tool proceeds to a pilot, the parallel results from live exams once they close. The OCC's own description of field-exam rigor generally — "the field auditor should obtain written account verifications and perform sufficient reconciliations and testing to ensure that the borrower's financial records are accurate" (OCC, Comptroller's Handbook: Asset-Based Lending) — is the same standard a bank should hold its own tool-verification process to: written, reconciled, and able to withstand an examiner's own later review.
One exam, replayed and scored

Agreement across a gold set, over time

For how a field exam's own accuracy is measured once a tool is live — the sample agreement rate, and the unsourced, stale and refusal rates alongside it — see verifying an AI-drafted exam's accuracy. For what the six agents in this program do and the limits each carries, see what none of the six agents may do. To see a replayed exam end to end, start from Field Exam Automation.
Questions people ask
How do you backtest an AI model?
By running it on data where the right answer is already known — for a field-exam tool, a set of exams a lender has already closed and signed — and compares a model's output to what the original exam actually found, the way the 2026 interagency model-risk guidance (SR 26-2) describes outcomes analysis and back-testing more generally.
What is outcomes analysis in model validation?
A component of model validation in the federal banking agencies' revised guidance (SR 26-2, April 2026, which replaced SR 11-7): it "compares model outputs to corresponding real-world outcomes to assess model performance". Back-testing is one of its forms, comparing a model's outputs against what actually happened over a period the model was not developed on.
How many past exams should a pilot replay?
Enough to cover a range of outcomes, not just one impressive file — a reasonable starting gold set spans several recent quarters and a mix of borrower sizes, including some clean exams and some with real findings, so the test checks both whether the tool catches problems and whether it stays quiet when there are none.
How do you adjudicate when the AI and the team disagree?
By going back to the original workpapers and underlying documents, not by assuming either side is right. A field exam manager checks whether a new flag the tool raised actually holds up against the evidence, and whether a finding the tool missed was realistically visible to the original examiner's sample.
Is a pilot required before using AI for a field exam?
There is no rule requiring one: the 2026 interagency model-risk guidance places agentic AI outside its scope and leaves it to each bank's own governance. The discipline that guidance applies to other models, outcomes analysis against established thresholds and ongoing monitoring, is the sensible one to borrow, and a parallel pilot after a successful backtest is how a bank applies it before full reliance.
What data do I need to backtest a lending AI tool?
The original data downloads for each exam in the gold set, the exam's final signed findings, and the workpapers behind each finding, covering a period the tool was not tuned or configured on, so the comparison tests genuine performance rather than memorized answers.