Piloting a funding-review tool on your own closed packages first

Piloting AI software on your own closed loan files — not on a new deal — is the only honest test of whether a funding-review tool actually works, before it ever touches money that has not already been decided. A package that is already signed, already funded, and whose exceptions are already known and on file is a package where a reviewer can tell, immediately, whether the tool found what the desk's own people already found. A brand-new deal cannot tell you that; nobody yet knows the right answer to compare against.

The pilot structure below applies to Equipment Finance Pre-Funding Review specifically, but the shape of it — replay old files first, agree every pass/fail check before you start, and let a person adjudicate every disagreement — holds for piloting any agentic review against funding decisions that already carry real money.

Don't pilot on new deals — pilot on signed ones

The reason to start on already-closed packages rather than live ones is not caution for its own sake; it is the only way to get a result that means anything. Federal bank model-risk guidance frames the same principle for any model a bank relies on: "validation generally occurs prior to a model's first use" (SR 26-2). A pilot that runs a new review alongside live funding decisions is not validation — it is going live with an extra step, and any disagreement the tool raises on a live deal has nobody's known-correct answer to check it against.

A closed package removes that problem. The documentation officer who filed it, the funding specialist who released it and the credit analyst who approved it all already know what was wrong with it, if anything was. Replaying the review against that package turns "did the tool get it right" into a question with a checkable answer on day one of the pilot, not months into a live rollout.

What to bring: 20–30 closed packages, with their known exceptions already on file

A pilot is only as good as the set of files behind it. Twenty to thirty packages is enough to cover most of a desk's normal variation — different vendors, different equipment types, titled and non-titled units, at least a few packages that had a real exception and a few that were clean — without being so large that reviewing the disagreements becomes its own project.

The checks, and why a check that cannot fail is not a check

Before the first package is replayed, the desk should write down every check the pilot has to pass — and each one has to be capable of failing, or it is not actually testing anything. "The tool seemed helpful" is not a check. "The tool flagged the same serial mismatch our documentation officer caught on file EF-24-00318" is.

The settings a pilot tests against, not a vendor default

What federal bank guidance expects before a model is relied on

A pilot gated this way is not an invention specific to this product; it is close to what bank examiners already expect of any model before it is relied on. The Federal Reserve's current letter states that "outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use" and that this analysis "can take many forms, including testing conducted during model development... or standalone activities such as back-testing or outlier analysis" (SR 26-2). A pilot replaying closed packages is exactly this kind of back-testing, done before the model's first live use rather than after.

The National Institute of Standards and Technology's reasoning for why this matters is the same reasoning behind every check on the list above: "confabulations can occur across GAI outputs and contexts", and "risks from confabulations may arise when users believe false content, often due to the confident nature of the response" (NIST AI 600-1). A pilot that only checks whether the tool's output looks plausible is checking the wrong thing; a pilot that checks the output against a known-correct answer on a closed file is checking the thing that actually matters.

Third-party due diligence: what to ask the vendor before go-live

Alongside the model itself, a bank's own third-party risk practice has a separate, ongoing question: is the vendor behind the model still performing as expected after go-live, not just at the demo. The Office of the Comptroller of the Currency frames it plainly: "a community bank's ongoing monitoring of the third party's performance enables bank management to determine if the third party is performing as required for the duration of the contract" (OCC, Third-Party Risk Management: A Guide for Community Banks). A pilot is the first instance of that monitoring, not a one-time event that ends at go-live.

A pilot is also the moment to put a question to both the vendor and the bank's own technology function that most banks have not yet had to answer for an agentic tool. The OCC's Spring 2026 Semiannual Risk Perspective says plainly that "GenAI and agentic AI models are novel and rapidly evolving and are not within the scope of the revised model risk management guidance", and that "the agencies plan to issue in the near future a request for information that addresses model risk management generally and considers, in particular, banks’ use of AI, including genAI, agentic AI, and AI-based models" (OCC, Semiannual Risk Perspective, Spring 2026). Until that guidance exists, a bank's own pilot evidence is the strongest documentation it will have of whether the tool is safe to rely on — worth keeping in the same file a future examiner might eventually ask for.

The vendor questions the OCC lists before any third-party relationship

The OCC's 2024 guide for community banks turns the interagency third-party guidance into questions a bank should be able to answer before it signs. Several apply directly to a funding-review tool: "What are the underlying activities to be performed, and what are the bank and third party’s prospective roles in the activities?"; "How will the bank integrate third-party technology with the bank’s existing systems and infrastructure?"; "What physical and/or system access would the third party have to bank facilities, systems, and records"; "What are the information security implications, and how will the third party access, process, and protect customers’ information?"; and "Has the bank considered how to exit the activity or transition the activity to an alternative third party or in-house?" (OCC, Third-Party Risk Management: A Guide for Community Banks).

A pilot on closed packages answers the first two concretely, because the roles and the integration have to work for the replay to run at all. The exit question deserves its own line in the pilot file: if the funder stopped using the tool, would its settings, its exception history and its audit trail leave with the funder in a form another reviewer could read? A pilot that cannot answer that has tested the review, not the relationship.

Week by week: from policy settings to the first live release

A pilot built this way can run in about six weeks, because most of the work is replay rather than new development; the schedule below is illustrative, not a promise.

Nothing in that schedule requires a new deal to be at risk. The first live release in week six still sits inside the desk's existing release limits and second-signature rule, so the first real dollar the pilot touches is still covered by the same governance that would have applied without a pilot at all.

Reading a disagreement — some are your own file's old errors

Not every disagreement between the tool and a closed file's history means the tool is wrong. Some will be exactly that. Others will surface an old inconsistency the desk's own people missed at the time — a serial that was never actually double-checked against the plate photo, just accepted because the rest of the package looked clean. Reading a disagreement is a four-step job, and it is the one step in the pilot that cannot be automated.

A desk that treats every disagreement as a tool error will conclude the pilot failed even when it worked. A desk that treats every disagreement as proof the tool is right will miss the cases where it genuinely got something wrong. The adjudication step exists specifically so neither assumption gets to win by default.

What "done" looks like before the pilot ends

A pilot is finished when every pre-agreed check has a recorded pass or fail against the closed-package set — not when the demo looked good, and not on a fixed calendar date regardless of the result. The scope that ends the pilot is deliberately smaller than the scope of full go-live.

The follow-ups a live desk checks every morning, once the pilot is behind it

A pilot that ends with every check scored, every disagreement adjudicated, and the desk's own settings — not a vendor default — proven against real files, is a pilot a funder can stand behind to its own credit committee, its own auditors, and eventually to an examiner, without needing to borrow a framework that does not yet exist for agentic AI.

Questions people ask

How do I pilot equipment-finance funding-review software?

Replay the review against 20 to 30 of your own already-closed, already-signed packages, whose real exceptions you already know, rather than running it alongside new live deals. Score it against checks agreed before the pilot starts, adjudicate every disagreement with the original documents, and only move to live packages once every check has a recorded result.

How long does it take to implement funding-review software?

A pilot built around replaying closed packages typically runs about six weeks: loading settings and the package set, replaying and comparing against known outcomes, adjudicating disagreements, and scoring the result before the first live release. There is no published industry-standard timeline; this is a reasonable schedule, not a guarantee.

What data do I need to test AI underwriting or funding-review software?

A mix of 20 to 30 closed packages — some clean, some with a real exception — each with the complete original documents and the desk's own record of what, if anything, was wrong with it. That known-correct answer is what makes the pilot testable; without it, there is nothing to check the tool's output against.

What should I ask a vendor about model validation before go-live?

Ask what was validated before the tool's first use and how its outputs are checked against real outcomes afterward — the two things federal model-risk guidance asks of any model, even though current guidance does not yet formally cover agentic AI. Ask separately how the vendor itself is monitored on an ongoing basis, which is a third-party-risk question, not a model question.

How do I test AI on closed loan files?

Run the review exactly as it would run on a new package, list every place its findings differ from what your desk already recorded for that file, and have a named person — not the vendor — decide which side was right for each difference. Some disagreements will reveal an old error in the file, not a mistake by the tool.

What third-party due diligence does an AI vendor need before I rely on it?

The same due diligence any third-party service needs: an assessment, before the relationship starts, of its ability to perform as expected and operate safely, followed by ongoing monitoring of its actual performance for the life of the contract. A gated pilot on closed packages is the first round of that monitoring, not a substitute for it.