To pilot ag lending software properly, run it on farm renewal files your own officer has already closed and signed, score every figure and every exception against what the file actually held, and read each disagreement before you read any total. That is the whole method, and it is harder to arrange than a demonstration because the thing being tested is your own past work as much as the software's. Botminds' Agricultural Loan Renewal Management prepares each operating line for renewal — spreads, carryover, the cash flow at your price deck, insurance and lien filings, a drafted memo with every figure cited — and a pilot of it is worth running the same way a bank already tests a model that matters: against outcomes it can check, by people who did not build it.
What follows is a four-week plan with the sample, the scoring sheet and the decisions to take before anything runs, plus an honest list of what a back-test cannot tell you. The season figures are from the fictional sample bank in the solution; the supervisory material is from the federal banking agencies and from USDA's lending regulations, linked where it appears.
What a pilot of ag lending software on your own files actually tests
Four things, and it is worth being explicit about them because a pilot that tries to test everything tests nothing. It tests whether the system can read your documents — your accountants' statement layouts, your crop plan templates, the filing office records your state produces. It tests whether it applies your rules rather than a generic bank's. It tests whether its output is checkable, which is a question about citations rather than correctness. And it tests the disagreements: where the system and your officer differ, and who turns out to be right.
What it does not test is the thing a demonstration is best at showing — speed. The time to prepare a file is close to irrelevant to a pilot's conclusion, both because it will be fast and because, as the arithmetic in the real cost of a farm loan renewal shows, it is the input that moves a business case least. Spend the pilot's attention on the four questions above.
"Outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use."
SR Letter 26-2, Revised Guidance on Model Risk Management (Federal Reserve, OCC and FDIC)
How many renewal files make a fair pilot?
Enough to meet every failure mode at least twice, which in practice means twenty to thirty closed renewals rather than five or a hundred. Five files will all be clean, because clean files are what people reach for. A hundred is a project with a steering committee, and the marginal file stops teaching anything after the first two dozen.
Composition matters more than count, and bank examiners have already published a good list of what to deliberately include. When selecting an agricultural loan sample, the OCC's handbook tells examiners to consider, among others, "loans with multiple extensions", "loans identified as having carryover that are not adversely risk rated by the bank", "loans with significant policy or underwriting exceptions" and "loans with modified repayment terms" (Comptroller's Handbook: Agricultural Lending). Those four categories are exactly the files a pilot should be weighted towards, because they are where a renewal's analysis is hardest and where last season's shortcuts show.
On top of those, span the ordinary variation in your book: maturity month, commodity mix, cash rent against owned ground, borrowers with and without carryover, at least two files where the crop insurance had a gap, and one file that went badly enough to be rated down. Draw it before you see any results, and write the list down.
A four-week pilot of Agricultural Loan Renewal Management on your own closed renewals
| When | Who owns it | What happens | Why in that order |
|---|---|---|---|
| Before anything runs | Credit risk | Draw the sample, write the scoring sheet, set the thresholds that would make this a yes or a no, and name who adjudicates a disagreement. | Nothing has been seen, so nothing is arguable later |
| Week 1 — your rules | Ag Lending Manager, credit risk | Load the credit policy, price deck, memo template, pricing grid, approval authorities and the lien and inspection intervals as settings. Confirm each change is versioned. | A system run on somebody else’s policy is testing the wrong thing |
| Week 2 — the run | The vendor, watched | The agents read each file’s documents as the file stood at renewal. Nothing is sent to a borrower, nothing is filed, no memo leaves the room. | The officer’s own memo and conditions stay sealed until scoring |
| Week 3 — scoring | Credit risk, with the officer who signed | Mark every decision figure agree, disagree or unsourced. Mark every proven exception listed or missed. Open a sample of citations. | Line by line, against the denominator fixed in week 0 |
| Week 4 — adjudication | Credit risk and the Chief Credit Officer | Classify each disagreement: the agents were wrong, the old file was wrong, or it was a judgement call. Write the result against week 0’s thresholds. | The second category is the most valuable output of the whole exercise |
| Then — one season in parallel | Ag lending | Run a real renewal season both ways on a slice of the book, to test the document chase a back-test cannot reach. | The largest cost of the change, and the one most often left out |
What to bring: credit policy, price deck, memo template, past files
A pilot run on the vendor's defaults measures the vendor's defaults. Four things have to come from your side of the table before the first file is read.
Your credit policy, as settings. Minimum coverage, working capital to revenue, maximum debt to assets, how stored grain is valued in the borrowing base, the crop insurance minimum, the lien warning period and inspection interval, how far ahead of maturity a file should be ready, and every approval authority. In the solution these are settings rather than configuration work, and changes are versioned and apply to files prepared from then on — which is a property to verify in the pilot rather than take on trust, because it is what makes a before-and-after comparison possible later.
Your price deck and stress case. Whoever sets the deck at your bank should set it in the system, in front of you, and you should see the stress test appear beside a plan. A cash flow built on a borrower's prices is not the same analysis as one built on yours, and a pilot that does not check which prices were used has not tested the thing a credit committee cares about most.
Your memo template and pricing grid. The memo is the deliverable. If the draft comes back in a shape your approver does not read, the pilot has found something real — either a configuration job or a reason to say no.
The files themselves, as they stood at renewal. Not as they stand now. A file that has since collected a signed assignment, a filed continuation and a year of payment history will flatter any system. Pull the documents the officer actually had, and hold back the officer's memo and conditions until scoring.
The settings a pilot of Agricultural Loan Renewal Management has to load first

How to back-test AI against loan files you have already closed
The mechanics are simple once the preparation is done. Give the system each file's documents as they stood at renewal. Let it produce what it produces — spreads, the comparison with last year's plan, the cash flow at your deck, the insurance and lien findings, the drafted memo. Then compare it with what your officer produced, figure by figure and exception by exception.
The comparison needs a fixed denominator, agreed in week 0: the figures a recommendation actually turns on, rather than every cell in a spread. On a farm renewal that list is short and recognisable — the year-end line balance, the inventory count and its valuation, last year's yields and realised prices, planned acres by crop and lease, the input budget, the term payment schedule, insured acres and coverage level, and the lien filing dates. Score each as agreeing, disagreeing or unsourced; the full framework is in the four measures of accuracy.
There is a regulatory precedent for the comparison itself, and it is one ag lenders already live with. USDA's rules for guaranteed farm loans require an annual analysis of the borrower's financial condition, and specify that "for loans secured by personal property, lenders will also compare the borrower's actual to planned income and expenses for the past year" (7 CFR § 762.140). Comparing what happened with what was planned is the same operation a back-test performs on the analysis itself — one level up.
A back-test of one closed renewal, end to end
A person decides
Reconstruct the fileAssemble the documents the officer had at renewal: statements, crop plan, leases, schedule of insurance, lien search, inspection reports, inventory count.
Never: Never includes anything that arrived after the renewal was signed
A person decides
Seal the officer’s workThe memo, the conditions and the approval go into an envelope until scoring, so the two analyses are genuinely independent.
Never: Never shown to the system or to the people running it
Agent
Prepare the renewalStatements spread, last year set against its plan, this year’s cash flow built at your deck, insurance and filings checked, a memo drafted with citations.
Never: Never sends a request, files a continuation or approves anything
A person decides
Score against the fixed denominatorEach decision figure agree, disagree or unsourced; each proven exception listed or missed; a sample of citations opened.
Never: Never scores a near-miss as a match
A person decides
Adjudicate with the officer in the roomEvery disagreement classified as the system’s error, the old file’s error, or a judgement call, with the document on the table.
Never: Never decided by whoever argues longest
A person decides
Write the result against week 0’s thresholdsReport the measures separately, name the concentrated weaknesses, and record what will be re-tested after go-live.
Never: Never reduced to a single percentage
What counts as a disagreement in a pilot, and how to score it
A disagreement is any decision figure where the system and the closed file differ, or any exception one of them raised and the other did not. It is not a failure until somebody decides whose failure it is, and there are only three answers.
Three kinds of disagreement, and what each one obliges you to do
The system was wrong
It read a figure that is not on the page, missed an exception the documents proved, or cited a source that does not support the number.
What it obliges: count it against the measure it belongs to, and look for concentration. Ten errors spread across ten files is noise; ten in one document type is a defect with an owner.
The old file was wrong
Carryover named without an inventory count, a coverage ratio built on the wrong payment schedule, acres never reconciled to the schedule of insurance, a lapse date taken from a rule rather than a filing.
What it obliges: a conversation with credit risk about last season, not a mark against the vendor. This is the finding that justifies the exercise whatever you decide about the software.
It was a judgement call
A yield assumption inside a defensible range, whether a lease renews, how much of a crop a borrower will have contracted by March.
What it obliges: record it and exclude it from the measures. Then check the system showed its basis clearly enough for the officer to disagree knowingly.
Two scoring rules keep the exercise honest. A near-miss is not a match: a flag that says coverage is thin without naming the ratio and the minimum has not found the exception. And a figure that is right but unsourced is scored as unsourced, because an approver cannot act on it and an examiner cannot check it.
What SR 26-2 says about outcomes analysis and back-testing
The agencies' revised model risk management guidance is the best short description of this method that a bank can point a committee at, with one caveat to state up front. SR 26-2 — issued jointly by the Federal Reserve, the OCC and the FDIC in April 2026 — puts generative and agentic AI outside its own scope in a footnote, while saying an organisation's risk management and governance practices should still determine appropriate controls for tools it does not cover. So what follows is a method worth borrowing, not a requirement being met.
On what outcomes analysis is, the guidance is plain: it "compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use" (SR 26-2). On its forms, it is broader than most banks assume: "outcomes analysis can take many forms, including testing conducted during model development, reports or analysis performed as part of ongoing monitoring, or standalone activities such as back-testing or outlier analysis." A pilot on closed files is the standalone back-testing case.
Two further sentences earn their place in a pilot report. The guidance notes that outcomes analysis "may identify material errors or persistent deviations outside of the banking organization's established performance thresholds", in which case "model adjustment, recalibration, or redevelopment may be warranted" — which is why thresholds belong in week 0 rather than in the discussion of the results. And on exactly the territory a renewal occupies: "when a model's design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment." A farm renewal is expert judgement from end to end. The measurement is how you find out how good yours has been.
What a pilot cannot test: the season's document chase
A back-test holds every document still, and that is its blind spot. Four things stay untested until a real season.
The chase is the first. On a closed file every document is present; the whole question of whether a borrower responds to a drafted request, whether the accountant sends the statement in January instead of February, whether an insurance agent answers, is invisible. In the sample season a renewal package opened in October and filled over twenty weeks — that is the behaviour that shortens a season, and no back-test can show it. Test it in the parallel season, on a slice of the book.
The connections are the second. Loan servicing, lien search and insurance feeds differ by bank, and some may be simulated in any system you evaluate. Ask directly which are live, which a pilot would have to stand up, and which would remain a manual step on day one.
The people are the third. Whether an officer trusts a flag enough to act on it, and whether an approver reads a memo in the new shape, is a question about habits. And the fourth is the unusual year: a back-test over three normal seasons says nothing about a drought, a price collapse or a change to the price deck mid-season.
Where a 24-file pilot's own hours go, in our suggested plan
Sixty hours is the honest price of a defensible answer, and it is cheap against a season. What it buys is not only a verdict on the software: it is a measured baseline of your own desk, a list of what last season got wrong, and a scoring sheet you can re-run whenever something changes.
How long does an ag lending software pilot take?
Four working weeks for the back-test, on the plan above, and one renewal season for the parallel run. The constraint is rarely the software; it is getting twenty-four files reconstructed as they stood at renewal and getting the officer who signed them into a room three times.
Work backwards from the season rather than forwards from today. If your operating lines mature between mid-February and 1 April, the back-test wants to be finished by the end of October, because the settings work, the connections and any configuration then have the autumn — and because a renewal package that is supposed to open in October has to exist in October. A pilot that finishes in January has proved something true about a season it is too late to change.
Two scheduling rules save a lot of grief. Do not run a pilot across your own busiest eight weeks: the people who have to adjudicate the disagreements are the people renewing the book. And do not let the parallel season be a half-season — run the slice of the book you choose all the way to approval, including the memo and the signatures, because the approval step is where a new shape of memo either works or does not.
The deliverable a pilot is really scoring in Agricultural Loan Renewal Management

What to agree before go-live
A pilot that ends in a decision and nothing else has thrown away half of what it produced. Six things belong in writing before the first real renewal is prepared, and all six are easier to agree while the results are fresh.
What to settle in writing between the pilot and go-live
- The thresholds you measured against, and what happens if they slip
Written in week 0, re-stated at go-live. Name who is told and what changes if a measure falls below them in the live season.
- What re-starts the measurement
A settings change such as a new price deck or policy minimum, a new document source, and the turn of the season. Put all three in the procedure.
- Who owns the settings, and how changes are versioned
The price deck belongs to whoever sets it today. Confirm a change applies to files prepared from then on, not retrospectively.
- The list of what the system never does alone
Send to a borrower, file a continuation, approve a renewal, price outside the grid, estimate a missing figure. In writing, from the vendor.
- Which connections are live on day one
And what the manual step is for each that is not. This is the most common gap between a successful pilot and a disappointing first season.
- Who measures it afterwards, and how often
Independent of the ag lending department whose season gets shorter, with loan volume out of their incentives — the same rule the OCC sets for loan review.
- What the parallel season covers, and when it ends
Which slice of the book, through to approval and signature, and the date you stop running both ways.
One finding from the pilot deserves to outlive it. Where the old file turned out to be wrong, the bank has learned something about its own renewals that no vendor decision changes — and the OCC's handbook is a reminder of what is at stake in those files: "loans to sound borrowers that are refinanced or renewed in accordance with prudent underwriting standards should not be classified unless well-defined weaknesses exist that jeopardize repayment" (Comptroller's Handbook: Agricultural Lending). The protection in that sentence is conditional on the standards, and the standards are evidenced by the file. A renewal whose analysis cannot be produced does not get the benefit of it.
For the measures to score a pilot against, see how accurate AI is at farm loan underwriting; for the cost side of the same decision, the real cost of a farm loan renewal; and for what you would be comparing against, the three ways to run a renewal season. To see a prepared renewal and its settings for yourself, start from Agricultural Loan Renewal Management.
Questions people ask
How do you pilot ag lending software?
Run it on twenty to thirty renewal files your own officer has already closed and signed, with the documents as they stood at renewal and the officer's memo sealed until scoring. Fix the sample, the figures you will score and the thresholds before anything runs. Then score every figure and exception, and adjudicate each disagreement as the system's error, the old file's error, or a judgement call.
How many files should a software pilot cover?
Twenty to thirty closed renewals. Five will all be clean files; after about two dozen the marginal file stops teaching anything. Weight the sample towards the files bank examiners themselves look for — loans with multiple extensions, carryover that was not adversely rated, significant policy or underwriting exceptions, and modified repayment terms — plus your ordinary spread of maturity months, commodities and tenure.
What should you bring to a loan software pilot?
Your credit policy as settings, your price deck and stress case, your memo template and pricing grid, and the files as they stood at renewal. A pilot run on a vendor's defaults measures the vendor's defaults. Change one rule while you watch, so you learn early whether a policy change is a setting, a change request or a release.
What counts as a disagreement in a pilot?
Any decision figure where the system and the closed file differ, or any exception one raised and the other did not. It is not a failure until it is classified. Three categories cover almost all of them: the system was wrong, the old file was wrong, or it was a judgement call inside a defensible range. Only the first counts against the software.
How long does an ag lending software pilot take?
About four working weeks for the back-test, plus one renewal season run in parallel on a slice of the book. Work backwards from your maturity window: if lines mature between mid-February and 1 April, finish the back-test by the end of October, so the settings, connections and configuration have the autumn and the first renewal packages can open on time.
Does supervisory guidance require back-testing of AI underwriting?
Not for agentic systems. SR 26-2, the agencies' revised model risk management guidance, places generative and agentic AI outside its scope in a footnote while expecting an organisation's own risk management practices to determine the controls. Its description of outcomes analysis — comparing outputs to real-world outcomes, including standalone back-testing — remains the clearest method to borrow, and it notes that when a design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment.