SystemAgricultural Lending

Decide · running a pilot

How to pilot ag lending software on your own renewal files

The strongest pilot runs on renewal files your own officer already closed and signed, scores every disagreement, and treats some of them as the old file's errors rather than the system's. This guide sets out the sample, the scoring sheet, the four weeks and what to agree before Agricultural Loan Renewal Management, or anything else, goes live.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

To pilot ag lending software properly, run it on farm renewal files your own officer has already closed and signed, score every figure and every exception against what the file actually held, and read each disagreement before you read any total. That is the whole method, and it is harder to arrange than a demonstration because the thing being tested is your own past work as much as the software's. Botminds' Agricultural Loan Renewal Management prepares each operating line for renewal — spreads, carryover, the cash flow at your price deck, insurance and lien filings, a drafted memo with every figure cited — and a pilot of it is worth running the same way a bank already tests a model that matters: against outcomes it can check, by people who did not build it.

What follows is a four-week plan with the sample, the scoring sheet and the decisions to take before anything runs, plus an honest list of what a back-test cannot tell you. The season figures are from the fictional sample bank in the solution; the supervisory material is from the federal banking agencies and from USDA's lending regulations, linked where it appears.

What a pilot of ag lending software on your own files actually tests

Four things, and it is worth being explicit about them because a pilot that tries to test everything tests nothing. It tests whether the system can read your documents — your accountants' statement layouts, your crop plan templates, the filing office records your state produces. It tests whether it applies your rules rather than a generic bank's. It tests whether its output is checkable, which is a question about citations rather than correctness. And it tests the disagreements: where the system and your officer differ, and who turns out to be right.

What it does not test is the thing a demonstration is best at showing — speed. The time to prepare a file is close to irrelevant to a pilot's conclusion, both because it will be fast and because, as the arithmetic in the real cost of a farm loan renewal shows, it is the input that moves a business case least. Spend the pilot's attention on the four questions above.

"Outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use."

SR Letter 26-2, Revised Guidance on Model Risk Management (Federal Reserve, OCC and FDIC)

How many renewal files make a fair pilot?

Enough to meet every failure mode at least twice, which in practice means twenty to thirty closed renewals rather than five or a hundred. Five files will all be clean, because clean files are what people reach for. A hundred is a project with a steering committee, and the marginal file stops teaching anything after the first two dozen.

Composition matters more than count, and bank examiners have already published a good list of what to deliberately include. When selecting an agricultural loan sample, the OCC's handbook tells examiners to consider, among others, "loans with multiple extensions", "loans identified as having carryover that are not adversely risk rated by the bank", "loans with significant policy or underwriting exceptions" and "loans with modified repayment terms" (Comptroller's Handbook: Agricultural Lending). Those four categories are exactly the files a pilot should be weighted towards, because they are where a renewal's analysis is hardest and where last season's shortcuts show.

On top of those, span the ordinary variation in your book: maturity month, commodity mix, cash rent against owned ground, borrowers with and without carryover, at least two files where the crop insurance had a gap, and one file that went badly enough to be rated down. Draw it before you see any results, and write the list down.

A four-week pilot of Agricultural Loan Renewal Management on your own closed renewals

WhenWho owns itWhat happensWhy in that order
Before anything runsCredit riskDraw the sample, write the scoring sheet, set the thresholds that would make this a yes or a no, and name who adjudicates a disagreement.Nothing has been seen, so nothing is arguable later
Week 1 — your rulesAg Lending Manager, credit riskLoad the credit policy, price deck, memo template, pricing grid, approval authorities and the lien and inspection intervals as settings. Confirm each change is versioned.A system run on somebody else’s policy is testing the wrong thing
Week 2 — the runThe vendor, watchedThe agents read each file’s documents as the file stood at renewal. Nothing is sent to a borrower, nothing is filed, no memo leaves the room.The officer’s own memo and conditions stay sealed until scoring
Week 3 — scoringCredit risk, with the officer who signedMark every decision figure agree, disagree or unsourced. Mark every proven exception listed or missed. Open a sample of citations.Line by line, against the denominator fixed in week 0
Week 4 — adjudicationCredit risk and the Chief Credit OfficerClassify each disagreement: the agents were wrong, the old file was wrong, or it was a judgement call. Write the result against week 0’s thresholds.The second category is the most valuable output of the whole exercise
Then — one season in parallelAg lendingRun a real renewal season both ways on a slice of the book, to test the document chase a back-test cannot reach.The largest cost of the change, and the one most often left out
Our suggested plan. The order is the part that matters: every judgement that could be influenced by a result — the sample, the denominator, the thresholds — is made in week 0, before anything has been seen.

What to bring: credit policy, price deck, memo template, past files

A pilot run on the vendor's defaults measures the vendor's defaults. Four things have to come from your side of the table before the first file is read.

Your credit policy, as settings. Minimum coverage, working capital to revenue, maximum debt to assets, how stored grain is valued in the borrowing base, the crop insurance minimum, the lien warning period and inspection interval, how far ahead of maturity a file should be ready, and every approval authority. In the solution these are settings rather than configuration work, and changes are versioned and apply to files prepared from then on — which is a property to verify in the pilot rather than take on trust, because it is what makes a before-and-after comparison possible later.

Your price deck and stress case. Whoever sets the deck at your bank should set it in the system, in front of you, and you should see the stress test appear beside a plan. A cash flow built on a borrower's prices is not the same analysis as one built on yours, and a pilot that does not check which prices were used has not tested the thing a credit committee cares about most.

Your memo template and pricing grid. The memo is the deliverable. If the draft comes back in a shape your approver does not read, the pilot has found something real — either a configuration job or a reason to say no.

The files themselves, as they stood at renewal. Not as they stand now. A file that has since collected a signed assignment, a filed continuation and a year of payment history will flatter any system. Pull the documents the officer actually had, and hold back the officer's memo and conditions until scoring.

The settings a pilot of Agricultural Loan Renewal Management has to load first

The settings screen: the price deck set by the Chief Credit Officer, how stored grain is valued in the borrowing base, the stress test shown beside every plan, policy minimums for coverage, working capital and debt to assets, carryover treatment, how many days before maturity a file should be ready, approval authority by role, the crop insurance minimum, and the lien filing warning and inspection intervals.
Fictional sample settings in the solution. Load your own in week 1 and change one while you watch: whether a rule is a setting, a change request or a release is a question a pilot should answer early.

How to back-test AI against loan files you have already closed

The mechanics are simple once the preparation is done. Give the system each file's documents as they stood at renewal. Let it produce what it produces — spreads, the comparison with last year's plan, the cash flow at your deck, the insurance and lien findings, the drafted memo. Then compare it with what your officer produced, figure by figure and exception by exception.

The comparison needs a fixed denominator, agreed in week 0: the figures a recommendation actually turns on, rather than every cell in a spread. On a farm renewal that list is short and recognisable — the year-end line balance, the inventory count and its valuation, last year's yields and realised prices, planned acres by crop and lease, the input budget, the term payment schedule, insured acres and coverage level, and the lien filing dates. Score each as agreeing, disagreeing or unsourced; the full framework is in the four measures of accuracy.

There is a regulatory precedent for the comparison itself, and it is one ag lenders already live with. USDA's rules for guaranteed farm loans require an annual analysis of the borrower's financial condition, and specify that "for loans secured by personal property, lenders will also compare the borrower's actual to planned income and expenses for the past year" (7 CFR § 762.140). Comparing what happened with what was planned is the same operation a back-test performs on the analysis itself — one level up.

A back-test of one closed renewal, end to end

  1. A person decides

    Reconstruct the file

    Assemble the documents the officer had at renewal: statements, crop plan, leases, schedule of insurance, lien search, inspection reports, inventory count.

    Never: Never includes anything that arrived after the renewal was signed

  2. A person decides

    Seal the officer’s work

    The memo, the conditions and the approval go into an envelope until scoring, so the two analyses are genuinely independent.

    Never: Never shown to the system or to the people running it

  3. Agent

    Prepare the renewal

    Statements spread, last year set against its plan, this year’s cash flow built at your deck, insurance and filings checked, a memo drafted with citations.

    Never: Never sends a request, files a continuation or approves anything

  4. A person decides

    Score against the fixed denominator

    Each decision figure agree, disagree or unsourced; each proven exception listed or missed; a sample of citations opened.

    Never: Never scores a near-miss as a match

  5. A person decides

    Adjudicate with the officer in the room

    Every disagreement classified as the system’s error, the old file’s error, or a judgement call, with the document on the table.

    Never: Never decided by whoever argues longest

  6. A person decides

    Write the result against week 0’s thresholds

    Report the measures separately, name the concentrated weaknesses, and record what will be re-tested after go-live.

    Never: Never reduced to a single percentage

Our method. Step two is the one most often skipped and the one that decides whether the result means anything: an analysis compared with a memo the comparers have already read is not an independent test.

What counts as a disagreement in a pilot, and how to score it

A disagreement is any decision figure where the system and the closed file differ, or any exception one of them raised and the other did not. It is not a failure until somebody decides whose failure it is, and there are only three answers.

Three kinds of disagreement, and what each one obliges you to do

The system was wrong

It read a figure that is not on the page, missed an exception the documents proved, or cited a source that does not support the number.

What it obliges: count it against the measure it belongs to, and look for concentration. Ten errors spread across ten files is noise; ten in one document type is a defect with an owner.

The old file was wrong

Carryover named without an inventory count, a coverage ratio built on the wrong payment schedule, acres never reconciled to the schedule of insurance, a lapse date taken from a rule rather than a filing.

What it obliges: a conversation with credit risk about last season, not a mark against the vendor. This is the finding that justifies the exercise whatever you decide about the software.

It was a judgement call

A yield assumption inside a defensible range, whether a lease renews, how much of a crop a borrower will have contracted by March.

What it obliges: record it and exclude it from the measures. Then check the system showed its basis clearly enough for the officer to disagree knowingly.

Our framework for adjudication. Only the first column counts against the software; the second is the pilot's most valuable output, and the third should be recorded rather than resolved.

Two scoring rules keep the exercise honest. A near-miss is not a match: a flag that says coverage is thin without naming the ratio and the minimum has not found the exception. And a figure that is right but unsourced is scored as unsourced, because an approver cannot act on it and an examiner cannot check it.

What SR 26-2 says about outcomes analysis and back-testing

The agencies' revised model risk management guidance is the best short description of this method that a bank can point a committee at, with one caveat to state up front. SR 26-2 — issued jointly by the Federal Reserve, the OCC and the FDIC in April 2026 — puts generative and agentic AI outside its own scope in a footnote, while saying an organisation's risk management and governance practices should still determine appropriate controls for tools it does not cover. So what follows is a method worth borrowing, not a requirement being met.

On what outcomes analysis is, the guidance is plain: it "compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use" (SR 26-2). On its forms, it is broader than most banks assume: "outcomes analysis can take many forms, including testing conducted during model development, reports or analysis performed as part of ongoing monitoring, or standalone activities such as back-testing or outlier analysis." A pilot on closed files is the standalone back-testing case.

Two further sentences earn their place in a pilot report. The guidance notes that outcomes analysis "may identify material errors or persistent deviations outside of the banking organization's established performance thresholds", in which case "model adjustment, recalibration, or redevelopment may be warranted" — which is why thresholds belong in week 0 rather than in the discussion of the results. And on exactly the territory a renewal occupies: "when a model's design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment." A farm renewal is expert judgement from end to end. The measurement is how you find out how good yours has been.

What a pilot cannot test: the season's document chase

A back-test holds every document still, and that is its blind spot. Four things stay untested until a real season.

The chase is the first. On a closed file every document is present; the whole question of whether a borrower responds to a drafted request, whether the accountant sends the statement in January instead of February, whether an insurance agent answers, is invisible. In the sample season a renewal package opened in October and filled over twenty weeks — that is the behaviour that shortens a season, and no back-test can show it. Test it in the parallel season, on a slice of the book.

The connections are the second. Loan servicing, lien search and insurance feeds differ by bank, and some may be simulated in any system you evaluate. Ask directly which are live, which a pilot would have to stand up, and which would remain a manual step on day one.

The people are the third. Whether an officer trusts a flag enough to act on it, and whether an approver reads a memo in the new shape, is a question about habits. And the fourth is the unusual year: a back-test over three normal seasons says nothing about a drought, a price collapse or a change to the price deck mid-season.

Where a 24-file pilot's own hours go, in our suggested plan

Scoring, line by line (24 files)
18 h
Loading your policy, deck and memo template
12 h
Adjudication sessions with the officer
12 h
Fixing the sample, scoring sheet and thresholds
8 h
Writing the result and the decision
6 h
Running the files and collecting output
4 h
Our suggested budget, not a measured one: about 60 hours of your own people's time for 24 closed renewals. At a loaded hourly cost of the kind used in the cost guide, that is roughly $3,900 of bank time — worth naming in the business case rather than discovering.

Sixty hours is the honest price of a defensible answer, and it is cheap against a season. What it buys is not only a verdict on the software: it is a measured baseline of your own desk, a list of what last season got wrong, and a scoring sheet you can re-run whenever something changes.

How long does an ag lending software pilot take?

Four working weeks for the back-test, on the plan above, and one renewal season for the parallel run. The constraint is rarely the software; it is getting twenty-four files reconstructed as they stood at renewal and getting the officer who signed them into a room three times.

Work backwards from the season rather than forwards from today. If your operating lines mature between mid-February and 1 April, the back-test wants to be finished by the end of October, because the settings work, the connections and any configuration then have the autumn — and because a renewal package that is supposed to open in October has to exist in October. A pilot that finishes in January has proved something true about a season it is too late to change.

Two scheduling rules save a lot of grief. Do not run a pilot across your own busiest eight weeks: the people who have to adjudicate the disagreements are the people renewing the book. And do not let the parallel season be a half-season — run the slice of the book you choose all the way to approval, including the memo and the signatures, because the approval step is where a new shape of memo either works or does not.

The deliverable a pilot is really scoring in Agricultural Loan Renewal Management

The renewal memo screen: the request, what last year did against its plan, the carryover and its treatment, this year's plan and repayment, insurance and collateral, the policy exception with its mitigants and the recommendation — every figure numbered back to the document it came from, with proposed terms from the pricing grid and conditions that can be switched on and off.
Fictional sample file in the solution. Score the memo, not the screens: it is what the approver reads, what the file is judged on, and the only place the pilot's four questions all come together.

What to agree before go-live

A pilot that ends in a decision and nothing else has thrown away half of what it produced. Six things belong in writing before the first real renewal is prepared, and all six are easier to agree while the results are fresh.

What to settle in writing between the pilot and go-live

  1. The thresholds you measured against, and what happens if they slip

    Written in week 0, re-stated at go-live. Name who is told and what changes if a measure falls below them in the live season.

  2. What re-starts the measurement

    A settings change such as a new price deck or policy minimum, a new document source, and the turn of the season. Put all three in the procedure.

  3. Who owns the settings, and how changes are versioned

    The price deck belongs to whoever sets it today. Confirm a change applies to files prepared from then on, not retrospectively.

  4. The list of what the system never does alone

    Send to a borrower, file a continuation, approve a renewal, price outside the grid, estimate a missing figure. In writing, from the vendor.

  5. Which connections are live on day one

    And what the manual step is for each that is not. This is the most common gap between a successful pilot and a disappointing first season.

  6. Who measures it afterwards, and how often

    Independent of the ag lending department whose season gets shorter, with loan volume out of their incentives — the same rule the OCC sets for loan review.

  7. What the parallel season covers, and when it ends

    Which slice of the book, through to approval and signature, and the date you stop running both ways.

One finding from the pilot deserves to outlive it. Where the old file turned out to be wrong, the bank has learned something about its own renewals that no vendor decision changes — and the OCC's handbook is a reminder of what is at stake in those files: "loans to sound borrowers that are refinanced or renewed in accordance with prudent underwriting standards should not be classified unless well-defined weaknesses exist that jeopardize repayment" (Comptroller's Handbook: Agricultural Lending). The protection in that sentence is conditional on the standards, and the standards are evidenced by the file. A renewal whose analysis cannot be produced does not get the benefit of it.

For the measures to score a pilot against, see how accurate AI is at farm loan underwriting; for the cost side of the same decision, the real cost of a farm loan renewal; and for what you would be comparing against, the three ways to run a renewal season. To see a prepared renewal and its settings for yourself, start from Agricultural Loan Renewal Management.

Questions people ask

How do you pilot ag lending software?

Run it on twenty to thirty renewal files your own officer has already closed and signed, with the documents as they stood at renewal and the officer's memo sealed until scoring. Fix the sample, the figures you will score and the thresholds before anything runs. Then score every figure and exception, and adjudicate each disagreement as the system's error, the old file's error, or a judgement call.

How many files should a software pilot cover?

Twenty to thirty closed renewals. Five will all be clean files; after about two dozen the marginal file stops teaching anything. Weight the sample towards the files bank examiners themselves look for — loans with multiple extensions, carryover that was not adversely rated, significant policy or underwriting exceptions, and modified repayment terms — plus your ordinary spread of maturity months, commodities and tenure.

What should you bring to a loan software pilot?

Your credit policy as settings, your price deck and stress case, your memo template and pricing grid, and the files as they stood at renewal. A pilot run on a vendor's defaults measures the vendor's defaults. Change one rule while you watch, so you learn early whether a policy change is a setting, a change request or a release.

What counts as a disagreement in a pilot?

Any decision figure where the system and the closed file differ, or any exception one raised and the other did not. It is not a failure until it is classified. Three categories cover almost all of them: the system was wrong, the old file was wrong, or it was a judgement call inside a defensible range. Only the first counts against the software.

How long does an ag lending software pilot take?

About four working weeks for the back-test, plus one renewal season run in parallel on a slice of the book. Work backwards from your maturity window: if lines mature between mid-February and 1 April, finish the back-test by the end of October, so the settings, connections and configuration have the autumn and the first renewal packages can open on time.

Does supervisory guidance require back-testing of AI underwriting?

Not for agentic systems. SR 26-2, the agencies' revised model risk management guidance, places generative and agentic AI outside its scope in a footnote while expecting an organisation's own risk management practices to determine the controls. Its description of outcomes analysis — comparing outputs to real-world outcomes, including standalone back-testing — remains the clearest method to borrow, and it notes that when a design relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment.

Agricultural Loan Renewal Management.
See it on your own files.

An agentic solution from Botminds. Every operating line due before planting, prepared by agents and lined up by date, so officers spend the season deciding, not assembling. Book a demo, or explore how it works.