How accurate is AI at agricultural loan underwriting? Nobody can answer that with one percentage, and a vendor who offers one is answering a different question. Accuracy on a farm renewal splits into four measures that move independently: how often a figure the agents wrote matches the figure a credit analyst reads off the same document, how many figures carry no source at all, how many of the exceptions the file really contained went unlisted, and whether each citation actually holds the number it is attached to. Botminds' Agricultural Loan Renewal Management prepares every farm operating line for renewal — the spreads, the carryover, this year's cash flow at the bank's price deck, insurance and lien filings, and a drafted memo — and numbers every figure in that memo back to the statement, crop plan, schedule of insurance or filing it came from. That citation is what makes the four measures testable at all.
The hard part is not running the test. It is deciding what counts as a wrong answer when the agents and the old file disagree, and resisting the temptation to collapse four different kinds of error into one headline percentage that hides all of them. The worked examples below use the fictional sample season in the solution — invented farms, an invented bank and invented people — and the research is from regulators and standards bodies, linked under each figure.
Accuracy is "closeness of results of observations, computations, or estimates to the true values or the values accepted as being true."
NIST, AI Risk Management Framework 1.0, quoting ISO/IEC TS 5723:2022
How accurate is AI at agricultural loan underwriting?
Read that definition twice, because it contains the whole problem. Accuracy is closeness to a true value or to a value accepted as being true. On a farm renewal those two are not the same thing. The true value of the corn still in a borrower's bins on 18 January is whatever a count and a local cash bid say it is. The accepted value is whatever last year's officer wrote in the file — and sometimes the officer never counted, and wrote the whole year-end balance down as carryover. A measurement that scores the agents against the old file is scoring them against the accepted value, which is the right place to start and the wrong place to stop.
So the useful answer to the question is procedural rather than numerical. On work that cites its own sources, accuracy is measured by taking renewal files your own officer has already closed and signed, running the agents over the same documents, and reading every disagreement one at a time until each is classified: the agents were wrong, the old file was wrong, or the two made a judgement call differently. Each of those three has a different remedy, and lumping them together produces a number that cannot be acted on.
The bar is also not "better than nothing". A renewal memo is the document an approver and later an examiner reads, and for the bank's purposes a figure with no source is as unusable as a figure that is wrong. That is why the unsourced rate below sits beside the match rate rather than underneath it.
Why general AI accuracy studies do not transfer to this work
Most published AI accuracy figures measure a model answering from what it has learned. A renewal does the opposite: every figure is supposed to come from a document in front of it, and the document is in the file to be re-read. Those are different tasks with different failure modes, and a percentage earned on one tells you very little about the other.
The framework most US banks reach for says as much about its own limits. NIST notes that "the current lack of consensus on robust and verifiable measurement methods for risk and trustworthiness, and applicability to different AI use cases, is an AI risk measurement challenge", and warns that "measurement approaches can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts" (NIST AI 100-1). A single accuracy number for an eight-week renewal season is the textbook example of the first failure and an invitation to the second.
NIST also separates the laboratory from the desk: "while measuring AI risks in a laboratory or a controlled environment may yield important insights pre-deployment, these measurements may differ from risks that emerge in operational, real-world settings." For a renewal the gap between those two settings is large and specific. A clean test corpus has legible statements and complete crop plans. A February file has an accountant's statement that arrived as a photograph, a crop plan that lists acres the lease no longer covers, and a bin count somebody took on a cold morning.
There is a third mismatch, and it is the one that catches buyers. NIST observes that "risk metrics or methodologies used by the organization developing the AI system may not align with" those used by the organization deploying it. A builder measuring field-level extraction on page images and a bank measuring whether a renewal memo can be approved are both measuring accuracy, honestly, and getting numbers that have nothing to say to each other.
Four measures of accuracy for Agricultural Loan Renewal Management, and the formula for each
| Measure | The question it asks | How to compute it | How to read the result |
|---|---|---|---|
| Match rate | Of the figures the agents wrote into the spread, the cash flow and the memo, what share equals what a person reading the same document writes? | figures that agree ÷ figures checked | Reported per section, never as one number. Spreads and the cash flow move together; carryover and insurance do not. |
| Unsourced rate | What share of the figures carry no citation to a document at all? | figures with no cited source ÷ figures checked | Zero. An unsourced figure is a defect whether or not it happens to be right, because nobody can check it. |
| Missed-flag rate | Of the exceptions the file really contained, what share did the agents fail to list? | 1 − (exceptions listed ÷ exceptions present) | Judged against the exceptions the closed file proves, including the ones the officer wrote in by hand. |
| Citation accuracy | Of the figures that do carry a citation, what share of those citations actually contain the figure? | citations that support their figure ÷ citations checked | Tested by opening the cited page, not by trusting the reference. A right number with a wrong source is still wrong. |
Four measures: match rate, unsourced rate, missed-flag rate, citation accuracy
Take them in the order a reviewer will care about them. The match rate is the one everybody asks for, and it needs a denominator before it means anything. "Ninety-six per cent accurate" over a renewal file could mean ninety-six per cent of several hundred spread line items, where a run of zero-value rows carries the score, or ninety-six per cent of the fifteen figures the memo's recommendation actually turns on. Count the second kind. On a renewal those are the year-end balance, the inventory and its valuation, last year's yields and realised prices, the planned acres by crop and lease, the input budget, the term payment schedule, the insured acres and coverage level, and the lien filing dates.
The unsourced rate is the measure a credit function should set to zero and hold there. A figure with no citation cannot be checked by the approver, cannot be defended to an examiner, and cannot be scored by this exercise at all. It is worth separating two causes: a figure the system estimated because the document was missing, and a figure it read correctly but failed to attach to a page. The first is a policy failure and the second is a plumbing failure, and the fix for each is different.
The missed-flag rate is harder to measure and more valuable than the match rate, because it is measured against what a file contained rather than what a system reported. Closed files make it possible: the conditions the officer added by hand, the exception memo the manager signed, the continuation that loan administration filed in March are all evidence of exceptions that were really there. Score the agents on whether each one appears in their output, and resist counting a near-miss as a hit. A flag that says coverage is thin without naming the ratio and the minimum it breached has not found the exception; it has found the neighbourhood.
The citation accuracy measure is the cheapest to run and the one most often skipped. Take a sample of figures that do carry a source, open the cited page, and check that the number is on it. A right figure with a wrong citation looks like success on the match rate and fails the only test an approver applies. This is also where a long document hurts: page thirty-one of an accountant's statement and page thirteen of last year's are easy to confuse and impossible to confuse quietly, because the citation is in the memo for anyone to open.
What NIST's AI Risk Management Framework says about measuring accuracy
NIST's framework is not a testing standard, and reading it as one leads to disappointment. What it does give a bank is a short list of conditions a measurement has to satisfy before the resulting number means anything. The first is the test set: "accuracy measurements should always be paired with clearly defined and realistic test sets — that are representative of conditions of expected use — and details about test methodology; these should be included in associated documentation" (NIST AI 100-1).
For a renewal season, "representative of conditions of expected use" has a concrete meaning. If three-quarters of your operating lines are row-crop farms with cash-rent leases, a test built on your four largest diversified livestock operations is not representative, however much attention those four get in a credit committee. Build the sample the way the season is: by maturity month, by commodity, by whether there was carryover last year, and including the two or three files that went badly, because those are the files where the measure earns its keep.
The framework's second condition is that the method travels with the number. Its measurement function asks that "test sets, metrics, and details about the tools used during TEVV are documented" — TEVV being NIST's shorthand for test, evaluation, verification and validation. In practice that means writing down, before you run anything, which figures are in the denominator, who decides a disagreement, and what counts as a missed exception. Decide those afterwards and the result is a negotiation, not a measurement.
The third condition is who does the measuring, and it is the one banks already have machinery for. NIST asks that "internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates". A bank's own version of that rule is older and sharper, and it is in the next section.
How to back-test Agricultural Loan Renewal Management against a renewal you have already closed
A person decides
Credit risk picks the filesDraws a sample of closed, signed renewals across maturity months, commodities and carryover history, including the two that went badly.
Never: Never lets the vendor choose the sample
A person decides
The reviewer fixes the denominatorLists the figures the recommendation turns on and the exceptions the closed file proves, before anything is run.
Never: Never changes the list after seeing results
Agent
The agents read the same documentsStatements, crop plan, schedule of insurance and lien search are read from the file as it stood at renewal, and a memo is drafted.
Never: Never sees the officer’s memo or conditions
Agent
Each figure is citedEvery number in the draft carries the document and page it came from, so a reviewer can open the source rather than trust the output.
Never: Never estimates a figure a document does not give
A person decides
A reviewer scores line by lineMarks each figure agree, disagree or unsourced; marks each proven exception listed or missed; opens a sample of citations.
Never: Never scores a near-miss as a hit
A person decides
Credit risk adjudicates disagreementsClassifies each one: the agents were wrong, the old file was wrong, or it was a judgement call. Only the first counts against the system.
Never: Never blends the three into one percentage
How do you test AI underwriting accuracy on files you already closed?
Closed files are the only test set a bank already owns in quantity, and they have a property no vendor demonstration has: somebody has already done the work and signed their name to it. The sequence above is not complicated; what makes it work is the order. The sample and the denominator are fixed before the first run, because both become arguable the moment results exist.
Two practical rules save most of the arguing. First, run the agents on the file as it stood at renewal, not as it stands now. A file that has since collected a continuation, a signed assignment and a year of payment history will make any system look prescient. Second, withhold the officer's memo and conditions until scoring, so that the agents' output and the officer's work are genuinely independent before they are compared.
Expect a real share of the disagreements to be the old file's. That is not a flattering finding for the system under test; it is the most useful thing the exercise produces. A carryover figure that nobody reduced by the value of grain in the bins, a coverage ratio that counted a term payment twice, an insurance schedule that nobody matched to the crop plan acre by acre: each is a defect in last season's work that this season's would have repeated. The point of back-testing a renewal is partly to grade the software and partly to find out what the desk has been getting wrong.
What the agents hand back on one renewal, with the evidence for each flag

What a flag's evidence should include, and how to score it
A flag is not a measurement unless it comes with enough evidence to be checked, and this is where scoring gets interesting. On the sample file above, all four flags were right. Two of them could have been right for the wrong reason, which a scoring sheet that only records right or wrong would never discover.
The four flags on one sample renewal: what evidence each needs, and the way each could be hollow
| Flag | The evidence it must carry | How it can be right and still fail |
|---|---|---|
| Carryover on the operating line | The year-end balance from the statement, the bin count with its date, and the cash bid used to value the grain | A balance called carryover with no inventory count behind it — the number is right and the conclusion is not |
| Coverage below the policy minimum | The capacity figure, the schedule of term payments it is divided by, and the price deck the revenue was built at | A ratio quoted without the payment schedule it used, so nobody can see which debts were counted |
| A lien filing that lapses in March | The financing statement number, its filing date and the lapse date read from the filing office record | A lapse date inferred from a five-year rule rather than read from the filing |
| Rented acres not on the insurance | The acre count in the crop plan, the acre count on the schedule of insurance, and the farm each belongs to | A gap stated in total acres without naming the farm, which cannot be fixed by anyone |
The carryover flag is the clearest case. The sample file's operating line stood at a six-figure balance on 31 December, and the agents named only part of it as carryover, because a bin count taken in January valued the unsold corn that made up the rest. A system that called the whole balance carryover would have produced a defensible-looking flag, a wrong conclusion and a borrower conversation that did not need to happen. The arithmetic that separates stored grain from real carryover is worked through in full elsewhere; for scoring purposes, what matters is that the inventory count has a date and a price source attached to it.
The coverage flag works the same way. A ratio is only checkable if the payment schedule it was divided by is on the page, which is also the difference between a flag an approver can act on and a flag that starts an argument. The term debt coverage ratio and the other farm financial measures set out which payments belong in that denominator.
What happens when an AI agent is unsure, or a figure is missing
Most of the published worry about AI in credit is about confident wrong answers. On a renewal the more common event is a document that has not arrived, and how a system behaves then is a design choice, not an accuracy result. In this solution the rule is explicit: the statement reader never estimates a missing figure, it lists it as missing, and the insurance checker never counts unassigned insurance as security. A missing figure therefore surfaces as a gap in the file and a request to the borrower, not as a plausible number in the memo.
That turns "unsure" into something a measurement can see, which is the only reason the unsourced rate can be held at zero. It also means a back-test has to score three outcomes per figure, not two: present and cited, present and uncited, or correctly reported as missing. A system that silently fills gaps scores better on a naive match rate and worse on every measure a credit function cares about, because the figures it invents are indistinguishable from the ones it read.
One illustrative scorecard: the same disagreement count, read two ways
The one number
94%
1,128 of 1,200 figures agreed with the closed file. Quoted alone, it says the system is fine and the desk should buy it.
The same data, four measures
- Match rate 97.8% — of the 72 disagreements, 39 were the old file's error and 6 were judgement calls, leaving 27 against the system
- Unsourced rate 0% — no figure reached the memo without a document behind it; 14 were reported as missing instead
- Missed-flag rate 5.6% — 170 of 180 proven exceptions listed; the 10 missed were all lien lapse dates on one filing office's records
- Citation accuracy 99.2% — 2 of 250 sampled citations pointed at the prior year's statement
Read the right-hand column and the decision changes shape. The system is more accurate than the blended figure suggested and has one concentrated weakness worth fixing before go-live, in a single data source. That is an actionable result. "Ninety-four per cent" is not.
How often should AI underwriting be back-tested, and by whom?
Banks already have the answer to the second half of that question, and it predates AI by decades. The OCC's agricultural lending handbook tells boards to "establish audit and loan review functions that are independent of Ag loan production, approval, and credit administration functions", and adds that "the compensation program for auditors and loan review personnel should not include incentives for Ag loan volume generated by the lending department" (Comptroller's Handbook: Agricultural Lending). The function that scores a renewal-preparation system should sit in exactly that place: not in the ag lending department whose season it is meant to shorten, and not with the people who configured it.
On frequency, the honest answer is that it depends on what has changed. Three triggers are worth writing into the procedure. A settings change — a new price deck, a different policy minimum, a different rule for valuing stored grain — changes what the agents are being asked to do, and in this solution those changes are versioned and apply only to files prepared afterwards, which makes a before-and-after comparison possible rather than theoretical. A new document source, such as a different accountant's statement format or a new filing-office record, is the second. And the turn of the season is the third: the sample that represented your book last February will not represent it after a year of commodity prices moving.
Between those, the measure that earns continuous attention is the missed-flag rate, because it is the one whose failures cost the bank money rather than time. NIST's framing is that "validity and reliability for deployed AI systems are often assessed by ongoing testing or monitoring that confirms a system is performing as intended" — on a renewal desk, that monitoring is mostly the record of how often an officer added a condition the agents had not raised.
Where accuracy is read in Agricultural Loan Renewal Management: a memo whose figures cite their documents

What accuracy still does not cover: judgement
Every measure on this page scores assembly: reading a document, carrying a number, matching an acre, reading a date. None of them scores the decision. Whether a borrower who missed last year's yields by seven bushels and sold at fifty cents under plan should get a larger line this year is a judgement about a family, a landlord, a lease and a season, and no match rate speaks to it.
The regulator's view of a weak renewal is about documentation rather than arithmetic. The OCC's handbook warns that "when loan restructurings are not supported by adequate analysis and documentation, however, examiners are expected to exercise reasonable judgment in reviewing and determining loan classifications until bank management is able to provide information to support its conclusions and credit risk ratings" (Comptroller's Handbook: Agricultural Lending). Read against a renewal season, that is a statement about where the risk actually sits. A file loses its argument because the analysis behind it cannot be produced, far more often than because a figure in it was wrong by a hundred dollars.
Which is why the four measures matter and also why they are not the whole case. They tell you whether the assembly can be trusted, so the officer's eight weeks can go on the judgement instead. In this solution the division is fixed rather than configurable: the agents prepare, the ag loan officer recommends and signs with the meaning of the signature recorded, and above the officer's authority or with any policy exception the Ag Lending Manager approves. What each of the six agents may and may not do is set out separately.
What to ask a vendor to show you about accuracy, before a pilot
- Which figures are in your denominator?
Ask for the list. If spread line items and the fifteen figures a recommendation turns on are counted alike, the headline rate is carried by the easy ones.
- Show me the unsourced rate separately.
And how the system behaves when a document is missing: a figure estimated, or a gap reported and a request drafted.
- How is a missed exception counted?
Against the exceptions a closed file proves, including conditions the officer added by hand — not against the exceptions the system chose to look for.
- Can I open the source for any figure, in one click?
Citation accuracy cannot be tested on a system whose evidence lives somewhere a reviewer cannot reach.
- Who chose the test files, and were any excluded?
A sample drawn by the vendor from files the vendor has seen is a demonstration, not a measurement.
- What happened to the disagreements you lost?
Ask for the adjudication, not the summary: how many were the old file’s error, how many were judgement, how many were the system’s.
- What changes invalidate the result?
A new price deck, a changed policy minimum, a new document source. Ask what is versioned and what is not.
Running that list against your own closed files is the next step, and it has its own method: how to pilot ag lending software on your own renewal files covers the sample size, the scoring sheet and what to agree before go-live. To see the citations on a prepared renewal for yourself, start from Agricultural Loan Renewal Management.
Questions people ask
How accurate is AI at agricultural loan underwriting?
There is no single published figure, and any vendor quoting one should be asked for its denominator. Accuracy on a renewal is four separate measures — match rate against a person reading the same document, the share of figures with no source, the share of real exceptions missed, and whether each citation holds its figure — and they move independently. The only credible number is the one your own credit function produces on your own closed files.
Can AI invent a figure on a loan file?
It can, which is why the behaviour on a missing document is a design question rather than an accuracy result. In this solution the statement reader never estimates a missing figure; it lists it as missing, and the missing-items requester drafts a plain request to the borrower for the officer to send. A back-test should score three outcomes per figure — cited, uncited, or correctly reported as missing — so a system that quietly fills gaps cannot score well.
What is a match rate, and what is a good one?
A match rate is the share of figures that agree with what a person reads off the same document. There is no published benchmark for farm renewals, and the figure is meaningless without its denominator: a rate over every spread line item flatters a system, while a rate over the fifteen figures a recommendation turns on tests it. Report it per section of the memo and publish the denominator beside it.
How often should AI underwriting be back-tested?
No supervisory guidance sets a frequency for agentic systems. Three triggers are defensible: any change to the settings the renewals are prepared against, such as the price deck or a policy minimum; any new document source; and the turn of the season, because last year's sample no longer represents your book. Between those, watch how often an officer adds a condition the agents did not raise.
Does the Federal Reserve's model risk guidance apply to this?
Not directly. The agencies' revised model risk management guidance, SR 26-2, puts generative and agentic AI outside its own scope in a footnote, while saying that an organisation's existing risk management and governance practices should still determine the controls for tools it does not cover. Its sections on outcomes analysis remain the best-written description of what a back-test should look like; what SR 26-2 says about outcomes analysis is set out in the pilot guide.
Who inside a bank should measure this?
Someone independent of the department whose season gets shorter. The OCC's agricultural lending handbook asks boards to keep audit and loan review independent of Ag loan production, approval and credit administration, and to keep loan volume out of those reviewers' incentives. NIST asks the same thing in different words: assessors who were not front-line developers of the system.