AI in loan servicing is, today, mostly reading: finding the clause, assembling the term as amended, drafting the answer and checking it, while people keep the decisions and the sending. A pilot is how a servicing or agency desk finds out whether that reading is good enough on its own documents — and most pilots fail to answer the question, not because the software disappoints but because nobody decided in advance what would count as a pass. This guide sets out a four-week trial built as a replay: thirty questions your desk has already answered, re-run on your own executed files and scored against the replies that actually went out. It is written around Botminds' Loan Document Search and the screens are its own, but the design works for any tool you are considering.
The sample figures on this page come from a fictional agency book of 212 facilities. The supervisory material is quoted and linked where it appears; nothing here is advice about your own regulatory obligations.
How is AI used in loan servicing today?
Usefully narrowly, and that is the honest starting point. The US Treasury's survey of the industry records that "respondents noted that financial firms are increasingly using AI — and particularly experimenting with Generative AI — for internal business operations, including but not limited to risk management, regulatory compliance, treasury management, fraud detection, and back-office functions" (Treasury, December 2024). The OCC's view of where banks have actually got to is narrower still: "to date, the OCC has observed that use cases are primarily productivity and customer experience enhancement tools" (OCC, Semiannual Risk Perspective, Spring 2026).
On a loan servicing desk that resolves into four things worth piloting and a long tail that is not. Reading the loan file: splitting executed documents into sections with pages and defined terms. Answering document questions: a short reply with the clause behind each sentence. Reconciling terms: comparing what the servicing system bills from with what the documents say. Drafting correspondence: the reply itself, for a person to review and send. Notice that none of those is a decision. The GAO's review of AI in financial services frames the upside in general terms — "benefits can include improved efficiency, reduced costs, and enhanced customer experience" (GAO-25-107197) — and on this desk the efficiency is specifically the hours before the answer, not the answer itself.
What a pilot has to prove, and in what order
A pilot that sets out to prove "the tool works" proves nothing, because that claim has no failing case. Decide instead on four propositions, in order, and stop early if one fails: the tool reads the right documents; its answers are supported by the clauses they name; it knows what the loan said on a date; and it is faster end to end than what you do now, counting the review. The order matters because a failure at any level makes the levels below it meaningless — a speed figure for a tool that cites superseded clauses is a measure of how fast it can be wrong.
Two pilots of the same tool, and why only one produces a decision
The demonstration pilot: Whatever anyone thinks of during the session, answered live.
The replay pilot: Thirty questions the desk closed in the last quarter, with the replies that were sent.
The demonstration pilot: A sample file, often a clean agreement with no amendments.
The replay pilot: Your own executed files for the facilities those questions were about, amendments and all.
The demonstration pilot: Impressions, collected afterwards, by people who watched.
The replay pilot: Sentence by sentence against the clause, marked before anyone discusses it, by someone who knows the file.
The demonstration pilot: None that anybody wrote down.
The replay pilot: Agreed in advance: what share of sentences must be clause-supported, how the as-of pairs must behave, what silence must look like.
The demonstration pilot: "The team was impressed."
The replay pilot: "On thirty closed questions it matched the sent reply in this many, differed in these ways, and took this long including review."
The replay design has one more property worth paying for: it produces a regression suite. The thirty marked questions and their answers are reusable the next time a version changes, which turns a one-off purchase decision into a repeatable check.
What you bring: documents, questions and the answers already sent
Everything needed is already inside the bank. Assembling it is a morning's work for someone who knows the book, and it is the step most often underestimated, because the replies that were sent usually live in individual mailboxes rather than anywhere searchable.
The day-one list, and why each item is on it
- Executed agreements and every amendment, for ten to fifteen facilities
Pick facilities with real amendment histories rather than your cleanest files. At least two should have the same section amended more than once, because that is what the as-of test needs.
- Fee letters, side letters and notices from the same files
They carry terms that the credit agreement does not, and they also carry the access question: who in the bank is allowed to see them.
- Read-only access to the terms your servicing system bills from
Read-only is not a limitation of the trial, it is the design. The comparison is worth running from day one; nothing should write back.
- Your servicing procedures manual
Used for how the desk works, never for what a loan says. Keeping that line explicit from the start prevents a procedure being quoted back as if it were a term.
- Thirty closed questions with the replies that were sent
Real questions from borrowers, lenders and relationship managers, with dates. Include a few you got wrong or had to correct: those are the most informative of all.
- Five questions the documents do not answer
Chosen so that a plausible general answer exists. This is the test nobody volunteers and the one most likely to change your mind.
- Two documents that must never be cited
An unexecuted draft amendment and a superseded conformed copy, left in the file. If either is ever quoted as the agreement, week one has failed.
Week one: the loan files, and what counts as executed
Week one is about documents, not answers. Load the files, then spend the time on the one question that determines everything downstream: does the system know which documents were executed, and does it keep the page and section of every passage it reads?
The examiner's own loan administration checks are a good source of week-one tests, because they are the checks a reviewer would run on the file by hand. The OCC's handbook has examiners "examine notes for completeness and reconcile date, amount, and terms to trial balance" and asks about "monitoring trailing document exceptions" (OCC, Lending and Loan Portfolio Risk Management, July 2026). Translated into a pilot: for each facility, confirm the file is complete, confirm the parties, dates and amounts read out of the documents match what the system holds, and get a list of what is missing. A tool that produces a credible missing-documents list in week one has already paid for part of the trial.
The book after week one: what was read, and where the system already disagrees
Two week-one failures are worth naming because they are cheap to detect and expensive to discover later. The first is a tool that indexes the whole file and treats a draft as evidence; test it with the two seeded documents. The second is a tool that reads text but not pagination, so an answer can name a section and not show you the page it came from. Neither is fixable by tuning.
Week two: thirty closed questions, scored against what was sent
This is the week that produces the number. Run each of the thirty questions, print the answer, and mark it with the agreement open — sentence by sentence against the clause it names, then as a whole against the reply that was actually sent. Use three marks and no more: matches the sent reply in substance; differs, with the difference written in one line; no answer, where the tool declined or hedged.
Mark before you discuss. The strongest bias in these trials is the tendency to read a confident answer as correct because it is well presented, and the only defence is a marker who has the clause in front of them and a form to fill in. Have a second person re-mark a sample of ten; where the two disagree, the question usually turns out to be ambiguous, which is itself a finding.
Expect the differences to cluster rather than scatter. A tool that gets a term wrong on one facility usually gets the same kind of term wrong on others — an amendment effective date read from the signature page, a defined term resolved from the original rather than the amended text. Grouping the differences by cause rather than counting them is what turns week two into a decision instead of a score. The five questions the documents do not answer are marked separately: a pass is an explicit statement that they are not addressed, plus the nearest clause. Those five are the ones to look at first, because a well-cited answer to a question the file does not answer is the failure that a busy desk will never catch on its own.
Where a week-two answer goes when it is accepted
Week three: the loan system check, and the first corrections
Week three turns the trial from a reading exercise into something with money attached. Point the comparison at the terms your servicing system bills from and let it run overnight: margin, maturity, amortisation, commitment and utilisation fees, soft call windows, floors, sublimits. Then take each difference and confirm it by hand against the executed document. The count that matters is confirmations, not flags.
On the fictional sample book the first night produced nine differences across eight facilities out of 2,968 terms compared, each presented with the system value, the amended term, the clause behind it and what the difference costs. Two were on one borrower — a soft call window that had been reset by Amendment No. 3 and a maturity extended by a year — and the costliest was a margin billed at SOFR plus 4.25% when the second-quarter pricing certificate had moved it to 4.00%, worth about $180,000 a quarter. Those figures are sample data, not a measured result at a bank; your own first night is the point of the exercise.
The first night's output, and the column that makes it a business case
Two rules keep week three clean. Confirm every difference against the executed document before it is counted, because an unconfirmed flag is a cost, not a benefit. And route every correction to the named approver even during the trial: a pilot that changes the servicing system informally has destroyed the control it was supposed to be testing. On the sample book every difference goes to the loan operations supervisor, who approves each correction before it is made; nothing automated touches the system.
Week four: the decision, and the numbers that carry it
Week four is writing, not testing. Produce one page with six numbers and the method behind each, and bring the marked answers as the appendix. A committee will accept a modest result that is clearly measured far more readily than a strong result that cannot be explained.
The decision sheet: six numbers, and how each was produced
| Number | How it was produced | What it decides |
|---|---|---|
| Questions matching the sent reply, out of 30 | Marked with the agreement open, before discussion, by someone who knows the file. | Whether the tool can do the work at all. Read the differences by cause, not as a score. |
| Share of sentences supported by the clause they name | Every sentence of the thirty answers, opened and checked. | Whether the answers are reviewable. This is the number second line will ask for. |
| As-of pairs correct, out of the pairs tested | The same question asked either side of an amendment, on the facilities chosen for it. | Whether the tool understands that a credit agreement has a history. A single miss is structural. |
| Silent questions handled, out of 5 | The five questions the documents do not answer. | Whether it will invent an answer when nobody is watching. |
| Minutes to a checked answer, against your baseline | Timed on the same thirty questions, including the officer’s review, against your own current time measured the same way. | The efficiency case. Measure both ends yourself; do not accept a vendor’s figure for either. |
| Differences confirmed, and what they are worth | Week three’s flags, each confirmed against the executed document, priced where a price is calculable. | Usually the part of the business case that survives scrutiny, because it is money the bank is billing wrongly today. |
Where the desk's own effort goes across the four weeks
Estimated share of the pilot team’s time
- Week 1 · files and document checks · 30%
- Week 2 · running and marking the thirty · 35%
- Week 3 · confirming differences · 20%
- Week 4 · writing it up · 15%
The four weeks, and what has to exist by each Friday
- Week 1 · the filesDocuments list
- Week 2 · the thirtyMarked answers
- Week 3 · the termsDifferences
- Week 4 · the decisionDecision sheet
What supervisors expect while you are piloting
There is no approval to obtain and no form to file, and the supervisory posture is more welcoming than most project sponsors assume:
The OCC supports banks' efforts to integrate AI into core functions, while managing the risk in a safe and sound manner and in compliance with applicable laws and regulations.
OCC, Semiannual Risk Perspective, Spring 2026
Two practical consequences for a trial. First, the absence of a rulebook is not the absence of an expectation: the federal agencies' revised model risk guidance states that it sets no enforceable standards or prescriptive requirements, and that falling short of it will not by itself draw supervisory criticism (OCC Bulletin 2026-13), and the same bulletin places generative and agentic systems outside its scope entirely. What fills the space is your own documented design. The twelve questions worth asking before the trial starts are a reasonable skeleton for that document.
Second, keep the evidence as you go rather than reconstructing it afterwards. The marked answer set, the confirmed difference list, the named approver on each correction and the scope note that says what the tool was and was not allowed to do are what an examiner or an internal auditor will ask for, and they are nearly costless to keep while the work is happening and nearly impossible to recreate three months later.
What not to automate, now or later
A pilot is also the moment to write down the limits, while nobody has a stake in loosening them. Five of them are worth fixing permanently on a servicing desk.
Sending. No automated step should send anything to a borrower or a lender. The reply goes out from the servicing officer's own mailbox after they have read it, which is both the control and the reason the officer stays accountable for its content. Writing to the servicing system. The comparison reads; a named person approves every correction. Deciding a waiver, a consent or a forbearance. These are credit decisions with a governance path of their own, and a document answer is an input to them, not a substitute. Resolving genuine ambiguity. Where two provisions can both be read to apply, the right output is both readings with their clauses and an escalation, never a confident choice. Answering where the file is silent. The most important refusal, and the one that has to survive the first impatient week after go-live.
Those five are the difference between a reading tool and an unsupervised one, and they are also what makes the efficiency real rather than borrowed: the hours saved come from the finding, the assembling and the arithmetic, which is where they always were. How the seats on the desk divide that work is worth reading alongside the pilot design, because the pilot is in the end a test of the handovers as much as of the software.
If the four propositions hold and the decision sheet reads well, the step after the pilot is narrow rather than broad: one product, one team, the same thirty questions kept as the regression check, and a date to re-run them. Breadth is the reward for a second clean quarter, not the first.
Questions people ask
What data do you need to start an AI loan servicing pilot?
Executed agreements and every amendment for ten to fifteen facilities, the fee and side letters from the same files, read-only sight of the terms your servicing system bills from, your servicing procedures, and thirty questions you have already answered with the replies that were sent. No integration, no data migration and no new system of record are needed for four weeks.
How long does an AI pilot take in a bank?
Four weeks of elapsed time for the trial itself, with roughly one person at half time, is enough to produce a decision on a loan servicing use case of this shape. The variable that stretches the calendar is almost never the technology; it is getting the documents released, the read-only access approved and the sent replies out of individual mailboxes.
How do you measure an AI pilot in loan operations?
Against what you do now, on the same work. Match rate on closed questions, share of sentences supported by the clause they name, as-of pairs correct, silent questions handled, minutes to a checked answer against your own timed baseline, and confirmed term differences with their cost. Define each one before the first question is run.
Who should run the pilot?
Someone from the desk who knows the facilities, not a project office. The marking in week two is a reading judgement that cannot be delegated to a person who has never seen the agreement. Loan operations should own week three, and second line should see the design in week one rather than the conclusion in week four.
What if the pilot result is mixed?
Mixed is the common outcome and it is informative if the differences were grouped by cause. A tool that is strong on answers and weak on one class of term is a scoping decision, not a rejection: pilot it again on the subset where it was strong, with the weak class excluded explicitly. A tool that failed the silence test or the as-of test is a different matter, because both are structural.
Does a pilot need the whole loan book?
No, and using the whole book usually makes the result harder to read. Ten to fifteen facilities chosen for their amendment histories give you every behaviour worth testing. The term comparison is the one part that benefits from scale later, because its value is proportional to how many facilities it covers — but it proves itself on fifteen just as well as on two hundred.