An AI contract review tool reads contracts and returns something more useful than a list of files: the clause, a summary, an extracted term, or an answer to a question. Choosing one for a loan book is harder than choosing one for a contract repository, because the thing being reviewed is not one document but a signed agreement plus every amendment since, and because the supervisory guidance a bank would normally lean on for a model purchase says, in terms, that it does not cover this kind of system. That leaves the buyer to design the test. The twelve questions below are tool-neutral — ask them of anyone, including Botminds' Loan Document Search — and every one of them can be answered in an afternoon on your own executed files rather than in a slide.
The screens on this page are a fictional sample agency book. The supervisory sources are quoted and linked where they appear; nothing here is legal or compliance advice about your own obligations.
Three facts to take into the first meeting
$30bn
total assets above which the agencies expect their revised model risk guidance to be most relevant
Out of scope
where that same guidance places generative and agentic systems
12
questions you can put to any vendor and answer on your own agreements in an afternoon
What is an AI contract review tool, and what is it not?
The category is broad enough to be unhelpful without a qualifier, so start by naming which of four things is being sold. A repository with search finds documents and passages. An extraction tool pulls named terms into fields. A drafting assistant proposes language for a contract being negotiated. An answer desk takes a question about an existing agreement and returns a conclusion with the clauses behind it. They are sold with overlapping language and they fail in different ways; only the fourth is doing the job a servicing desk needs.
The US Treasury's survey of AI in financial services puts this family of uses plainly: "Generative AI can automate back-office functions like recordkeeping, predictive texting, transcribing audio, and advanced document searches" (Treasury, December 2024). Advanced document search is the honest description of most of the market. A loan desk wants the step after it.
It is also worth saying what no such tool is. It is not a second opinion on what the agreement means in law, it is not a substitute for the custody of your loan files, and it is not a system of record. If a demonstration implies otherwise, that is the first question answered.
Start with the rulebook: what supervisors say, and what they do not cover
Many banks' instinct is to route an AI purchase through model risk management and let the framework decide. On generative systems that instinct now returns a surprising answer. The federal banking agencies' revised guidance says so directly: generative and agentic AI models, it states, are novel and changing fast, and for that reason they sit outside its scope altogether (OCC Bulletin 2026-13, issued with the Federal Reserve Board and the FDIC, 17 April 2026).
The same bulletin expects the guidance to matter most to banking organisations with more than $30 billion in total assets, and the supervisors have said more is coming: the OCC's Spring 2026 risk report records that "the agencies plan to issue in the near future a request for information that addresses model risk management generally and considers, in particular, banks' use of AI, including genAI, agentic AI, and AI-based models" (OCC, Semiannual Risk Perspective, Spring 2026).
Read correctly, none of that is permission and none of it is prohibition. It is an absence, and the absence is the buyer's problem.
The supervisory map, and what it leaves on your desk
| What the revised guidance does | What follows for a purchase | What the buyer still has to prove |
|---|---|---|
| Places generative and agentic models outside its scope. | You cannot discharge the question by declaring the tool in or out of a model inventory. | A control design of your own, written down before the trial, and evidence it worked. |
| Is expected to be most relevant above $30 billion in total assets. | Smaller banks are not exempt from thinking; they are exempt from a particular rulebook. | A proportionate version of the same evidence, sized to the book. |
| Sets no enforceable standards and no prescriptive requirements. | There is no checklist whose completion ends the argument. | Your own acceptance criteria, agreed with second line before the demonstration. |
| Notes that practices appropriate for one organisation may be inappropriate for another. | A peer’s approach is evidence of what is possible, not of what is sufficient for you. | Why your controls fit your book, your role on the facilities and your risk profile. |
The supervisors have been explicit about what worries them, which is a useful map of where a buyer should concentrate. The OCC "recognizes there are unique challenges related to the use of genAI and agentic AI such as lack of explainability, data privacy and data poisoning issues, cybersecurity threats, and validation challenges where industry approaches are evolving." Three of those five — explainability, data poisoning and validation — are directly addressed by the questions below.
The twelve questions, in the order to ask them
Ask them in this order, because each depends on the ones before it. There is no point asking about dates if the tool cannot tell an executed amendment from a draft.
Twelve questions for any tool, and the answer to listen for
- 1. Which documents does it read, and how does it know which were executed?
A loan file is full of text that reads like the agreement and is not: drafts, markups, term sheets, unsigned signature pages. Listen for a rule — execution status established when the document is read and carried with every passage — not for "the model can usually tell".
- 2. Does it keep the section and page of every passage it uses?
If the page is not kept at read time it cannot be reconstructed later. Listen for a yes with a demonstration: click a figure in an answer, land on the executed page with the clause highlighted.
- 3. Does every sentence of an answer carry the clause it rests on?
Not the answer as a whole — each sentence. An answer with three sources at the bottom cannot be checked sentence by sentence, which is the only way it gets checked at all.
- 4. What happens to a sentence the clause does not support?
The useful answer is that it is removed or rewritten before anyone sees it, and that the rewrite is visible. Listen for what the check does, not for an accuracy percentage.
- 5. Can it find a new source to justify a sentence after the fact?
It should not be able to. A checker allowed to go looking for support will always find some, and the test stops being a test.
- 6. Does it build each clause as amended, and from which date?
Ask to see one section with three versions stacked: the one in force, and the two it replaced, each with the amendment and effective date that moved it.
- 7. Can I ask the same question as of an earlier date?
Pick a date before your last amendment. The answer must change, must name the version in force then, and must say that the later amendment was not yet signed.
- 8. What does it do when the documents do not answer?
It should say so and name the closest clause. A tool that fills the gap from how such agreements usually work is the single worst outcome of this purchase, because the output is indistinguishable from a right answer.
- 9. How does it count business days, and on whose calendar?
A credit agreement can use two different calendars for two different obligations. Ask which calendar it used for a notice period, and ask to see the count.
- 10. Which of these behaviours are settings, and can you show me the screen?
Shown, not described. A rule that lives in a vendor’s configuration rather than on your own screen is a rule you cannot evidence to a reviewer.
- 11. Who can see what?
Side letters and fee letters are not for everyone, and a relationship manager should see answers for their own clients. Ask how access is enforced and how you would prove it after the fact.
- 12. What can it change without a person, and who approves a correction?
The answer you want is nothing, and a named approver. Ask specifically whether anything writes to the servicing system, and what the approval record looks like.
The questions about citations that most demonstrations fail
Questions three to five are where a polished demonstration usually comes apart, and they are easy to run live. Ask for an answer to a real question on one of your facilities, then take one sentence — not the first — and ask to see the clause behind that sentence. Three things can happen. The clause opens at its page and says what the sentence says: good. The clause opens and is about the right subject but does not support the specific claim: that is the failure the published evaluations measure. Nothing opens, because the sources are attached to the answer rather than to its sentences: the tool cannot be checked at the granularity at which it will be wrong.
What a checkable answer carries, element by element
| Element | Why it matters | How to test it in the room |
|---|---|---|
| A clause reference on each sentence | Errors occur at sentence level; review has to happen there too. | Pick the third sentence and ask what it rests on. |
| The executed page behind the reference | A section number without a page is a promise, not evidence. | Click it. You should land on the page with the text highlighted. |
| The amendment that put that text in force | The same section number can carry four texts over a facility’s life. | Ask which document the quoted words come from, and its effective date. |
| A visible record of what the check changed | A check with no output is indistinguishable from no check. | Ask what the citation step rewrote or removed in this answer. |
| A refusal where the documents are silent | The hardest behaviour to retrofit, and the easiest to test. | Ask something the agreement does not address and listen for an explicit "it does not say". |
What question two looks like when the answer is yes
The questions about dates that every loan book needs
Questions six and seven have no analogue in general contract review, which is why a tool built for a contract repository can look excellent and still be unusable on a loan book. A repository's documents are static. A credit agreement is overwritten in parts, repeatedly, and the question "what does the agreement say" has no answer without a date attached.
Run the test with two dates that straddle your most recent amendment to a section that matters — a margin, a soft call, a maturity, a financial covenant level. The right behaviour is three-part: the two answers differ; each names the document and effective date of the version it used; and the earlier answer states that the later amendment was not yet signed rather than silently ignoring it. On the sample facility this is visible on the face of the chain, where the version in force sits above the two it replaced, each struck through with the date it stopped applying.
There is a maintenance question hiding inside question seven that is worth asking separately: what happens to answers already saved when a new amendment arrives? If saved answers are not re-checked against the clauses the amendment touched, the tool has quietly become a source of stale answers with citations on them — which is worse than no saved answers at all. The mechanics of the chain and the as-of control are set out in the guide on which version governs today.
The questions about who sends, and who approves
Questions ten to twelve are where a purchase becomes a control design, and they are the ones a second-line reviewer will care about most. The practical form of question ten is not "is it configurable" — everything is configurable — but "show me the screen my team would use, and tell me who in my organisation can change each of these".
Question ten, answered by showing rather than telling
Question eleven is about visibility and is easy to under-ask. On an agency facility the loan file contains documents that are not for the whole bank: fee letters, side letters, arrangements with individual lenders. The useful test is two logins. Open the same facility as a member of the agency team and as a relationship manager, and look at what the second one cannot see. A tool that answers this with a role matrix on a slide has not answered it.
Question twelve is the one that should have the shortest answer. Nothing automated should write to the servicing system, and every correction should carry the clause behind it to a named approver. On the sample desk the nightly comparison produces differences, each with the system value, the amended term, the clause and what the difference costs, and the loan operations supervisor approves each one; the comparison itself changes nothing. The seats that approval sits between are worth mapping before the trial, not after.
Is AI contract review accurate? Read the published evaluations first
Accuracy claims in this market are hard to verify from outside, and the researchers who tried said so: "Because of the closed nature of these systems, systematically assessing these claims is challenging" (Magesh and others, 2024). Their preregistered evaluation of purpose-built legal research tools found hallucination rates well above zero despite retrieval over a real corpus — the figures, and what they do and do not transfer to a loan book, are set out in the guide on testing a cited answer.
Three conclusions follow for a buyer. First, treat a vendor's own accuracy figure as a claim about a corpus you have not seen, scored by a method you did not agree; it is not a lie, and it is not evidence about your files. Second, prefer an architecture you can check over a number you cannot: a tool whose every sentence opens its clause can be audited by your own staff forever, while a tool with a better number and no sentence-level sources cannot be audited at all. Third, ask what the system does when it is unsure, because the failure rate that matters is not how often it is wrong but how often it is wrong invisibly.
How do you test an AI contract review tool on your own agreements?
Bring four things and the test designs itself. None of them requires an integration, and all four can be assembled by the desk in a morning.
What to bring to a one-afternoon trial, and what each item proves
| What you bring | How much | What it proves |
|---|---|---|
| One facility with a real amendment history | A credit agreement and three or four executed amendments touching the same sections. | Questions six and seven: whether the tool has a concept of an effective date at all. |
| Questions you have already answered | Twenty, with the replies that were actually sent. | Questions three to five: the answers can be marked against a known result rather than argued about. |
| Questions the documents do not answer | Five, chosen so a plausible general answer exists. | Question eight, which is the one most tools fail and no vendor volunteers. |
| Two documents that should never be cited | An unexecuted draft amendment and a superseded conformed copy, in the file. | Question one: whether execution status is a rule or a hope. |
Decide your pass marks before the session, not after seeing the output — the temptation to adjust a bar to fit a result you liked is strong and well documented in every field that measures anything. And do the marking with the agreement open. An answer that sounds right is the thing being tested.
What a good answer to each question sounds like
Across all twelve, good answers have a family resemblance: they are specific, they are demonstrable in the room, and they describe a refusal rather than a capability. Bad answers have a resemblance too.
The same question, answered two ways
A demonstration answer: "Every answer is fully sourced and grounded in your documents."
An answer you can check: "Each sentence names one clause. Pick a sentence and I will open the executed page it came from. Anything the clause did not support was rewritten before you saw it — in this answer, one sentence was."
A demonstration answer: "The system always uses the most up-to-date version of your contracts."
An answer you can check: "Here is §2.11(f) in three versions with their effective dates. Ask the same question as of 1 March 2025 and you will get the middle one, with a note that the latest amendment was not signed until 22 July 2026."
A demonstration answer: "It will give you the best available answer based on the documents and standard market practice."
An answer you can check: "It says the documents do not address it and names the closest clause. That is a configured behaviour and it is on this screen."
A demonstration answer: "We benchmark at over 95% accuracy."
An answer you can check: "On your twenty questions, mark it yourself. We have not measured your corpus and neither has anybody else."
A demonstration answer: "It integrates bidirectionally with your servicing platform."
An answer you can check: "It reads your servicing system and writes nothing to it. Differences go to a named approver with the clause and what each one costs."
One answer, with the four things the twelve questions are looking for
When the twelve have been answered, the remaining decision is commercial rather than technical, and it is worth writing down what you concluded while the session is fresh. The next step most desks take is a short trial on live questions — a four-week pilot on your own loan files sets out how to design one whose result a credit committee will accept.
Questions people ask
Do bank regulators allow AI contract review?
US banking supervisors have not prohibited it, and the OCC has said it supports banks' efforts to integrate AI into core functions while managing the risk safely and soundly. What they have not done is publish an acceptance test. The federal agencies' revised model risk guidance explicitly places generative and agentic models outside its scope, so a bank cannot discharge the question by pointing at that framework.
Does model risk management cover generative AI?
Not under the revised guidance issued in April 2026, which states that generative and agentic AI models are novel and rapidly evolving and as such are not within its scope. The agencies have signalled a request for information on the subject. In the meantime, most banks apply the spirit of their existing framework — documented purpose, controls, monitoring and an owner — without claiming the framework formally applies.
What should an AI contract review tool cite?
The clause, at the sentence it supports, with the document and page it was read from, and the amendment that put that text in force. A list of source documents at the foot of an answer is not a citation for this purpose: it cannot be checked against the claim it is supposed to support, which is the whole function of the reference.
How long should an evaluation take?
An afternoon for the twelve questions on a single facility, and about four weeks for a trial that produces numbers a committee will accept. Longer evaluations tend not to produce better evidence; they produce more meetings. The constraint is rarely the tool, it is getting twenty closed questions and the replies that were sent out of the team that owns them.
Should we ask for a proof of concept on redacted documents?
Prefer real executed files under the usual agreements if your legal and information security teams allow it. Redaction removes exactly the detail that makes the test meaningful — defined terms, dates, parties, amounts — and a tool can look fine on a redacted agreement and fail on the real one. If redaction is unavoidable, redact the parties and keep the structure.
What questions should you ask an AI vendor beyond the product?
Where your documents are processed and retained, whether they are used to train anything, what happens to your data at the end of the contract, how a version change is notified and how you would re-run your acceptance test against it. The last is the one most often forgotten: a tool that passed in March is a different tool in September unless somebody checks.