AI contract search for treasury agreements: the second line's questions

AI contract search is the use of a language model to find and answer questions from a body of signed contracts: which clause applies, what a client agreed to, whether a term changed. For a bank's treasury agreements, the risk is not that the model cannot read; it is that it answers fluently from the wrong version, invents a citation, or quotes one client's terms to another. The interagency model risk guidance issued in April 2026 leaves generative and agentic AI models outside its scope, so a bank's second line cannot lean on it and has to set its own tests. Botminds' Treasury Management Agreement Search is built around those tests: answers only from the version in force on the date asked, every quote with its page and signer, a confidence floor below which the question goes to Legal, and a replay against approved answers before any agent changes.

The figures below come from a fictional sample bank running the solution, and are labelled as sample results or design targets. The guidance quoted is the regulators' and NIST's own text.

What AI contract search should mean for a bank

For a bank, AI contract search should mean retrieval first and language second. The model's job is to find the passages that answer the question, in the documents the client actually signed, and to put them in front of a person with enough context to check them. Writing a fluent paragraph is the easy part; choosing the right page from fourteen documents signed across seven years is the hard part.

Treasury agreements make the retrieval problem sharper than most contract sets. A client's terms sit across a master agreement, a schedule per service, amendments that replace single sections from a date, and user forms that change whenever staff change. How the version in force is worked out is a separate guide; for the second line the point is that a search that ignores dates will be wrong in a way that looks right.

Does model risk guidance apply to AI contract search?

Not directly, as of April 2026. The revised interagency guidance on model risk management, issued as Federal Reserve SR 26-2 and OCC Bulletin 2026-13 and superseding SR 11-7, leaves generative and agentic AI outside its scope (SR 26-2).

The agencies gave their reason in one line: these models are new and changing fast, so the guidance does not try to cover them. They also said a request for information on model risk, and on banks' use of AI including generative and agentic AI, would follow soon (OCC Bulletin 2026-13). The agencies' exact words, and what the new definition of a model means for scoring tools, are set out in whether a customer risk rating model is a "model" under the 2026 guidance. Until then the second line is on its own, and the absence of a rule is not the absence of an expectation. A bank that puts an AI answer in front of a client, or relies on it in a dispute, owns that answer. A second line can reasonably borrow the old discipline anyway: a written purpose, testing before use, ongoing monitoring and someone independent to challenge it.

The guidance also narrows what counts as a model at all, leaving out fixed rule-based software that has no statistical, economic or financial theory behind it. Parts of an AI contract search system fall into that excluded category, such as masking numbers or counting business days, and parts plainly do not. A sensible inventory says which is which.

Is AI contract search accurate?

It can be, on the one measure that matters: whether the answer is supported by the right passage. Raw fluency is not accuracy. NIST's Generative AI Profile names the failure directly, calling it confabulation: false content stated with confidence, sometimes with invented reasoning or citations offered to justify it (NIST AI 600-1, July 2024; the definition in full is quoted in ChatGPT against a checked answer on sanctions questions). A contract search that invents a section number is worse than one that answers nothing.

So accuracy should be measured as three separate rates, not one. How often the answer cites a passage that exists and says what the answer claims. How often that passage comes from the version in force on the date asked. And how often the system declines when it should, because the document is missing, unsigned or unreadable. A vendor that reports a single accuracy figure has not measured the second and third.

Can AI cite the clause it relied on?

Yes, if citing is built into how it answers rather than added afterwards. NIST's profile lists checking the sources and citations in a generative system's answers among the actions to measure its risk, both before it goes live and while it runs (NIST AI 600-1; the same step applied to compliance answers is in using ChatGPT for AML compliance questions). For a treasury agreement a citation is complete only with five parts: the document, the section, the page, the effective date and the signer. The last two are what make a quote usable in a dispute, because Article 4A puts the burden on the bank to prove it followed the procedure the client agreed to (UCC §4A-202(b)).

In the solution, a citation opens the scanned page where the passage sits, highlighted, with the signature page a tab away. The original scan stays in document imaging. An answer the agents cannot support with a passage is not written; the position agent drafts nothing below 0.85 confidence.

One line on whether the two sides match, and the version the answer did not use

How to validate AI answers on contracts

Validate on the bank's own questions, with answers a lawyer has already approved, and keep the set out of anything the system is tuned on. The questions should look like the ones staff actually ask: what a client agreed to before a date, whether a limit was raised, which form named a user. Then score the three rates above, and add a fourth: how often an answer goes to Legal that did not need to. A system that sends everything to Legal is safe and useless.

Re-run the set before any change. In the sample bank's settings, every agent carries a version, a prompt version and a threshold, and a new version is re-run against 420 past questions with approved answers before it goes live. That is the discipline model validation asks of a traditional model, applied to a system the guidance does not cover.

What the sample bank's numbers show, and what they do not

The headline sample figure is a coverage rate, not an accuracy rate. Of 14,260 questions staff asked between July 1 and October 21, 2026, 92% were answered with a citation and no escalation; the other 8%, about 1,140 questions, were escalated rather than answered outright, which is the route the solution takes when the agents fall below the confidence floor, find conflicting documents or cannot read a page. That split is what a second line should want to see: a system that knows when to stop. It says nothing yet about how many of the 92% were right, which is what the approved-answer replay measures.

Read the other numbers the same way. A 0.85 floor is a setting, chosen by the bank and changeable only when the director approves the document administrator's proposal. The 420 approved answers are a test set, and its value depends on how closely its questions match the ones staff ask this year. The zero is a design target for answers drawn from a superseded version, measured in the live solution; a second line should ask to see it measured on its own bank's documents during a pilot.

Where AI contract search goes wrong in a bank, and the controls that catch it

Four failures account for most of the risk, and each has a control a reviewer can see. Wrong version: answered by dating every section and naming the versions not used. Invented citation: answered by refusing any answer without a retrieved passage, and by opening the cited page. Leakage between clients: answered by reading one client's file per question and by access drawn from directory groups, so a relationship officer sees only the clients in their portfolio. Exposure of account data: answered by masking account, routing and token numbers to the last four everywhere, including in examiner exports, with no setting to switch it off.

The fifth failure is organisational: the AI answer that leaves the bank without a person checking it. The sample bank's chain is the control. The client services officer drafts, Legal approves, the director approves anything over $250,000 or where the bank accepts liability, and only the wire fraud claims analyst sends. The agents never send anything to a client and never accept liability.

Who did what, when, why, and with which evidence

What Treasury Management Agreement Search is, and is not, for the second line

The solution answers questions about signed treasury agreements and payment orders, for the people who handle client disputes and the lawyers who approve the answers. It is search with a narrow job: find the passage, date it, compare it with the payments hub, draft for a person. It is not a general contract review tool for negotiating new agreements, and it does not decide whether a client was defrauded or whether a procedure is commercially reasonable as a matter of law.

A second line evaluating it should ask for the same evidence it would ask of any vendor: the test set and how it was scored, the thresholds and who can change them, and the trail. The questions a buyer asks before choosing a tool are in twelve questions for contract analysis software. The product itself is at Treasury Management Agreement Search.

Questions people ask

What is an AI hallucination in contract search?

It is a confident answer that the documents do not support: an invented clause, a wrong number or a citation to a section that says something else. NIST calls it confabulation and notes that it can extend to the citations themselves. The control is to require a retrieved passage behind every answer and let the reviewer open it.

Is AI contract search a model under the 2026 guidance?

Generative and agentic AI models are expressly outside the scope of SR 26-2 and OCC Bulletin 2026-13, and the agencies have said they plan a request for information on AI. That leaves the bank to set its own controls, and a second line can borrow the model risk discipline of purpose, testing, monitoring and challenge.

How many test questions are enough?

Enough to cover the question types staff actually ask, with each type represented several times and answers approved by a lawyer. The sample bank uses 420 past questions with approved answers and re-runs them before any agent version goes live. A set that never changes should be refreshed as the agreements and the questions change.

Should AI contract search answer clients directly?

Not on disputes. In the sample bank the agents draft and a client services officer, Legal and, above $250,000, a director stand between the draft and the client, and only the claims analyst sends. A direct answer would put an unreviewed reading of a contract in front of the party most likely to contest it.

What confidence threshold should an AI answer need?

There is no regulatory figure; it is a choice the bank should make from its own test results. The sample bank sets 0.85, below which the question goes to Legal with a one-business-day clock. The right floor is the one where the answers above it pass the bank's own test at the rate it is willing to defend.