AI agents in compliance are software that reads a question or a case, works through a defined task and acts within set limits: it finds the rule, drafts an answer, opens a claim in another system, routes what it cannot settle to a person, and records every step. That last part is what separates an agent from a chatbot, and it is why a pilot cannot stop at asking whether the answers sound right. The way to decide is to replay a month of your own staff's questions with answers your compliance team has already settled, measure both the answers and the actions, and go live category by category. Botminds' Deposit Compliance Search is a set of seven agents that answer front-line Reg E, Reg CC and Truth in Savings questions and open the Reg E claim on the call, and the 30-day plan below is how we would test it, or any tool like it, on your files.
Examples come from a fictional sample credit union, Norrell Federal Credit Union, as it runs through the solution. Regulator and research sources are linked where they appear.
What are AI agents in compliance?
They are narrow workers with written limits. In Deposit Compliance Search each agent has one job and a list of things it never does: the Question agent removes member identifiers and asks one clarifying question when a missing fact would change the answer; the Rule agent retrieves the rule, the official interpretation and regulator guidance in force on the day and labels each; the Procedure agent finds the institution's own section and never uses one under review; the Answer agent writes the answer and routes below a confidence threshold; the Claim agent opens a Reg E claim but never decides it, credits money or sets liability; the Change-watch agent finds answers built on superseded text; the Coaching agent drafts lessons a supervisor approves.
The limits are the compliance design. A pilot that measures only answer quality tests three of the seven agents. The others take actions with legal consequences, such as dating a claim, starting two clocks and telling 26 people that an answer they received is no longer right, and those actions need their own pass marks. Testing an assistant's answers with known traps covers the answer side in depth; this page adds the actions and the go-live decision.
What a pilot has to prove that a demonstration cannot
A demonstration shows that the tool can answer a question. A pilot has to show how often it answers your questions correctly, from your procedures, in the versions in force, and how often it knows to stop. The government framework most institutions borrow for this defines reliability as the "ability of an item to perform as required, without failure, for a given time interval, under given conditions" (NIST AI Risk Management Framework, AI 100-1). The conditions that matter are yours: your product names, your procedure wording, your members' way of describing a scam.
The regulators leave the burden there too. The banking agencies' April 2026 model risk guidance places generative and agentic AI outside its scope and leaves their controls to each institution's own governance (FDIC FIL-15-2026). Their guidance on third parties is plain that buying a tool changes nothing about accountability: the institution answers for work a vendor does exactly as if its own staff had done it (88 FR 37920, 2023; the interagency guidance on third-party relationships explains it in full). A pilot is how an institution produces its own evidence.
Financial institution representatives told us they have been more cautious about adopting AI for activities where a high degree of reliability or explainability is important or where they are unsure how regulations would apply to a particular use of AI.
GAO-25-107197, Artificial Intelligence: Use and Oversight in Financial Services (2025)
Which questions should a pilot use?
Last month's real questions, de-identified, in the proportions staff actually asked them, plus a small set of seeded traps. Real questions test what staff need; traps test what can go badly wrong. The answer key should be written by the compliance team before the replay, with the source for each answer, so nobody grades the tool against its own output.
The sample credit union's month shows why proportions matter. Its staff asked 1,684 questions in the first 22 days of October: almost half about Regulation E, nearly a third about Regulation CC, and a smaller share about Truth in Savings. A pilot sample drawn evenly across topics would over-test certificates and under-test the scam calls and check holds that fill the day.
One month of front-line questions by rule area: the pilot sample should follow it
Stratify within each area too. In the sample month, check holds and the next-day amount led Regulation CC with 196 questions, and scam-call transfers led Regulation E with 142. A replay set of a few hundred questions, drawn in those proportions with every category represented, gives enough answers per category to see a pattern; the arithmetic of small test sets explains why 20 is not enough.
How long should an AI compliance pilot run? Thirty days, in four weeks
Long enough to replay a month, run a procedure change through the system, and watch staff use it on live calls with a safety net. Thirty days does that if the inputs are ready on day one: the rules and their versions, the institution's procedures with owners and review dates, last month's questions and the answers given, and the settings the Compliance Officer wants to start with.
A 30-day pilot: replay, stress, shadow, decide
- Replay last monthWeek 1
- Traps and a procedure changeWeek 2
- Live calls, shadowedWeek 3
- Score and decide by categoryWeek 4
The procedure change in week 2 is the test most pilots skip. Change one real section, as the sample credit union did when it corrected its exception-hold notice procedure to match 12 CFR 229.13(g), and check that the agents find every answer built on the old text. On the sample, the change reached 37 answers given since August 24 to 26 staff, and 4 holds whose notices may have gone out late; within the hour 19 of the 26 had acknowledged the correction. A pilot should count whether every affected answer was found, not only how many people acknowledged.
A procedure change, old beside new, and every answer it reaches
How do you decide go-live, category by category?
Set the pass marks before the replay, write them down, and apply them to each category separately: check holds, scam-call transfers, provisional credit, fee notices, certificates and so on. A category that passes goes live; one that fails stays routed to Compliance while its procedure or its sources are fixed. The marks below are ours, offered as a starting point; the right levels are the Compliance Officer's call.
What to measure in an AI agents pilot, and a starting pass mark for each
| Measure | What counts as a failure | Suggested pass mark |
|---|---|---|
| Correct answer | Wrong on the facts as given, or a different answer from the key without a reason | 19 in 20 or better in the category |
| Grounded citations | A cited passage that does not support its line | None tolerated; one is a finding to fix |
| Current sources | A superseded amount, a withdrawn document, a section under review | None tolerated |
| Routing | A confident answer where the key says Compliance decides | None tolerated in traps; review every miss |
| Claim notice date | A Reg E claim dated to anything but the first notice | None tolerated |
| Recall coverage | An answer built on changed text that the recall missed | Every affected answer found |
| Member details | A name, account, card or phone number kept in a saved question | None tolerated |
The zero-tolerance rows are the ones tied to a rule. A claim dated wrong moves a deadline under 12 CFR 1005.11; a hold notice procedure that is out of date affects notices due under 12 CFR 229.13(g). Those are not quality preferences, and a category with one such failure should not go live until the cause is found.
Before a category goes live
- The answer key was written before the replay
By Compliance, with the source for each answer, and kept with the results.
- Every procedure section the category uses is approved and current
Sections under review are excluded; the agents answer from the rule alone until approval.
- The pass marks were met in this category, not only overall
Including the zero-tolerance measures.
- Staff know how to flag an answer that does not fit
With a reason code, so flags can be counted and reviewed after go-live.
- The threshold and clocks are set and approved
Confidence threshold, the clock with a member waiting, the clock otherwise, each approved by the Compliance Officer.
- The evidence is exportable
Questions, answers, sources and versions, routings, claims, changes and settings, for an examiner or internal audit.
Who reviews AI answers during a pilot?
The compliance team, with two people on every disagreement. One analyst scores the answer against the key; where the tool and the key disagree, a second reviewer decides whether the tool or the key is wrong, because a pilot regularly finds that the key was. The same separation runs through the solution: procedure changes, answers approved for reuse and setting changes are drafted by one person and approved by another, and the drafter can never approve.
In week 3, supervisors join. Staff use the answers on live calls; the supervisor sees which questions recur, which answers were flagged as not fitting and why, and which questions went to Compliance. On the sample floor, 22 answers were flagged "doesn't fit" in the month, each reviewed by Compliance. In a pilot, every flag is data about where the procedures are thin.
What does model risk guidance say about AI agents?
Little, directly. The revised interagency guidance of April 17, 2026 leaves generative and agentic AI models out of its scope, says it matters most to banks with more than $30 billion in assets, and describes itself as setting no enforceable standards (FDIC FIL-15-2026; its exact words are quoted in the customer risk rating model guide). For a credit union or a community bank, the practical expectation comes from the general duty to comply with the deposit rules, whoever or whatever does the work.
The guidance's methods still translate. It describes outcomes analysis and ongoing monitoring, and for vendor products it expects the bank to keep checking, on its own results, that the vendor's model is accurate, still fits its purpose and stays reliable (revised guidance, section VII; CRA compliance AI quotes the vendor section). A pilot is the first outcomes analysis; the flags, overrides and routings after go-live are the monitoring. Reg E and Truth in Savings already require evidence of compliance to be kept for two years (12 CFR 1005.13(b)), and the pilot's records belong in it.
Answers recalled after the change: who received the old text, and who has acknowledged
Four ways an AI compliance pilot goes wrong
Most failed pilots fail on design rather than on the tool. The first failure is an answer key written after the replay, or by reading the tool's answers; it measures agreement, not accuracy, and it hides exactly the cases where the tool was confidently wrong. The second is testing against sources that are not versioned: if the pilot cannot say which version of a procedure section or which edition of the rule text an answer used, it cannot tell a wrong answer from a stale one, and the July 1, 2025 Regulation CC amounts make that distinction concrete.
The third is reporting one overall score. A tool can answer nine in ten questions correctly overall while failing most of a small category that carries a deadline, such as provisional credit timing, and an overall figure will not show it. The fourth is skipping the procedure change. A pilot that never changes a procedure has not tested what happens on the first day the institution's own answer moves, which, at the sample credit union, happened twice in six weeks: once in September for scam claims and once in October for hold notices.
Where Deposit Compliance Search fits a pilot, and where it does not
It fits institutions that have written deposit procedures, even imperfect ones, and a log or mailbox of the questions staff ask. The solution answers from the rules and those procedures, labels every line by authority, opens Reg E claims in the dispute system with the notice date and both clocks, and exports an evidence pack of every question, source, version, routing, claim and setting for an examiner, internal audit or a supervisory committee. Where the procedures are silent, the pilot will show it quickly, because those questions route.
It does not decide claims, move money, change procedures or talk to members, and it is not a member-facing chatbot. It will not replace the compliance team's judgment on unsettled points, goodwill decisions or anything involving a report about a member. A pilot that expects it to will fail, and should.
Questions people ask
How do you pilot AI agents in compliance?
Replay a stratified sample of last month's real questions against an answer key the compliance team wrote first, add seeded traps, run one real procedure change through the system, then let staff use it on live calls with review. Score answers and actions by category and go live only where both pass.
What accuracy is good enough?
It depends on the measure. For ordinary correctness a category might need 19 answers in 20; for anything tied to a rule's deadline, such as the notice date on a Reg E claim or a current dollar amount, the right tolerance is none, because one error is a finding to fix before go-live.
How long should an AI compliance pilot run?
About 30 days, in four weeks: a replay, a stress week with traps and a procedure change, a shadowed week on live calls, and a scoring week. Shorter pilots usually skip the procedure change, which is where answers go stale after go-live.
What is the difference between an AI agent and a chatbot in compliance?
A chatbot answers; an agent also acts within set limits, such as opening a claim in another system, routing a question with a clock or recalling answers after a change. That is why a pilot for agents measures the actions as well as the words.
Do we need to tell examiners about an AI pilot?
No rule cited here requires notice of a pilot, but examiners will expect the institution to show how it knows the tool works and what it does with errors. A pilot's answer key, scores, decisions and the evidence of each question and answer are the natural record to keep.