Banking chatbots are governed by the same regulations as the people they stand in for: there is no federal chatbot rule, and the CFPB's position is that a chatbot answers to every federal consumer financial law that would bind the process it replaces (CFPB, Chatbots in consumer finance, 2023). For deposit questions that means Regulation E, Regulation CC and Truth in Savings, plus the prohibition on misleading consumers, and it means a wrong answer is the institution's wrong answer. The practical compliance question is therefore not which rule applies but whether the tool answers correctly, and the only way to know is to test it with questions whose answers you already know. Botminds' Deposit Compliance Search, a question box for staff on a live call that answers with the rule, the official interpretation and the institution's own procedure, is built to be tested that way.
Why a compliance chatbot has to be tested before anyone relies on it
17–33%
of answers from leading AI legal research tools contained a hallucination in a peer-reviewed test
65%
of questions answered both correctly and with supporting citations by the best tool tested
74%
of the Reg E violations the FDIC cited in 2025 were in error resolution
Which regulations apply to banking chatbots?
The rules that apply to the conversation, whoever or whatever is having it. A member who tells a chat window that a transfer was not hers has, in substance, told the institution; Regulation E treats notice of an unauthorized transfer as given once the consumer has taken the steps reasonably needed to tell the institution, in person, by phone or in writing (12 CFR 1005.6(b)(5); Reg E and account takeover quotes the rule). A chatbot that quotes a hold period is making a Regulation CC statement, and one that quotes a savings rate is making a Truth in Savings statement.
Above the deposit rules sits the general prohibition on unfair, deceptive or abusive practices, which is why the CFPB frames the risk as legal and not only commercial: institutions "risk violating legal obligations, eroding customer trust, and causing consumer harm when deploying chatbot technology". The model risk guidance the banking agencies revised in April 2026 does not settle the matter either way. It places generative and agentic AI outside its scope, leaves their controls to each institution's own governance, and describes itself as setting no enforceable standards (FDIC FIL-15-2026). The burden of showing that a chatbot works falls on the institution that deploys it.
What the CFPB said about chatbots in banking
The CFPB's 2023 issue spotlight is still the most specific regulator text on the subject, and it remained posted in October 2026; it was not among the guidance documents the CFPB withdrew on May 12, 2025 (90 FR 20084). It says of itself that it creates no obligations and defines no rights, so it is a statement of risk, not a rule. Its two central findings are the ones a deposit chatbot test should be built around.
The first is recognition. A chatbot, like a tightly scripted representative, may start a dispute only when the customer uses particular words, and the spotlight warns that the technology "may fail to recognize that a consumer is invoking their federal rights" (CFPB issue spotlight; what that costs a P2P claims desk is in claims fraud using AI). The second is accuracy: "In instances where financial institutions are relying on chatbots to provide people with certain information that is legally required to be accurate, being wrong may violate those legal obligations."
Staff assistant or member chatbot: why the risk differs
Most banks met chatbots as a member-facing channel, but the tool compliance teams now evaluate is often internal. GAO's 2025 report on AI in financial services describes "one regional financial institution" that "is piloting an internal generative AI chatbot that answers employees' questions about policies and procedures" (GAO-25-107197). The two carry different risks, and a test should know which it is testing.
A member-facing chatbot and a staff compliance assistant fail in different places
Member-facing chatbot: The member, directly, with no one to catch a wrong statement.
Staff compliance assistant: A trained employee, who decides what to say and can flag an answer that does not fit.
Member-facing chatbot: The bot must spot a dispute in the member's own words; the CFPB's first finding.
Staff compliance assistant: The employee spots it; the assistant must not talk them out of it, for example by advising a signed form first.
Member-facing chatbot: A wrong hold date or rate reaches the member as the institution's statement.
Staff compliance assistant: A wrong answer reaches the member through a person who trusted it, often many times before anyone notices.
Member-facing chatbot: Dispute recognition across phrasings, escalation to a person, accuracy of disclosures.
Staff compliance assistant: Accuracy, citations, currency of the rule text, and routing when the answer is not settled.
The staff side has a subtler failure. A representative who gets the same wrong answer from an assistant ten times stops checking it, and the error spreads at the speed of the tool. That is why the test below scores currency and routing as well as correctness, and why knowing who was given last month's answer matters after the tool goes live.
How accurate are AI legal research tools?
Less accurate than their marketing, according to the only preregistered study of commercial tools. Researchers ran 202 legal questions through three commercial AI research products and a general-purpose model, and found that each commercial tool hallucinated on 17% to 33% of the questions (Magesh et al., Journal of Empirical Legal Studies, 2025; ChatGPT for AML compliance quotes the finding). Fully accurate answers, meaning both correct and properly cited, ranged from 65 percent for the best tool down to 19 percent for another.
Share of 202 legal questions answered correctly and with supporting citations
The study's most useful contribution for a compliance team is its vocabulary. An answer is "misgrounded" when its "key factual propositions are cited but the source does not support the claim", and the authors count that as a hallucination alongside plain falsehood. Such errors, they write, are "potentially more dangerous than fabricating a case outright, because they are subtler and more difficult to spot." They also found the tools struggling with "respecting the hierarchy of legal authority", which in deposit compliance means treating a CFPB FAQ as if it were the rule. What outranks what in a Reg E answer is the reference for that check.
How do you test a compliance chatbot? Seed it with known answers
Write questions whose correct answer, and the source that supports it, you settle in advance, then run them through the tool and score every answer against the key. The seeds should include traps: questions where the obvious answer is wrong, where the rule changed recently, or where the honest answer is "this needs Compliance". A tool that only sees easy questions will look better than it is.
The same study offers a caution about traps used alone. Accuracy was highest on "false premise" questions, in which the question itself contains a mistaken understanding of law, and "lower on the categories that represent real-world use by attorneys". A seeded test proves the tool knows the traps; it does not prove the tool handles the ordinary questions staff ask all day. That is the job of a replay of real questions, which a 30-day pilot on last month's questions sets out.
Eight traps every deposit compliance test set should contain, with the known answer
- The police report
Can we require a police report before opening a Reg E claim? Known answer: no; the CFPB EFT FAQs answer this question directly (Error Resolution Q4), and the investigation starts on notice.
- The shared code
A caller posing as the fraud team got her one-time code. Known answer: unauthorized; a consumer fraudulently induced to share access information has not furnished an access device (CFPB EFT FAQs, Unauthorized EFTs Q6).
- The member who sent it herself
Same scam, but she pushed the payment. Known answer: authorized under 12 CFR 1005.2(m); route goodwill to Compliance. A tool that calls both cases unauthorized has learned a slogan.
- The old amounts
How much of a check is available next day? Known answer: $275 since July 1, 2025; $225 is wrong (12 CFR 229.11(c)). Same for $6,725 against the old $5,525.
- The business account
An LLC's operating account reports an unauthorized ACH debit. Known answer: Regulation E does not apply; answer from the agreement and procedure.
- "Reg DD" at a credit union
Known answer: Regulation DD excludes credit unions (12 CFR 1030.1(c)); the rule is NCUA Part 707, with the same 30-day change notice.
- Withdrawn guidance
Ask about overdraft fees on transactions authorized against a positive balance. Known answer must not cite CFPB Circular 2022-06 as current; it was withdrawn on May 12, 2025.
- The question with no settled answer
A question the rule and the procedure do not answer. Known answer: route to Compliance. A confident answer here is a failure, however plausible.
Score four things, not one
A single "accuracy" number hides the failures that matter. Score each answer on four separate questions, and report them separately: is it right, does the cited source say it, is the source the version in force today, and did the tool hand the question to a person when it should have. The third and fourth are where deposit compliance differs from general legal research, because the rules carry effective dates and the institution's own procedure changes more often than the law.
A four-part score for every test answer
| Score | Question the reviewer asks | Fails when |
|---|---|---|
| Correct | Is the answer right on the facts given? | It says "unauthorized" for a payment the member pushed herself |
| Grounded | Does each cited passage actually support the line it is cited for? | It cites 12 CFR 1005.11 for a point the section does not make (misgrounded) |
| Current | Is each source the version in force on the question's date? | It quotes $225, or relies on guidance withdrawn in 2025 |
| Routed | Did it hand unsettled or unsupported questions to a person? | It answers confidently where the rule and the procedure are silent |
One fact flipped, the answer rebuilt, and confidence below the line
How many test questions do you need?
More than most pilots use, because a clean result on a small set proves little. The arithmetic is simple: if a tool gets a given kind of question wrong 5 percent of the time, the chance it answers 20 such questions without a single error is 0.95 to the power of 20, about 36 percent. At 60 questions it falls to about 5 percent. A tool can pass a 20-question demonstration with a one-in-twenty error rate more than a third of the time.
Chance that a tool wrong 1 time in 20 passes a test with zero errors
Two consequences follow. Count errors by category, not overall, because 60 questions spread across eight traps and five rule areas leave only a handful in each. And treat the seeded set as a floor, then add real questions, since NIST's AI Risk Management Framework defines validation as objective evidence that a system meets the requirements of one specific intended use. The intended use is answering your staff's questions; the evidence has to come from them.
What a pass should buy, and what it should not
A pass should buy permission to use the tool for the categories it passed, with a person still deciding what to tell the member, and a schedule for testing again when rules or procedures change. It should not buy a member-facing deployment, an end to reviewing answers, or confidence in categories the test did not cover. The banking agencies' revised model risk guidance describes outcomes analysis as setting what a model produced against what actually happened (Model Risk Management, revised guidance, 2026); even where that guidance does not formally apply, the habit is the right one.
Deposit Compliance Search is designed to make each of the four scores visible. Every line of an answer is labeled Rule, Official interpretation, Regulator guidance or the institution's procedure, and every citation opens the passage in the version in force on the day, so a reviewer can check grounding and currency in one click. Below a 0.85 confidence threshold, on an unsettled point or when no source is found, the question goes to Compliance with a draft instead of an answer. It is a tool for staff, not a member-facing chatbot, and a test that needs to prove dispute recognition in members' own words is testing something else.
A cited answer in three layers, each line labeled by who issued it
Questions people ask
What is an AI hallucination?
GAO describes generative AI hallucinations as "credible yet erroneous responses". The legal research study adds a second kind that matters in compliance: an answer that cites a real source which does not support the claim. Both read as confident, which is why a test needs a known answer to compare against.
Can a chatbot answer compliance questions for staff?
It can answer many of them well, once tested, if it shows its sources and hands unsettled questions to a person. GAO reported a regional institution piloting an internal chatbot for employees' policy questions. The employee should still decide what to tell the member.
How do you measure AI answer accuracy?
Against an answer key written before the test, scoring correctness, whether the cited source supports the line, whether the source is current, and whether the tool routed what it should have. Report each by category; an overall percentage hides the trap questions that matter most.
Does model risk guidance cover compliance chatbots?
Not directly. The guidance the FDIC, OCC and Federal Reserve revised on April 17, 2026 places generative and agentic AI outside its scope, is aimed mainly at banking organizations over $30 billion in assets, and does not set enforceable standards. Institutions still need their own controls for these tools.
Should a chatbot take a Reg E dispute?
A member-facing chatbot may receive one whether or not it was designed to, because notice is given when the member takes reasonable steps to tell the institution. The safer design routes any message that may assert an error to a person or a claim process with the date of the message preserved.