AI email triage is software that reads each incoming customer message, decides what it is asking for, routes it to the right queue and usually drafts a reply. Most tools are judged on topic accuracy: how often the label they give a message matches the label a person would give it. At a bank that is the wrong test. A secure message that asks to close an account and mentions, in its third sentence, a card charge the customer did not make is a Regulation E notice of error, with a 10-business-day clock that started when it arrived, and a tool that labels it "account closure" has scored a correct topic and missed the only part with a legal deadline. The test that tells you whether AI email triage is safe at a bank is a planted one: take last quarter's real messages, hide notices of error inside them, and count how many the tool finds and dates correctly. Botminds' Secure Message Triage was built to that standard, and this page shows how to hold it, or any other tool, to it.
A 50-notice planted test: where to hide the notices
What is AI email triage, and what changes in a bank's secure inbox?
In most service desks, triage means sorting: one message, one topic, one queue, and a faster answer for the customer. In a bank's secure message center, triage also means dating. Regulation E applies its error rules to "any oral or written notice of error" that identifies the consumer and the account and says why there is an error (12 CFR 1005.11(b)(1)), and its 10 business days run from receipt. The triage step is where the bank either notices that a clock has started or does not.
Two features of secure messages make that harder than it sounds. Customers write several requests in one message, because the message center is the one place they are already signed in. And the topic they pick from the drop-down describes the reason they opened the screen, not everything they wrote: one sample member at a fictional credit union picked "Other" for a message holding a card dispute, a savings closure and a fee complaint. A triage tool that assigns one label per message cannot represent that message at all.
How accurate is AI email triage? Ask about misses, not topics
Accurate enough on topics is easy to claim and nearly useless as evidence, because the measure that matters at a bank is recall on notices of error: of the messages that contain one, how many did the tool find, and did it date each to the day the message arrived? In the sample weekend at the fictional Norrell Federal Credit Union, 241 of 2,186 secure messages, about 11%, held a notice of error. A tool that labels the main topic of every message correctly can still miss every one of those 241 that is not the main topic.
Topic accuracy and notice recall measure different things
Topic accuracy: Did the tool give the message the label a person would give it?
Notice-of-error recall: Of the messages that contain a notice of error, how many did the tool find?
Topic accuracy: One label per message.
Notice-of-error recall: Each request inside a message.
Topic accuracy: Getting the main request right.
Notice-of-error recall: Finding the second and third requests too.
Topic accuracy: Still scores a correct topic.
Notice-of-error recall: Scores a miss, and a legal clock nobody is counting.
Topic accuracy: Not measured.
Notice-of-error recall: Each found notice is also checked for the right receipt date.
The National Institute of Standards and Technology makes the same point in general terms. Its AI Risk Management Framework says measures of accuracy "should consider computational-centric measures (e.g., false positive and false negative rates)", and that results "may include disaggregation of results for different data segments" (NIST AI 100-1, 2023). For a bank's inbox the false negative that matters is a missed notice, and the segment that matters is the message where the notice is not the main request.
"Accuracy measurements should always be paired with clearly defined and realistic test sets – that are representative of conditions of expected use – and details about test methodology."
NIST, AI Risk Management Framework 1.0 (AI 100-1), January 2023
The planted test: hide 50 notices of error in last quarter's messages
A planted test answers the recall question with evidence you control. Draw about 1,000 real messages from last quarter, masked; have compliance write 50 notices of error modelled on real customer phrasings; plant each inside a message of the kind it would really hide in; and run the tool on the whole set without telling it which ones are planted. Then compare what it found with the answer key.
The design choices carry the test. Use your own messages, not the vendor's demo set, because NIST's "conditions of expected use" for a credit union in Ohio are not a retail bank's in Texas. Write the planted notices the way customers write, with typos and without the word "dispute". Keep the answer key away from whoever runs the tool. And plant some notices in plain sight as well, so a tool that fails the hidden ones cannot claim it was never shown a notice.
Fifty is a floor, not a target. If a tool misses none of 50, the most you can say at 95% confidence is that its true miss rate is below about 5.8%; at 100 planted with none missed, below about 3.0%. One miss in 50 moves the upper bound to about 9.1% (exact binomial bounds, our arithmetic). A bank that wants to say "we do not miss notices" needs either a larger planted set or a tool whose design makes misses visible rather than silent.
Where notices hide: five places a topic label never looks
Notices of error hide wherever the customer's main purpose is something else. The five places below account for the planted mix above, and each has a rule behind it that turns an ordinary sentence into a notice.
Behind an account request. "Please close my savings, and by the way there's a charge I didn't make" is a closure request to the customer and a notice of error to the regulation; the commentary requires error resolution even after an account is closed (comment 11(a)-4). One closure request that hid a dispute is followed there day by day.
Inside a fee complaint. A customer angry about an overdraft fee often explains it by naming the charge that caused it. If they did not make that charge, the fee complaint carries a notice, and the fee's refund depends on the dispute's outcome.
As a question. "What is this $71.44 from a place called QSR?" asserts nothing, but the commentary counts a request for information about a transfer as an error unless the customer plainly wants a copy for tax or records (comment 11(a)-5); the same line drawn on a phone call is in complaint vs inquiry on a bank call.
In a lost card report. A lost card alone is a card-block request; once the customer adds "and there's a charge on it", it is a notice of error (comment 11(a)-3).
In a flood. When an outage brings hundreds of near-identical messages, a bulk reply is the right answer for most of them and the wrong one for the few that also ask for something else. In one sample weekend, 9 of 412 outage messages also carried a notice of error, a complaint, a fraud report or a closure.
One approved reply for an outage flood, with every legal notice held out
How do you test an email triage tool? Score four outcomes, not one
Score every planted notice into one of three outcomes, and every clean message into a fourth. A notice the tool found and dated to the arrival day is a catch. A notice it found but dated later (the day it was read, or the day after a weekend) is a late catch, and it counts against the tool, because a clock started late is a deadline the bank may miss without knowing it. A notice it did not find is a miss. A clean message it flagged as a notice is a false alarm, which costs a specialist's time but no customer's rights.
A scorecard for a planted triage test, with the pass mark we suggest
| Outcome | How it is counted | What it costs | Suggested pass mark |
|---|---|---|---|
| Caught, dated right | Notice found, clock dated to the day the message arrived | Nothing | Every planted notice |
| Caught, dated late | Notice found, clock dated to a later day | Business days off the investigation and provisional credit | None |
| Missed | Notice not found at all | A legal clock running with nobody counting it | None, and each miss explained |
| False alarm | Clean request flagged as a notice of error | A specialist's review time | Reported, with a limit the bank sets |
| Sent to a person | Tool unsure and says so | A representative's decision | Counted separately; a flagged uncertainty is not a miss |
Examiners will recognise the method. The Federal Reserve's Consumer Compliance Outlook suggests that "a financial institution's compliance department can conduct transaction testing on previously denied error notices" to check that each investigation met the bank's burden of proof (Consumer Compliance Outlook, 2025). A planted triage test is the same discipline applied one step earlier, to the notices nobody recognised.
What happens when AI misroutes a dispute?
The clock keeps running whether or not the tool noticed it, and the bank owns the result. One public case of automation going wrong on notices of error is Bank of America's handling of prepaid unemployment-benefit cards in 2020 and 2021. The CFPB found the bank had changed its practices "to solely rely on an automated fraud filter, which it knew or should have known would incorrectly determine that no error had occurred and which led to its incorrectly freezing or blocking accounts", and separately that it had engaged in unfair practices "by impeding" cardholders' "efforts to file notices of error" (CFPB, Bank of America, N.A., 2022). The CFPB ordered a $100 million penalty; the OCC fined the bank $125 million.
A triage tool is not a fraud filter, but the lesson transfers in two directions. A tool that decides a message is not a notice, with no person seeing that decision, is a filter of the same kind. And a channel in which notices go unrecognised is a channel that, in effect, impedes them. That is why the planted test scores misses and late dates, and why Secure Message Triage's design starts a clock when in doubt and leaves every decision about the claim to the dispute specialist.
Can one email contain several requests? Usually, in a secure inbox
Yes, and a triage tool has to represent that or it will drop requests by construction. The Splitting agent in Secure Message Triage cuts each message into its separate requests, marks the sentence that decides each one, and links requests that depend on each other, such as a fee that posted because a disputed charge took the account below zero. Each request then gets its own clock and its own hand-off.
The agent's limits are written for this risk: it never merges two requests to save work, and below a confidence of 0.85 it leaves the part to the representative rather than guessing. On the sample member's three-request message, splitting, clock dating, routing and drafting finished in 38 seconds, and the reply a supervisor approved went out 26 minutes after she wrote.
Who reviews AI triage decisions?
People review at three points, and a buyer should ask to see all three. First, before anything is sent: a representative edits and sends every reply, and a supervisor who did not draft it checks every reply on a complaint and every reply that confirms a claim, closure, fee refund or account change. Second, where the agents are unsure: below 0.85 confidence the representative decides and gives a reason code. Third, afterwards: a sample agent draws 5% of the day's other sent replies for a supervisor to check the next morning, and it never passes or fails a reply itself.
Every split, clock, edit and approval on the audit trail
Is AI email triage safe for regulated messages? What to ask before you buy
It can be, if the tool is built to make misses visible, keeps every decision about a claim with a person, and lets you prove its recall on your own messages. The questions below separate tools that can show that from tools that can only show topic accuracy.
Ten questions to put to any AI email triage vendor for a bank's secure inbox
- Will you run a planted test on our own messages, blind?
Recall on hidden notices, with the answer key held by our compliance team, is the result that matters.
- Does the tool split a message into requests, or label it once?
A single label cannot represent a message with three requests.
- What date does it give a notice found on Monday that arrived on Saturday?
The right answer is Saturday: receipt, not reading.
- Does it treat a request for information about a transfer as a notice?
Comment 11(a)-5 says it must, unless it is plainly for tax or records.
- What happens below its confidence threshold?
A person should decide, with a reason, and the decision should be logged.
- Can it ever close a clock or decide a claim?
It should not. Dating is the tool's job; deciding is the specialist's.
- How does it handle an outage flood?
Bulk replies need a way to hold out the messages that also carry a notice or complaint.
- Which systems does it hand off to, and with which date?
The dispute should arrive with the customer's notice date, not the keying date.
- What is checked before a reply is sent?
Promises of refunds or timing outside policy should be blocked, not just flagged.
- What will an examiner see?
Every split, clock, edit and approval, by agent or person, exportable and masked.
Where Secure Message Triage fits: a bank's or credit union's secure message center, connected to its digital banking platform, core system, card processor's dispute system and complaint system, with the bank's own clock rules, calendar and approved language. Where it does not: it is not a general-purpose sales or marketing inbox tool; it does not investigate or decide disputes, which belongs to the bank's dispute team and to Botminds' Reg E Dispute Management; and it does not authenticate customers, which happens before a secure message is ever written. The Splitting agent is one of seven, and the solution dates each notice with a separate Clock agent whose tests are set out in its own guide.
Questions people ask
Can AI classify customer emails for a bank?
Yes, and banks already use AI in customer service; the question is what the classification is for. Sorting by topic is a convenience. Recognising a notice of error inside a message is a compliance task, because the 10-business-day clock runs from receipt. Test the second one separately.
How many planted notices does a test need?
Enough to make the result mean something. With no misses in 50, the 95% upper bound on the true miss rate is about 5.8%; with none in 100, about 3.0%; with none in 200, about 1.5% (exact binomial bounds, our arithmetic). Repeat the test after any change to the model, the rules or the message center.
Should the test use real customer messages?
Yes, masked. NIST's framework asks for test sets "representative of conditions of expected use", and a vendor's sample set does not carry your customers' phrasing, languages or product names. Mask account and card numbers before the set leaves your control.
Is email triage the same as a chatbot?
No. A chatbot answers a customer in real time; triage reads a written message, routes its requests and drafts a reply that a person sends. The legal point is the same for both: a dispute raised in either channel is a notice, and the rules do not change because a machine read it first.
What should happen to a message the tool is unsure about?
It should go to a person with the tool's reasoning attached, not be dropped into the most likely queue. In Secure Message Triage the threshold is 0.85; below it the representative chooses among the options shown and logs a reason, and the choice appears on the audit trail.
Can a triage tool start a Reg E clock too early?
It can flag a request that turns out not to be a notice, and that costs a specialist some review time. Starting early is the safe error: a notice the bank treated as one when it was a question harms no one, while a notice treated as a question leaves the customer's 10 business days running unseen.