To test AI complaint classification, or any complaint classification process, plant a batch of messages whose correct answers you already know into the normal intake, through the real channels and unmarked, and measure what share of each type comes back correctly classified and routed. The type that matters most is the one with a clock: a written notice of error must be recognised every time, because a miss means the acknowledgment and the 30-day response never start. Botminds’ Complaint Management is complaint management software whose classifier codes every complaint, sets its flags and starts its clock, and shows the sentence behind each decision; this page sets out a test second-line, QA and model risk teams can run on it, or on any classification process, before relying on it.
We know of no published accuracy benchmark for complaint classification, by people or by software. That makes a test with known answers the only honest evidence, and it makes the design of that test the thing to get right.
A 100-message seed set: 50 notices of error and 50 that should not be read as one
| Seeded message | How many | The right answer | Why it is hard |
|---|---|---|---|
| Notice of error, never called one | 24 | Reg X notice of error; acknowledge in 5 business days | An angry email or letter that asserts a servicing error without the words “notice of error”. |
| Notice of error labelled as a qualified written request | 10 | Reg X notice of error | The label suggests an information request. |
| Notice of error inside a longer complaint | 10 | Reg X notice of error, plus the complaint’s other codes | The error is one paragraph of several. |
| Information request, no error asserted | 12 | Reg X information request | Looks like a notice; asks rather than asserts. |
| Card or HELOC billing dispute in writing | 6 | Reg Z billing error | Different rule, different clock. |
| Transfer dispute by phone (transcript) | 6 | Reg E error; clock starts on the call | Oral, and in the contact-centre channel. |
| Complaint mentioning language, age or another protected characteristic | 10 | Fair lending flag | Often indirect, never labelled. |
| Repeat of an earlier notice, nothing new | 6 | Possible duplicate, for an analyst to decide | Must not be dropped; must not be treated as new without checking. |
| Template letter asserting a specific error | 6 | Reg X notice of error | Its wording suggests a campaign; its content is a real assertion. |
| Not a complaint (a note on a payment coupon, thanks) | 10 | Not a notice; a person confirms | Regulation X excludes notes on payment coupons. |
How accurate is AI complaint classification?
There is no general answer, because accuracy depends on what is being classified, and the only figure worth having is the one measured on your own complaints. Sorting a complaint into a product such as “mortgage” or “credit card” is an easy task with many clues. Recognising that a message is a notice of error, an oral Regulation E dispute or a fair lending signal is a hard one: the decisive words may be one sentence in a long letter, and the categories turn on legal definitions rather than topics.
The categories themselves are set by the program, not the vendor. A bank’s taxonomy usually has four layers: products, issues mapped to the CFPB’s issue names, the regulation and clock, and the risk flags. A vendor’s headline accuracy on some other dataset’s labels says little about the layers with legal consequences, the clock and the flags, which are the ones to measure type by type.
So ask for, and test, recall by type: of the notices of error that arrived, what share were recognised? Of the fair lending mentions, what share were flagged? Then precision, the share of flags that were right, which tells you how much analyst time the false alarms cost. An overall accuracy figure can be high while the rare, clocked types are missed, because they are a small part of the volume.
Why the classification step is the one to test
Everything downstream inherits it. The classification decides whether a Regulation X clock starts, which template the response uses, whether the fair lending officer sees the complaint, and which theme it joins. A drafted letter can be reviewed by its approver; a notice of error that was never recognised has no letter to review.
The enforcement record makes the point. The CFPB found that NOVAD, a reverse-mortgage servicer, “did not distinguish notices of error from other correspondence received from borrowers, and did not track the receipt of or responses to notices of error”, and put the harm at a minimum of $11.5 million (CFPB consent order, June 2024). That is a classification failure at a rate of nearly every notice. A seeded test is designed to detect a failure rate of a few percent, long before it becomes a pattern.
What regulators and standards say about testing AI accuracy
Less than buyers expect, but enough to design a test. The interagency model risk guidance issued in April 2026 replaced SR 11-7, states that generative and agentic AI models “are not within the scope of this guidance”, and “does not set forth enforceable standards” (Federal Reserve SR 26-2). The agencies have said a request for information on AI will follow. Meanwhile, the GAO found that regulators using AI themselves were “not using AI to make decisions autonomously or relying on it as a sole source of input” (GAO-25-107197), and responsibility for vendor output stays with the institution (Consumer Compliance Outlook, 2019).
The clearest public standard for the measurement itself is NIST’s AI Risk Management Framework. It asks for false positive and false negative rates, test sets “representative of conditions of expected use”, and results broken down by segment. A seeded test is a direct way to meet each of those.
“Accuracy measurements should always be paired with clearly defined and realistic test sets – that are representative of conditions of expected use – and details about test methodology.”
NIST AI Risk Management Framework 1.0, section 3.1
How do you test AI complaint classification before relying on it? Seed, run, score
The protocol has five steps, and the second is the one most teams get wrong. The seeds have to enter through the real channels, unmarked, alongside live complaints, or the test measures a clean demonstration rather than the program. A letter goes through the mail room and the scanner; a call goes through the contact-centre transcript.
A seeded complaint classification test, from answer key to sign-off
A person decides
Second line writes the answer keyDrafts the 100 seed messages and records, for each, the right type, flags, clock and theme, before anything is run.
A person decides
Seeds enter the real intakeThrough the mail room, mailboxes, the contact centre and the portal test channel, unmarked and spread over days.
Agent
Classification runs as normalThe agents read, code, flag and clock each message, and show the sentence behind every decision.
Never: see the answer key
A person decides
Score against the keyCount misses and false alarms by type and channel; read the cited sentence for every disagreement.
A person decides
Sign off or fixPass marks met: accept and schedule the next run. Missed: change the setting, rule or template and re-run.
Scoring is quicker when every decision carries its reason. In the solution, each classification and flag points to the sentence in the complaint that produced it. A tester who finds an information request coded as a notice of error can see whether the agent misread an assertion or the seed was ambiguous, which turns a count into a diagnosis.
The reason for each classification is on the record, so each score can be checked
How many test complaints are enough to measure accuracy? The arithmetic behind 50
A seeded test can only prove a miss rate is below some bound, and the bound depends on how many seeds of that type were planted. If none of 50 notices of error is missed, the 95% upper bound on the true miss rate is 5.8%. With 13 seeds and no misses, it is still 20.6%: the test has not ruled out missing one notice in five. One miss in 50 raises the bound to 9.1%.
What a perfect score can rule out depends on how many you planted
That is why the title says fifty. Below about fifty seeds of a type, a clean result still leaves room for a miss rate an examiner would not accept. Above a hundred, each run becomes a project. Fifty notices of error a quarter, plus a smaller rotating set of the other types, is a test a second line can sustain, and repeated runs tighten the bound further.
Which complaint classification errors matter most?
Not every misclassification costs the same, so pass marks should differ by type. A miss on a clocked or escalated type is a potential violation; a false alarm usually costs an analyst a few minutes. Set the bar where the consequence is.
Each kind of classification error, what it causes, and a pass mark to start from
| Error | What happens | Suggested pass mark |
|---|---|---|
| A notice of error read as a plain complaint | No acknowledgment, no 30-day response: a Reg X violation for that borrower. | Zero misses in 50; investigate any miss before go-live |
| A fair lending mention not flagged | The fair lending officer never sees it; an examiner might. | Zero misses among the fair lending seeds |
| An oral EFT dispute not clocked | The 10-business-day Reg E investigation does not start. | Zero misses |
| A new error treated as a duplicate | A required response is not sent. | Zero; duplicates go to a person, never closed automatically |
| A plain complaint read as a notice of error | An extra acknowledgment letter and a shorter clock than needed. | Track the rate; tolerate a few |
| A benign mention flagged for fair lending | A few minutes of the fair lending officer’s time. | Track the rate; tolerate more |
What a seeded test cannot tell you, and what to add
Seeds measure the classifier against messages you thought of. They do not measure the messages you did not think of, or how the mix of real complaints changes. Two additions close most of the gap. Sample live complaints every month and have a second reviewer code them blind, which is the quality assurance process the OCC required Bank of America’s complaints framework to include (OCC consent order, 2022). And re-run the seeds whenever something upstream changes: the taxonomy, a template, a channel, or the classifier itself.
Working the live sample: filters for clocked, flagged and new complaints
The seeded test, as a protocol a second line can sign
- Write the answer key first
Type, flags, clock and theme for every seed, signed before the run.
- Plant 50 of the type that matters most
Notices of error for a servicer; billing errors or EFT disputes for a card or deposit book.
- Include decoys and hard cases
Information requests, coupon notes, templates and repeats, so over-calling is measured too.
- Enter through the real channels
Unmarked, spread over days, mixed in proportion to live intake.
- Score by type and channel
A miss rate for letters can hide a worse one for calls. NIST asks for results broken down by segment.
- Read the reason for every disagreement
Use the cited sentence to tell a classifier error from an ambiguous seed.
- Re-run on every change, and at least quarterly
Taxonomy, template, channel or model: any of them can move the result.
What to ask a vendor about AI complaint classification
Ask questions that can be answered with evidence on your own complaints, not with a slide. The aim is to find out, before signing, whether the tool can be tested the way this page describes, and whether it fails safe when it is unsure.
Eight questions for an AI complaint classification vendor, and the answer to look for
| Question | A good answer | A warning sign |
|---|---|---|
| What is your recall on notices of error, fair lending mentions and oral EFT disputes? | Figures by type, with the test set described, and an offer to run your seeds | One overall accuracy number |
| Can we run a seeded test through our real channels before go-live? | Yes, unmarked, with results by type and channel | Only on a sample you upload to a demo |
| Does each classification show its reason? | The sentence in the complaint behind every code and flag | A confidence score with no text |
| What happens when the tool is unsure? | It routes the message to a person and never decides “not a complaint” on its own | It picks the most likely label |
| Who can clear a fair lending flag? | Only a named fair lending officer | Anyone with edit rights, or the tool |
| Whose taxonomy does it use? | Yours, with the CFPB’s issue names mapped in | The vendor’s fixed list |
| How will we know when the model changes? | Notice before any change, so seeds can be re-run | Silent updates |
| Can results be broken down by channel and language? | Yes, as NIST’s framework asks | Only in aggregate |
The solution answers each in the left-hand way: every code and flag shows its sentence, doubtful messages go to an analyst, only the named fair lending officer clears a fair lending flag, and the taxonomy is the bank’s own, nine products and 41 issues in the sample bank. None of that is a substitute for the test. It is what makes the test possible to score.
The same results belong in the board’s view of the program. The complaint KPIs a board should see include classification accuracy alongside timeliness, and which clock each message starts is the answer key for the clock column. The Complaint Management page shows the classifier on the sample bank.
Questions people ask
Does model risk guidance require us to validate an AI complaint classifier?
The April 2026 interagency guidance leaves generative and agentic AI out of scope and is not enforceable, so it does not mandate a method. The institution still answers for missed clocks and flags, so evidence of accuracy is the institution’s own protection, and the seeded test is a form of that evidence a second line can sign.
What are recall and precision in complaint classification?
Recall is the share of real cases the classifier caught: 49 of 50 seeded notices of error is 98%. Precision is the share of its flags that were right: if it also flagged 10 information requests as notices, precision is 49 of 59, or 83%. For clocked types, recall is the measure that protects borrowers; precision measures wasted effort.
Does a vendor’s accuracy figure transfer to our bank?
Not reliably. NIST’s framework asks for accuracy to be measured on test sets that match the conditions the system will be used in, and another bank’s mix of channels, products and wording is a different condition. Treat a vendor figure as a reason to run your own test, not as its result.
Should we test people the same way?
Yes. The same seeds run through a manual team give the baseline the agents should beat, and the comparison is often the most persuasive result for a committee.
How often should the test be repeated?
At least quarterly, and after any change to the taxonomy, a template, a channel or the classifier. Each clean run also tightens the bound: two quarters of 50 notices with no misses together rule out a miss rate above 3.0%.