AI for contact center quality assurance, and what it may not decide

AI for contact center quality assurance is software that transcribes conversations, tests them against rules and hands a quality team findings to work from. The question worth putting to it is not how accurate it is — nobody can answer that about your conversations before it has seen them — but which decisions it is allowed to make. In collections that question has teeth, because a finding scores a named collector, moves an agency's score, carries legal exposure and may be read back by an examiner a year later. Botminds' Collections Compliance Monitoring draws the line at proposing: six checks read every call, voicemail, text, email and letter from the day, each proposes findings with the quote and the rule, and a named person confirms or dismisses every serious one. This guide sets out where that line sits, how a programme like it is measured honestly, and what belongs in writing before the first night.

The screens and figures come from a fictional sample day inside the solution — Northmere Bank, its own collectors and three agencies, on Monday 12 October. The bank, the agencies, the collectors and the borrowers are invented, and no accuracy figure appears on this page: the solution publishes none, and the number that matters is the one your own run produces.

What is AI for contact center quality assurance?

Stripped of the category language, it is four capabilities in a row. Speech becomes text with speakers and timings. Text is tested against rules. Facts from other systems — a servicing record, a court file, an address — are joined to the conversation. And the results are written up as findings a person can act on. Only the first of those is distinctively about speech; the other three are ordinary software problems with an unusual input.

The framework most risk functions will ask you to speak in is NIST's AI Risk Management Framework, and its most useful point for this purpose is about roles rather than models: who decides and who oversees has to be defined and kept distinct, and the arrangement between people and a system can sit anywhere from fully autonomous to fully manual. That span is the thing to specify. A collections quality programme sits deliberately near the manual end for the decisions that count, and near the autonomous end for the work that would otherwise never get done.

Can AI decide if a call broke the law?

It should not be allowed to, and the reason is not modesty about software. A finding is not an observation; it is an act with consequences attached. It names a collector, becomes a coaching item, changes a score an agency is managed by, and becomes part of a record that a borrower's lawyer or an examiner may read. Decisions with those consequences belong to someone who can be asked why they made them.

What a check can do is everything up to that point, and doing it on every conversation rather than a sample is the part that is genuinely new. It can transcribe and time-stamp. It can place a contact on the borrower's own clock from the address on file rather than the phone's area code. It can count attempts per debt across seven days. It can read the servicing record and find that no repossession is assigned. It can quote the sentence. None of that is a verdict, and all of it is work a person would otherwise be doing badly because there is too much of it.

The regulator's framing supports keeping the two apart. An examination of a debt collection business is described in the CFPB's own procedures as being there "To assess the quality of the regulated entity's compliance management systems, including its internal controls and policies and procedures, for its debt collection business." Internal controls and procedures, not model outputs. A programme whose controls are legible — who decided, on what evidence, under which rule — is answering the question that is actually asked.

Propose and confirm: two jobs, kept apart

The separation is easiest to see laid out as the night runs. Six checks in order, each with a published limit, ending at the one step a person owns.

Read the "never" lines as the specification rather than as reassurance. "Never dials or blocks calls itself — it alerts" is an architectural decision with an operational cost: when five late contacts came from one dialer campaign on the sample day, the campaign was held after 20:00 Eastern by a person, not by the check that found them. That is slower. It is also the only version in which the organisation can say who stopped the campaign and why.

How do you measure an AI quality assurance system?

With two samples and a number you produced yourself. The first sample is the findings: a person hears each proposed serious finding and decides it, and the share that survive tells you how often the checks are right when they speak. The second is harder and matters more: a random slice of the conversations the checks called clean, listened to by a person who has not seen the verdict, which is the only way to learn what was missed.

NIST puts the obligation in a sentence worth quoting to anyone who wants to buy on a demonstration:

Measuring AI risks includes tracking metrics for trustworthy characteristics, social impact, and human-AI configurations.

NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023

The framework is also candid about why a figure from a vendor is weak evidence. Risk management of systems meant to augment or replace human activity, it notes, "requires some form of baseline metrics for comparison. This is difficult to systematize since AI systems carry out different tasks – and perform tasks differently – than humans." The baseline a collections desk actually has is its own previous programme, on its own conversations, with its own rule set — which is why this page carries no accuracy claim and why the method, not the number, is the thing to compare between suppliers. How that measurement is run in practice — the clean-call slice, the agreement session, the rule version — is set out separately rather than repeated here.

What the checks may never do

Four prohibitions are worth writing into a contract rather than trusting to design, because each of them is a thing a well-meaning deployment drifts towards.

No contact with a borrower. A monitoring programme that can send a message is no longer a monitoring programme, and its findings stop being a neutral record of what the collections operation did. No blocking or dialling. Holding a campaign, pausing a collector, stopping a letter batch — each is an operational decision with its own consequences for borrowers waiting on a payment arrangement, and each belongs to someone accountable for them. No writing to an agency. A remediation letter is a commercial and legal communication between two firms; on the sample day it is drafted with the recordings attached and sent by the agency management lead, who reads it first. And no closing a serious finding: the thing a check proposes, it may not dispose of.

There is a fifth that is about people rather than actions. The vulnerability listener routes an account; it does not label a person. The distinction sounds fussy until you imagine the alternative appearing in a record a borrower is entitled to see.

The record an examiner can follow: version, quote, decider

Three fields make a finding legible a year later, and all three have to be written at the time. The version of the rule set it was checked under, so that nobody has to reconstruct what the rule said in October. The quoted moment, so that the evidence does not depend on anyone's recollection. And the decider, with the time, so that the decision is attached to a person rather than to a system.

That record also has a retention obligation sitting under it: Regulation F requires records that are evidence of compliance or noncompliance to be kept until three years after the last collection activity on the debt, and recorded calls for three years after the call (12 CFR 1006.100). A programme whose findings expire before its audio does has a hole exactly where a late claim lands.

A finding becoming a decision, with the version, the quote and the decider

False positives, and who they must never cost

Every programme that checks everything will propose things that turn out to be wrong, and the design question is who pays for them. The answer has to be the quality team, in time, and nobody else. A proposed finding that reaches a collector's record before a person has heard it converts a measurement error into an employment one, and it also corrupts the measurement: collectors who believe the system is wrong about them stop volunteering the conversations worth hearing.

So the sequence matters. A coaching item exists only once a finding is confirmed. An agency's score moves only once a finding is confirmed. Until then the proposal is a queue entry with a quote attached, and the cost of being wrong is ten minutes of an analyst's morning. That is also why a wrong flag is treated as information about the rule rather than about the person: the dismissal reason names the sentence that misled the check, and that text is what the next rule version is drafted from.

What a confirmed finding turns into, and what an unconfirmed one does not

The dismissal that needs a second signature

The asymmetry in the solution's design is deliberate and worth copying. Confirming a serious finding takes one named person who has heard the moment. Dismissing one takes that person's stated reason plus a second person — the compliance manager — who reads the reason with the recording. Until he co-signs, the finding stays open.

Two things follow. The expensive direction is made harder than the cheap one: a desk under pressure can always clear a queue by agreeing with it, so the friction belongs on disagreement. And the dismissal reasons become a dataset. "Not a breach" teaches nothing; "the speaker at 00:41 was the borrower, not a third party — the check read the greeting as an identification" teaches the rule its next sentence.

What to put in writing before the first night

NIST's framework asks for one specific document, and it happens to be exactly the thing a model-risk reviewer, an auditor and a supplier's lawyer will each want: "Information about the AI system's knowledge limits and how system output may be utilized and overseen by humans is documented." Knowledge limits, and how a person oversees the output. Written down, before anything is scored on it.

For a collections programme that is six short sections. Which rules the checks decide and which they only surface, with the account facts each one reads and the system it reads them from. What each check may never do, in the operational sense — contact, dial, block, write to an agency, close a finding. Who confirms, who co-signs a dismissal, and who may change a rule. How the programme is measured: the clean-call slice, its size, who listens, and how disagreement is recorded. What is retained, in what form and for how long. And what happens when the checks are wrong often enough to matter — who can suspend a rule, and on whose authority.

The last one is usually missing, and it is the one an examiner's question eventually lands on. Supervisory findings in this area have turned on exactly the sort of thing a monitoring programme is meant to catch early: the CFPB reported in July 2024 that examiners found debt collectors' service providers communicating with consumers by telephone and text about a debt without disclosing that the communication came from a debt collector (Supervisory Highlights, Issue 34). Whatever a programme decides about such a conversation, the thing that has to exist is a record of what it decided and who agreed.

Where the analyst does the one job the checks may not do

If you are writing requirements rather than a policy, the procurement version of all this — channels, retention, agency oversight and what an examination asks for — is set out in the buyer's guide for a collections desk. And where the audio alone cannot settle a question at all, the five rules that need the account rather than the recording explains why some of these boundaries are not choices.

Questions people ask

Can AI decide if a call broke the law?

It should not be the thing that decides. A finding scores a named collector, moves an agency's score, carries legal exposure and may be read back by an examiner, so the decision belongs to a person who can be asked why they made it. What a check can do is everything up to that point: transcribe, place the contact on the borrower's own clock, count attempts, read the servicing record and quote the sentence — on every conversation rather than a sample.

How do you measure an AI quality assurance system?

Two samples, both of them yours. The proposed findings, re-heard by a person, tell you how often the checks are right when they speak. A random slice of the conversations the checks passed, listened to by someone who has not seen the verdict, tells you what they are missing — and nothing else can. Report both with the test set beside them: how many conversations, drawn how, over which channels and teams.

How accurate is AI call monitoring?

No honest figure can be given in advance, and this page does not give one. Accuracy depends on your rule set, your channels, your recording quality and your account data, and NIST's framework notes that systems meant to augment human activity are hard to baseline because they perform the task differently from people. Ask a supplier for its measurement method and for the right to run it on your own conversations, not for a number.

How do you stop false positives from scoring an agent?

By making confirmation the thing that creates consequences. In the solution a coaching item and an agency score change exist only after a person has heard the moment and confirmed the finding; until then it is a queue entry with a quote attached. A wrong flag then costs the quality team time and the collector nothing, and the dismissal reason — which names the sentence that misled the check — becomes the input to the next rule version.

Does an examiner accept AI findings?

An examination assesses the quality of your compliance management systems, internal controls and procedures, not a model. What has to hold up is the record: which conversations were checked, under which version of the rule set, what was found, who decided, and what was done about it — retained for as long as the rule requires. Findings that cannot quote their moment or name their decider are the problem, however they were produced.

What should AI never do in collections?

Contact a borrower, dial or block a call, write to an agency, close a serious finding, or label a person. Each of those is an act with consequences for someone outside the quality team, and each belongs to a named person. Holding a dialer campaign after a run of late contacts is the clearest example: the check found the pattern and alerted; a person held the campaign.