Automated quality assurance in a call center: pilot on last month

Automated quality assurance in a call center means software reviews recorded calls and chats against the QA scorecard and the bank's rules, instead of a reviewer listening to a sample. For a bank, the reason to automate is coverage of the findings that carry a legal clock: a spoken notice of error, a complaint, a misstated fee or hold. The way to judge a product is not its demonstration but a replay. Take a month of calls your analysts have already reviewed, run the software over the whole month with your rules and your records, and count what each side found, what only one side caught, and what the software flagged that was not there. Botminds runs Call Compliance Monitoring on a sample of a bank's own recorded calls and chats in that way, and this page sets out the pilot in four weeks.

The worked figures come from a fictional sample bank, Corvane National Bank, whose month of sample data includes 2,104,380 calls and 381,920 chats and a former hand sample of eight calls per banker a month. Everything from Corvane is sample data; the pilot design and thresholds are our recommendations.

What is automated quality assurance in a call center?

Automated quality assurance is software doing the review work that QA analysts used to do by listening: transcribing each call, checking it against a scorecard and the rules for its call type, and flagging what failed. In most contact centers the scorecard is about service, script adherence and tone. In a bank the part that matters most is compliance, because some of what a customer says creates an obligation the moment it is said.

The clearest case is Regulation E. The rule treats a spoken notice of error like a written one, provided it identifies the customer and account and says why the customer believes there is an error (12 CFR 1005.11(b)(1)), and the bank's 10 business days run from receiving it. A QA program that hears one call in 65 cannot see most of those clocks start. Automation's job is to hear all of them. It does not decide what the bank does about each one; at Corvane a QA analyst confirms or dismisses every finding, and nothing opens until they do.

Why a replay of last month beats a vendor demonstration

A replay beats a demonstration because it uses calls nobody chose to make the product look good, and because your own analysts have already decided some of them. A demonstration set is curated; a month of your calls includes the transferred calls, the noisy lines, the chats that resume after an hour and the banker who never says the word the software is listening for.

One limitation needs saying plainly. Your analysts reviewed a small sample last month, so for most calls there is no prior decision to compare against. The replay therefore produces three groups: findings on calls your analysts also reviewed, where agreement can be measured directly; findings on calls nobody reviewed, which your analysts decide during the pilot; and seeded calls, where the right answer is known in advance. Only the third measures misses cleanly, which is why it carries the most weight.

What data do you need for automated QA?

You need the recordings and chat transcripts, the call metadata that says what each call was, the rules and documents the calls are checked against, and read-only access to the records that show what the bank did afterwards. Without the last group, software can say a customer disputed a charge but not whether anyone opened a claim.

Calls and chats. Audio with speaker turns, chat transcripts, call type, banker and team, and the transfer path, since a call moved from a branch line can lose the recorded notice on the way. Rules and documents. The rule library by call type, procedures and knowledge articles, the consumer fee schedule, hold notice templates and the complaint policy. Records, read-only. Accounts, holds and opt-in status from the core; card transactions from the processor; complaints and claims from the complaint and case systems, so a finding can say "no claim exists" rather than guess.

The data stays the bank's responsibility whoever processes it. The interagency guidance on third parties is direct: handing work to a vendor leaves the bank exactly as responsible as if it had done the work itself (Interagency Guidance on Third-Party Relationships, 2023; what the guidance asks at each stage is in interagency guidance on third-party relationships). For a pilot that means a data agreement, redaction of card numbers, security codes and Social Security numbers before reviewers see anything, read-only connections, and no change to the original recordings.

How do you pilot automated QA? A four-week plan

Pilot automated QA in four weeks: connect and load in the first, replay the month and the seeded calls in the second, let your analysts decide every new finding in the third, and score the results against the tests you agreed in the fourth. Four weeks is long enough to cover a full month of traffic and short enough that the people involved remember what they decided.

Week 1. Extract last month's recordings, chats and call metadata; load the rule library, procedures, fee schedule, hold notices and complaint policy; connect read-only to the core, card, complaint and case systems. Calibrate the analysts who will decide findings, so their decisions in week 3 mean the same thing, and write the seeded calls. Week 2. Replay the month and the seeds overnight. Week 3. Analysts decide every finding the software kept, exactly as they would in production, with the excerpt, the rule and the citation in front of them; a manager samples their decisions. Week 4. Score against the tests, read every miss, and decide.

One cause, nine calls, one decision: the kind of result a replay should produce

How many calls do you need for a pilot?

Replay the whole month, and seed at least 60 notices of error per group you want to measure. The month gives coverage and real false positives. The seeds give a miss rate you can trust, and the number follows from simple arithmetic: if the software finds all n seeded notices, the most its true miss rate is likely to be, at 95% confidence, is about 3 ÷ n. Sixty seeds caught without a miss supports a miss rate under 5%; 150 supports under 2%.

Seed by speaker as well as by wording. Automatic speech recognition makes more errors for some groups of speakers than others, and a miss rate measured on one group says little about the rest; seeding 60 per group for the groups that matter in your customer base is the only way to see it. Testing transcripts by speaker first explains the evidence and the test.

What should you measure in a call monitoring pilot?

Measure what each side found, what each missed, what the software flagged wrongly, and how long a finding takes to reach a decision, broken down by finding type and by speaker group. One accuracy number for the whole pilot hides the finding type that matters most.

"Can automated QA score calls like a human?" is the question most pilots set out to answer, and on compliance findings it is the wrong one, because humans do not score calls alike either. Measure your analysts' agreement with each other first, as call center calibration describes, and hold the software to the standard the calibrated team reaches, not to a single analyst's answers.

What are the risks of automated call scoring?

The risks of automated call scoring are deciding without a person, missing findings for some customers more than others, and scoring bankers on words they learn to game. Each has a precedent or a published reason behind it, and each can be designed out.

Deciding without a person. The CFPB's 2022 order against Bank of America concerned notices of error denied by an automated fraud filter alone (how to test AI email triage starts from it). Software can propose; a person has to decide. Uneven misses. Transcription accuracy varies by speaker, and every finding inherits the transcript, so a pilot that does not report misses by speaker group cannot rule this out. Gaming. When software scores bankers, bankers learn which words it listens for. Corvane's agents do not rate, discipline or report bankers; supervisors see confirmed findings for their own teams and decide what to coach.

Validation. The interagency model risk guidance issued in April 2026 leaves generative and agentic AI outside its scope (OCC Bulletin 2026-13; see whether a customer risk rating model is a model), so the bank has to set its own test for going live. The pilot's seeded set is the start of it: Corvane keeps a golden set of 1,500 interactions, runs every new agent version on it, and has the Chief Compliance Officer approve the version before it goes live.

Coaching drafted for the supervisor, and no coaching when the cause was a system

After the pilot: what goes live first

Go live first with the findings that carry a clock, notices of error and complaints, because those are where a missed call costs the most and where the pilot's seeded test gives the clearest evidence. Misstatements of fees and holds come next, once the fee schedule and hold notices are loaded and checked; required statements and referral cues after that.

Two things should be in place on day one regardless. The case a confirmed finding opens must carry the call date: a Reg E claim dated from the review rather than the call hides a missed deadline. And the manager's check must run from the first week, because the first month of live findings is when decision habits form. CFPB examiners have cited institutions that waited for a returned dispute form before starting to investigate an oral notice (CFPB Supervisory Highlights, Fall 2014), a habit a live system finds in the calls themselves. The rest of the design is on the Call Compliance Monitoring page.

A confirmed finding becomes a claim dated from the call

Questions people ask

What is the difference between automated QA and speech analytics?

Speech analytics turns calls into searchable text and trends; automated QA uses that text to check each call against a scorecard and rules and to raise findings. In a bank the second needs the first, plus access to the fee schedule, hold notices and case records the rules are checked against.

How long should a QA software pilot take?

About four weeks for one month of traffic: a week to connect and load rules, a week to replay, a week for analysts to decide the new findings and a week to score against the tests. A pilot that runs for months usually means the tests were not agreed at the start.

Should a vendor see our customers' data during a pilot?

Only under a data agreement, with card numbers, security codes and Social Security numbers redacted before anyone reviews the calls, and with read-only access to your systems. The 2023 interagency guidance on third parties is clear that using a third party does not reduce the bank's own responsibility for the activity.

What if the pilot finds notices of error nobody opened?

Treat them as real: the obligation existed from the day of the call, whether or not the pilot found it. Take them to Compliance, open claims with the original call dates where the bank decides to, and look for the cause, as the sample bank's lookback did when one knowledge article put the same wrong advice on dozens of calls.

Can automated QA score calls like a human?

On rule-based findings it can be tested against a calibrated team, and should be held to the agreement those humans reach with each other. On soft scoring such as empathy, human reviewers often disagree among themselves, so a vendor's "human-level" claim needs the human agreement figure beside it.