A QA sample size calculator works out how many calls to review, and for call monitoring at a bank it has to answer two different questions. The first is how many calls you need to estimate a quality score or an error rate within a margin: a few hundred is often enough. The second is how many you need to hear the individual calls that create an obligation, such as a customer disputing a debit card charge, which under Regulation E is a notice of error the day it is said. That second number is far larger, because each such call has only the sample rate's chance of being heard: at 1.5%, a month with 20 spoken notices of error is missed entirely about three times in four. The calculator below does both sums. Botminds' Call Compliance Monitoring exists because the second sum cannot be made to work by sampling: its agents review every call and chat, and the sampling moves to checking the people who decide.
The starting values come from a fictional sample bank, Corvane National Bank: 2,486,300 calls and chats in a month, and a former QA rule of eight calls per banker a month for 4,800 bankers. Change them to your own.
QA sample size calculator: will your sample hear the calls that start a clock?
How do you calculate QA sample size for call monitoring?
Calculate QA sample size from the question the sample has to answer. To estimate a rate, such as the share of calls that pass the scorecard, use n = z² × p × (1 − p) ÷ e², where p is the expected rate, e the margin you can accept and z 1.96 for 95% confidence. To hear at least one of k rare events with 95% confidence, review n = N × (1 − 0.05^(1/k)) of N calls. The first gives hundreds; the second, for rare events, gives a large share of the whole month.
The difference is not a technicality. A quality score is a property of the whole population, and a sample estimates it well. A notice of error is a property of one call: it starts a 10-business-day clock for that customer under 12 CFR 1005.11, whether or not anyone at the bank listens. Estimating that "about 2.6 calls in every thousand carry a finding" is useful to a manager; it does nothing for the customer on the unheard call.
What is the chance a sample misses a complaint or a dispute?
The chance a random sample misses a given call is one minus the sample rate, and the chance it misses all k such calls in a month is roughly (1 − sample rate)^k. At a 1.5% sample, each spoken dispute has a 98.5% chance of going unheard, and with 20 of them in a month, 0.985^20 ≈ 0.74: about three months in four, QA hears none.
Odds a sample hears at least one of 20 spoken disputes in a month
A larger bank has more disputes, so its sample hears some, but it still misses almost all of them. At Corvane, the agents found 486 spoken notices of error in a month that bankers had not opened. A 1.5% sample would expect to hear about 7.5 of them and leave about 478 unheard. The sample would report a problem, correctly, and the bank would still owe 478 customers an investigation it did not know about.
At a 1.5% sample, most months pass with no spoken dispute heard at all
Months with 20 spoken disputes, at the sample bank's 1.5% rate
- QA hears none of them · 73%
- QA hears at least one · 27%
The same months at a 10% sample
- QA hears none of them · 12%
- QA hears at least one · 88%
What is a good QA sample size? It depends on the question
A good QA sample size is the smallest one that answers your question within the margin you need, and the table shows how much that depends on the question. Estimating a pass rate needs hundreds of calls. Estimating how often a rare finding occurs needs thousands. Hearing the individual rare calls needs most of the month.
Sample sizes for four questions a bank QA program asks
| Question | Assumption | Calls to review | What the answer gives you |
|---|---|---|---|
| What share of calls pass the scorecard? | 90% pass, ±5 points, 95% confidence | ≈ 140 | A quality score for the month |
| The same, more precisely | 90% pass, ±2 points | ≈ 860 | A score you can compare month to month |
| How often does a compliance finding occur? | 2.6 per 1,000 calls, ±1 per 1,000 | ≈ 9,900 | A finding rate, not which calls |
| Will we hear at least one of 20 spoken disputes? | 2,486,300 calls a month, 95% chance | ≈ 346,000 | One dispute heard; the other 19 mostly not |
One more piece of arithmetic is worth keeping beside any sample that comes back clean. If a sample of n calls finds no disputes at all, the most the true rate is likely to be, at 95% confidence, is about 3 ÷ n. Corvane's 38,400 reviews finding none would still be consistent with about 194 spoken notices of error a month across its volume. "We found none in the sample" is not evidence of none.
Four numbers to keep beside any QA sample
98.5%
chance any one call goes unheard at a 1.5% sample
73%
chance a month with 20 spoken disputes passes with none heard, at the sample bank's rate
3 ÷ n
the 95% upper bound on a rate when a sample of n finds nothing
6%
chance eight reviews a month hear a banker who mishandles four disputes a month
Reporting a QA score with its margin, per team and per banker
A QA score from a sample should be reported with its margin, and the margin shows how little a small per-banker sample can say. For a sample of n calls with a pass rate p, the 95% margin is roughly 1.96 × √(p × (1 − p) ÷ n). A team score built on 140 reviewed calls at 90% is 90% ± 5 points, good enough to track the team from month to month.
Shrink the sample and the margin swamps the score. A team of twelve bankers with 40 reviewed calls at 90% is 90% ± 9 points, so two teams scoring 84% and 93% cannot be told apart. A single banker with eight reviewed calls and seven passes scores 87.5% ± about 23 points, and at that size the normal approximation behind the formula is itself rough. A banker's monthly score from eight calls is mostly noise: useful as a prompt for a coaching conversation, not as a measure to rank people or to set pay.
The same arithmetic tells a QA manager where to spend a fixed review budget. Pool reviews to the level where the margin is small enough to act on, usually the team or the call type, and report per-banker results only as counts of findings with the excerpts behind them. A finding describes one call, with its excerpt; a score from eight calls is an estimate with a margin as wide as the scale.
How many calls should QA monitor per agent?
Per-agent quotas, such as Corvane's former eight calls per banker a month, are designed so that every banker gets feedback; they are not designed to find the calls that matter. A quota spreads reviews evenly across bankers, which is fair for coaching, and leaves the odds of hearing any one call the same low number for every banker.
Work it through for one banker. At Corvane's volume a banker handles about 518 calls and chats a month. Eight reviews hear about 1.5% of them. A banker who tells four customers a month to call the merchant first, each one a notice of error the bank was obliged to investigate, has about a 6% chance of any of those calls being reviewed. A knowledge article that sends the same wrong advice to every banker multiplies the problem: at Corvane one such article produced 37 calls by 11 bankers in three weeks, and the sample program's odds of hearing even one of them in those weeks were below even, about 44%.
Should QA sampling be risk-based?
Risk-based sampling improves estimates, and it helps find known risks: oversample collections calls if collections is where findings cluster, or new hires, or a product that just changed. It does not solve the problem of rare findings that can appear on any call, because the calls that carry them do not look different until someone listens.
A spoken notice of error can occur on a card servicing call, a balance inquiry that turns, a branch transfer or a chat. Selecting calls by keyword to "target" disputes runs into the problem examiners keep finding: customers do not use the bank's words. When CFPB examiners looked at error resolution, they found customers reporting unauthorized transactions being told to contact the merchant first (Supervisory Highlights, Fall 2014); nothing about those calls, in advance, marked them as disputes. Keyword spotting and AI call monitoring explains why listening for words misses them.
What percentage of calls should be monitored?
For findings that create an obligation on the call itself, such as notices of error and complaints, all of them; for scoring service quality, a sample sized to the margin you need. We know of no federal regulator that sets a call monitoring percentage. The FDIC's examination manual asks for monitoring and audit functions "that are appropriate for their size, complexity, and risk profile" (FDIC Consumer Compliance Examination Manual, II-3.1), which puts the choice, and the reasoning behind it, with the bank.
That reasoning is where sampling's arithmetic bites. A bank that knows customers dispute charges on calls, and knows its sample hears about one call in 65, is choosing not to hear the rest. The alternative used to be unaffordable: by hand, reviewing every call at about 18 minutes each would need close to one reviewer per banker, as what one reviewed call really costs works through. Agents that review every call change the cost, and move the sampling to where it still belongs.
Where sampling still belongs: checking the people who decide
When agents review every call, sampling moves from calls to decisions. The questions become whether the analysts' confirmations and dismissals are right, and whether the agent that filters flags is dropping real findings, and both are estimation questions that a sample answers well.
Corvane's settings show the design. The Compliance Monitoring Manager checks 10% of confirmations and every High-severity dismissal, drawn by a sampling agent that pre-checks each decision against the evidence; on October 20 that was 31 of 309 confirmations and all 9 High dismissals. Two percent of the verifier agent's removals go to QA review. And calibration sessions test whether analysts agree with each other at all, as call center calibration describes. Each of those samples is sized to estimate a rate, which is what samples are for.
Sampling the decisions, not the calls
Every call reviewed, and a month's decisions counted
The design behind both screens, and the limits on each agent, are on the Call Compliance Monitoring page.
Questions people ask
Is random sampling enough for compliance monitoring?
It is enough to estimate how often something happens, and not enough to find each occurrence. For obligations that start on the call, such as a Regulation E notice of error, the bank needs every occurrence, and a random sample hears each one only at the sample rate.
How many calls do banks monitor?
We know of no official statistic, and no regulator that sets a number. Where calls are reviewed by hand, a quota per banker is one way to set it; the fictional sample bank's was eight calls per banker a month, about 1.5% of its volume. The calculator above shows what any such quota can be expected to hear.
If our QA sample found no disputes, does that mean there were none?
No. If a sample of n calls finds none, the true rate could still be as high as about 3 ÷ n at 95% confidence. For 38,400 calls that is about 78 per million, or roughly 194 a month at the sample bank's volume.
What sample size do I need to estimate a QA score?
For a pass rate around 90% with a margin of ±5 points at 95% confidence, about 140 calls; for ±2 points, about 860. Use n = 1.96² × p × (1 − p) ÷ e², and calculate it per team or call type if you want to compare them.
Does doubling the sample fix the problem?
It doubles the chance of hearing any one call, from 1.5% to 3% in the sample bank's case, and doubles the cost. With 20 spoken disputes in a month, the chance of hearing at least one rises from about 26% to about 46%; the chance of hearing all of them stays near zero.
Should chats be sampled separately from calls?
Yes, if you sample at all: chats and calls carry findings at different rates, so a combined sample can hide a channel that is doing badly. Size each channel's sample to its own margin, and remember that a chat about an unrecognised charge is a written notice of error with the same clock as a call.