A call calibration process is the routine by which a quality team checks its own checking: several reviewers score the same conversations, the differences are argued out in a session, and whatever caused the difference — the scorecard, the wording of a rule, the reviewer's reading of it — is changed. It is hard work because the thing being measured is a judgement about a sentence someone said, and because the conversations a team calibrates on are almost never the conversations that hold the breach. Botminds' Collections Compliance Monitoring checks every call, voicemail, text, email and letter from a day against the rules rather than a sample of them, and that changes what calibration is for. The question stops being whether two reviewers agreed on a score. It becomes: what did the checks miss last night, and how would anyone here know?
The screens and figures on this page come from a fictional sample day inside the solution — Northmere Bank, its own collectors and three agencies, on Monday 12 October. The bank, the agencies, the collectors and the borrowers are invented. The research figures come from regulators and standards bodies, linked where they appear.
One night in Collections Compliance Monitoring, and the two samples a person hears
41,286
conversations checked overnight — every call, voicemail, text, email and letter of the day
1,214
carried a finding; 38 of them red, each heard and confirmed by a person one at a time
1%
of the clean conversations re-heard by a quality analyst the next day
826
conversations a 2% sample would have heard that night — about one of the 38 reds
What is a call calibration process?
A call calibration process is a scheduled comparison of reviewers against each other and against the rule they are applying. The familiar version runs like this: a facilitator picks a handful of calls, each reviewer scores them alone, the group meets, the scores go up on a screen, and every disagreement is traced to its cause. Sometimes the cause is a reviewer who was wrong. More often it is a rule that two careful people can read two ways — and the output of the session is not a corrected score but a corrected rule.
That is already the practice regulators expect to find. The CFPB's debt collection examination procedures tell examiners to look at a firm's own monitoring before they look at its calls — its policies and procedures for monitoring calls in and out, attempts to communicate and every other outgoing message. The same procedures describe what examiners do next, which is exactly what a calibration session does — they "should listen to live calls and taped calls", having reviewed the reports, complaint logs, performance metrics and exception and trend reports that the firm produced for itself. A programme that cannot show how it tests itself is being asked to explain, on exam day, a judgement nobody wrote down.
In a hand-sampled programme the calibration sample and the monitoring sample are drawn from the same small pool, so the session can only ever argue about conversations somebody already chose to hear. That is the limit this page is about.
The two errors, and why only one of them shows up by itself
Every monitoring programme can be wrong in two directions. It can flag a conversation that was fine — a false positive — or it can pass a conversation that broke a rule, a false negative. The two are not symmetrical, and the asymmetry is the whole design problem.
A false positive announces itself. Someone opens the finding, listens, disagrees, and the disagreement is on the screen in front of them. It costs an analyst ten minutes and, if it is confirmed by mistake, it costs a collector an unfair coaching item. A false negative is silent. Nothing appears, nobody opens anything, and the only events that can surface it are a borrower's complaint, a lawsuit, or an examiner who listened to a call the programme had passed. By then the cost is not ten minutes.
The standards vocabulary for this is plain enough to borrow. NIST's AI Risk Management Framework asks that "Measures of accuracy should consider computational-centric measures (e.g., false positive and false negative rates), human-AI teaming, and demonstrate external validity (generalizable beyond the training conditions)." Both rates, not one. A programme that only ever reviews its own flags is measuring the first and guessing at the second.
How do you know the system missed a breach?
You do not, unless you deliberately listen to conversations the checks called clean. There is no other route to the number. A review of flagged conversations, however careful, is a review of the set the checks already agreed was interesting; it can tell you the share of flags that were right, and it can tell you nothing at all about the share of breaches that were flagged.
The ordinary alternative — sample a few percent of everything, by hand — fails at the arithmetic, and the arithmetic is worth doing in the open rather than asserting. If a breach of a particular kind happens k times in a night, and a sample covers a share p of conversations drawn at random, the chance the sample meets it at least once is 1 − (1 − p)k.
The chance a random sample ever meets a breach: 1 − (1 − p) to the power k
A breach that happens once (k = 1)
1 − 0.98¹ = 2.0%
One bad conversation in a night. A 2% sample meets it one night in fifty. At p = 5% it is 5.0% — one night in twenty.
p = 2% sampled
Three times in a night (k = 3)
1 − 0.98³ = 5.9%
At p = 5%: 1 − 0.95³ = 14.3%. Three breaches, and on about sixteen nights in seventeen a 2% sample still hears none of them.
p = 2% sampled
Thirty-eight times (k = 38)
1 − 0.98³⁸ = 53.6%
At p = 5%: 1 − 0.95³⁸ = 85.8%. Only when a fault is this common does a small sample become likely to meet it — and by then detection is no longer the problem.
p = 2% sampled
What the 2% sample hears on average
p × k = 0.02 × 38 = 0.76
About one of the thirty-eight. On the sample day that is 826 of 41,286 conversations heard, with roughly one red finding among them.
the solution’s own line
New York City's debt collection rules are worth reading for the shape of the sampling idea rather than as a ceiling. Where an agency does not record everything, the rules describe the alternative: "If a debt collection agency elects to record a randomly selected sample of at least 5% of all oral communications made or received by the debt collection agency, it must maintain a record of the total number of oral communications made or received monthly and the total number of such recorded oral communications." A 5% recording sample is a floor for what is captured, not a claim about what a quality team will find in it. Run the second line of the figure above against it: at three breaches in a night, more than four nights in five the sample is quiet.
Outside observers hit the same wall from the other side. The National Consumer Law Center's six-month review of Regulation F found "There was apparent compliance with call frequency limits, but that is often hard to assess." Its reason is instructive: limits are counted per account, and borrowers who feel harassed stop answering unknown numbers, so the calls that would show the pattern are the ones nobody picks up. Apparent compliance, measured at a distance, is not a measurement.
The clean-call sample: listening to what was not flagged
The fix is unglamorous and it is the centre of this page: take a random slice of the conversations the checks called clean, give it to a person, and have that person listen as if no check had run. In Collections Compliance Monitoring this is a setting rather than a habit — clean conversations heard by a person, set at 1% a day with options of 0.5%, 1% or 2% — and it belongs to the quality analyst. On the sample day, 40,072 conversations carried no finding, so 1% is about 400 conversations, spread across in-house collectors and all three agencies.
Three design points decide whether that slice is worth anything. It must be drawn at random, not from a queue someone curates, or it measures the curator. The listener must not see the checks' verdict before forming their own, or the verdict anchors them. And the slice must be stratified enough to cover every channel and every team: a sample drawn only from calls says nothing about the 8,516 texts, 3,941 emails and 1,493 letters that went out the same day.
What comes back from the slice is the only honest estimate of the miss rate a bank can quote, and it is each bank's own number. We do not publish one here: the solution does not publish an agreement rate between its checks and its reviewers, and a figure invented for a marketing page would be exactly the thing this guide argues against. What the method gives you is a number you produced, from your own conversations, that an examiner can watch you produce again.
Calibration as a setting in Collections Compliance Monitoring, next to the rules it tests
Agreement, measured: the same ten calls, three reviewers
The classic session still has a job, and it is the one measurement a machine cannot do for you: are your own people reading the rule the same way? Run it the way a method is run. Pick ten conversations — some flagged, some not, drawn at random from both pools. Give each reviewer the audio, the transcript and the account, and nothing else. Collect the verdicts independently. Only then meet.
Three numbers come out of a session like that, and they are different questions. The first is agreement between reviewers: on how many of the ten did all three reach the same verdict? The second is agreement between the reviewers' consensus and the checks: where the group settled on a breach, had the night's run proposed one? The third is the one people forget — agreement between the same reviewer and themselves, measured by slipping two conversations from a previous session back into the set unlabelled.
NIST's framework asks for exactly this pairing of a number with the conditions that produced it: an accuracy figure, it says, belongs with a clearly defined, realistic test set that represents the conditions of expected use, and a written account of how the test was run. A calibration figure with no description of how the ten conversations were chosen is a number without a question attached.
It also asks for distance between the measurer and the measured, noting that independent review makes testing more effective and blunts internal bias and conflicts of interest. In the solution that distance is organisational. The quality analyst who runs calibration is not the compliance analyst who confirms the night's red findings, and the compliance manager who co-signs a dismissal is a third person again. Where a programme scores outside agencies, the point is sharper still: the same checks run on the bank's own collectors and on every agency's, so the scores compare like for like rather than agency by agency against different reviewers.
A wrong flag is a finding about the rule, not about the collector
When a flag turns out to be wrong, the useful question is never "was the analyst right to dismiss it?" It is "which sentence fooled the check, and what does that tell us about how the rule is written?" A collector who said "your daughter asked me to call" is not disclosing a debt to a third party; a collector who says "the truck will get picked up on Friday" is making a statement whose truth lives on the account, not in the audio. One of those is a wording problem in a rule. The other is a fact the check has to go and read.
Collections Compliance Monitoring makes the dismissal carry that reasoning rather than disappear into it. A red finding cannot be closed by the person who disagreed with it alone: dismissing one asks for the reason and goes to the compliance manager with the recording, and it is dismissed only when he co-signs. The reason text is what the next rules version is written from.
What happens to one flag, from proposal to the next rules version
Agent
The night’s checkProposes a finding against a named rule, quoting the second or the sentence it happened in, and records the rules version it ran under.
Never: Never decides a breach, and never closes a red finding.
A person decides
Compliance analystOpens the finding, hears the moment with the account in front of her, and confirms or dismisses it.
Never: Never confirms from the summary alone — the moment is the evidence.
A person decides
Compliance managerCo-signs every dismissal, reading the reason and the recording. Without the second signature the finding stays open.
Never: Never co-signs a dismissal without the stated reason attached.
A person decides
Quality analystCollects the dismissals and the clean-call misses, finds the sentence pattern behind each, and drafts the rule change.
Never: Never edits a live rule — a change is proposed as a new version.
A person decides
Compliance manager, againApproves the new rules version. Every check from that night on records that it ran under it.
Never: Never applies a new version retrospectively to findings already decided.
Two habits follow from the diagram. A dismissed flag should never silently cost a collector anything — in the solution a coaching item exists only once a finding is confirmed, which is why a false positive is an expense for the quality team rather than a mark against the person on the call. And a dismissal with the reason "not a breach" is worthless. The reason has to name the sentence and say what about it misled the check, because that text is the raw material of the next version.
How often should you run calibration sessions?
Two cadences, not one, because the two errors surface at different speeds. The clean-call slice runs daily and automatically: it is a setting, it costs the quality analyst a fixed slice of the morning, and its whole value is that it never pauses. The session with people in a room runs on a slower clock — weekly while a rule set is new or a channel has just been added, monthly once the disagreement rate has settled, and immediately after any of four events.
The four events are worth writing into the procedure: a new rules version goes live; a new channel or a new agency starts being checked; a borrower complaint or a lawsuit names a conversation the programme passed; or an examiner questions a finding. Each of those is evidence that the rule set and the people reading it may have drifted apart, and each is cheaper to answer with a session than with a reconstruction months later.
The CFPB's survey of third-party collection firms describes the work this sits inside. Compliance staff, it reports, spend their time on activities that "typically included listening to call recordings, training staff, responding to consumer complaints, monitoring changes in state laws, and implementing the firm's policies and procedures." Calibration is not an extra task beside that list. It is the part of it that decides whether the rest of the list is working.
Testing a rule change before it scores anyone
A new rule is a hypothesis about language, and it should be tested on conversations whose verdicts are already known before it is allowed to produce findings. The method is the same one a careful analyst would use by hand, run at the size the overnight pass makes possible.
Take a set of conversations already decided — the confirmed reds, the dismissed flags with their reasons, and a block of clean-call slices that people have listened to. Run the proposed rule across them. Three columns matter: conversations the new rule catches that the old one missed, conversations it now flags that people decided were fine, and conversations where it changes nothing. The first column is the case for the change. The second is its price. The third tells you whether the change was about this rule at all.
Then look at what the rule needs to read. Some rules are decided by words, and a wording change is enough. Others cannot be decided from the audio at any wording — whether an attorney is on the account, whether a repossession has actually been assigned — and for those the fix is not a better phrase but a field from the servicing record. That difference is the subject of what a transcript can and cannot judge, and it decides which rule changes are even possible.
Calibrating a 2% sample, and calibrating a full pass with a re-heard slice
A hand-pulled 2% sample: About 826 of 41,286 conversations, chosen by whoever pulls the sample, almost always calls rather than letters or texts.
Every conversation, with a daily clean-call slice: All 41,286 checked against the rule set; roughly 400 of the clean ones re-heard by a person the next day, across all five channels.
A hand-pulled 2% sample: False positives inside the sample. False negatives outside it are invisible by construction.
Every conversation, with a daily clean-call slice: Both. Flagged conversations are re-heard for false positives; the clean slice is listened to for what the checks passed.
A hand-pulled 2% sample: Scores on the conversations somebody already chose, and how two reviewers read one scorecard line.
Every conversation, with a daily clean-call slice: Which sentence fooled a check, which account fact a rule needed, and what the next rules version should say.
A hand-pulled 2% sample: Pulling recordings, choosing the sample, and listening to conversations end to end — most of them uneventful.
Every conversation, with a daily clean-call slice: Hearing the flagged seconds, then the clean slice. The choosing is a setting rather than a morning.
A hand-pulled 2% sample: "That call was not in the sample." The programme has no position on it.
Every conversation, with a daily clean-call slice: The conversation was checked, under a named rules version, with the result and the decider on the record.
A hand-pulled 2% sample: A sample plan, scores, and session notes.
Every conversation, with a daily clean-call slice: The same, plus every conversation checked, the version each ran under, and the method that measures the misses.
What the calibration record has to keep
A calibration session that leaves no trace is a conversation, not a control. What has to survive is narrow and specific, and most of it is the kind of thing nobody writes down in the room.
What one calibration session has to produce
- The conversations heard, by identifier
Which calls, voicemails, texts, emails or letters were in the set, and how they were drawn — flagged, clean slice, or deliberately re-used from an earlier session.
- Who heard them, and what each person decided alone
Independent verdicts before the discussion, not the verdict the room settled on. The disagreement is the data.
- Where the reviewers disagreed, and on what
Named conversation, named rule, and the sentence at issue. "Reviewer B was stricter" is not a finding.
- The rule text that caused it
The exact wording that two careful readings pulled apart, copied as it stood that day.
- The version the change goes into, and who approved it
A rules version with named approvers, so every later check carries the version it ran under and findings are never re-scored under a rule that did not exist yet.
- The clean-slice result for the period
How many conversations were re-heard, how many turned out to hold something the checks passed, and what kind.
The retention side is set by rule rather than by preference. Regulation F requires a debt collector that records its calls to keep the recording of each one for three years after the date of the call, and to keep records that are evidence of compliance or noncompliance until three years after the last collection activity on the debt (12 CFR 1006.100). A calibration session that re-hears a conversation from eight months ago depends on that retention working; so does an examiner who wants to hear the same conversation you decided about.
In the solution, the keeping is a by-product rather than a task. The review log records every conversation checked, the rules version it ran under, the finding and who decided, append-only, and it exports as an exam package. The decision line that matters for calibration is the one that names a person and a time — confirmed by a named analyst at 09:12, with the moment attached — because that is what makes an agreement figure reconstructable rather than asserted.
The record a calibration figure has to be rebuilt from
Questions people ask
What is a call calibration session, and who should be in it?
A session is a scheduled comparison in which several reviewers score the same conversations independently and then meet to trace every disagreement to its cause. Keep it to the people who have to live with the answer: the quality analyst who owns the method, the compliance analyst who confirms findings day to day, a team lead who coaches from them, and the compliance manager who approves rule versions. Where agencies are scored on the same rules, their trainers belong in at least the sessions that touch their collectors.
How do you measure agreement between reviewers?
Count the conversations on which the reviewers reached the same verdict independently, before any discussion, and report it with the set it was measured on — how many conversations, how they were drawn, and how many were flagged versus clean. Measure the same reviewer against themselves too, by returning two previously decided conversations to the set unlabelled. Any agreement figure quoted without its test set tells you nothing you can act on.
What is a false positive in call monitoring, and who should pay for it?
A false positive is a conversation the checks flagged that a person, hearing the moment and reading the account, decides was not a breach. It should cost the quality team time and nothing else. In Collections Compliance Monitoring a coaching item exists only after a person confirms a finding, so a wrong flag never reaches the collector's record; the dismissal needs a stated reason and a second signature, and that reason is what the next rules version is drafted from.
What sample should a person still listen to when everything is checked?
Two. The flagged conversations, because a finding is only a proposal until a person has heard the moment. And a random slice of the conversations the checks called clean — set at 1% a day in the solution's own settings, with 0.5% and 2% as the alternatives — because that is the only place a miss can be found. Draw the clean slice at random, across all five channels and every team, and have the listener form a verdict before seeing what the checks concluded.
How do you test a new rule before it goes live?
Run it over conversations whose verdicts are already settled: confirmed findings, dismissals with their reasons, and clean slices people have listened to. Report three columns — what it newly catches, what it newly flags that people had decided was fine, and what it leaves unchanged. Then approve it as a new version with named approvers, so every check afterwards records which version it ran under and nothing already decided is quietly re-scored.
Does a full pass mean fewer people, or different work?
Different work. The hours that went into pulling recordings and choosing a sample largely go, but the hours that decide anything stay with people: hearing the flagged moments, listening to the clean slice, running sessions, and writing rule versions. The solution's own estimate of a compliance analyst's week moves the time rather than deleting it — and the judgement, the dismissal reason and the second signature are all still a person's.