Call center calibration: when two reviewers hear one call differently

Call center calibration is the practice of having several QA reviewers score the same calls independently, comparing their decisions, and agreeing on the right answer, so that every reviewer applies the scorecard the same way. In a bank it matters most for the findings that start a legal obligation: whether a customer gave a Regulation E notice of error on the call, whether what they said counts as a complaint under the bank's policy. Those findings are rare, and rare findings make agreement look better than it is: two reviewers who both mark "no finding" on most calls will score high percent agreement even when they seldom agree on the calls that matter. Botminds' Call Compliance Monitoring has agents propose findings on every call, and calibration is how a bank knows whether its people, and then its agents, would decide each one the same way.

The worked example uses a fictional sample bank, Corvane National Bank, and a calibration set we built for this page; the statistics on agreement come from published research, linked where they appear.

What is call center calibration?

Call center calibration is a recurring exercise in which QA reviewers, and often supervisors, score the same calls without seeing each other's answers, then compare and settle the differences against the scorecard and the rules behind it. Its output is twofold: a measure of how consistently the team decides, and a set of written decisions ("a lost card with a charge the customer does not recognise is a notice of error") that make the next round more consistent.

Calibration is a measurement before it is a meeting. If the team cannot say how often two reviewers would reach the same decision on the same call, it cannot say whether a finding rate that changes from month to month reflects the bankers or the reviewers. Nor can it judge an automated reviewer: an agent tested against one analyst's answers inherits that analyst's habits, and a bank that buys agents to review every call needs to know what "right" means before it can count how often they get there.

Percent agreement hides the calls that matter

Percent agreement is the share of calls on which two reviewers made the same decision, and on rare findings most of that agreement is on calls where nothing happened. In the matrix above the two analysts agree on 46 of 50 calls. But six calls contained something at least one of them heard as a notice of error, and they agree on two of those six. On the question that starts a 10-business-day clock under 12 CFR 1005.11, the team is close to a coin toss.

Statisticians have known this for decades. Jacob Cohen objected in 1960 that percent agreement takes no account of agreement reached by chance, and introduced the kappa statistic to correct for it, as Mary McHugh's review of the statistic sets out (McHugh, Biochemia Medica, 2012). Kappa asks how much better the reviewers did than they would have by chance, given how often each one flags. The same review gives an example from laboratory work that maps onto call monitoring almost exactly: comparing human and automated detection of abnormalities, the study found "only moderate agreement between the human versus the automated raters for the kappa (κ = 0.555), but the same data produced an excellent percent agreement of 94.2%."

How do you measure QA reviewer agreement? Kappa, worked on 50 calls

Measure QA reviewer agreement with Cohen's kappa for two reviewers (Fleiss' kappa for three or more), alongside the plain count of how many flagged calls both reviewers flagged. Kappa takes three numbers you already have and four steps of arithmetic.

1. Observed agreement. The calls where both said notice plus the calls where both said no notice, over all calls: (2 + 44) ÷ 50 = 0.92.

2. How often each reviewer flags. Analyst A flagged 4 of 50 calls (8%) and Analyst B flagged 4 of 50 (8%).

3. Agreement expected by chance. The chance both flag, plus the chance both do not: (0.08 × 0.08) + (0.92 × 0.92) = 0.0064 + 0.8464 = 0.8528.

4. Kappa. (observed − chance) ÷ (1 − chance) = (0.92 − 0.8528) ÷ (1 − 0.8528) = 0.0672 ÷ 0.1472 = 0.46.

Report one more number next to it: of the calls either reviewer flagged, the share both flagged, here 2 of 6. It is the figure a compliance officer understands at once, and it does not depend on how many empty calls were added to the set. Kappa is the right summary across calibration rounds; the count of shared flags is what tells a manager which calls to put on the table.

What is a good calibration score?

For findings that start a regulatory clock, a good calibration score is a kappa of at least 0.80 together with agreement on most of the calls either reviewer flagged; anything under 0.60 means the team is not yet deciding the same way. Those thresholds are our recommendation for bank QA. The research behind them is more cautious than the bands most teams use.

Cohen's own scale calls 0.41 to 0.60 "moderate" and 0.61 to 0.80 "substantial". McHugh argues that "Cohen's suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable", and concludes that agreement below a kappa of 0.60 is inadequate and the results behind it should carry little weight. She also notes that many textbooks set 80% as the floor for percent agreement, a floor which on rare findings, as the bars above show, any pair of reviewers can reach by doing nothing. The same thresholds applied to due diligence reports are worked through in EDD report reviewer agreement.

Statistical results for testing hypotheses about the relationship between independent and dependent variables become meaningless if there is inconsistency in how raters score the variables.

Mary L. McHugh, Biochemia Medica, 2012

Calibrate the findings that start a clock before the courtesy score

Most calibration sessions spend their time on the soft items, because those are where reviewers argue: was the banker empathetic enough, was the hold explained well. In a bank the order should be reversed. The findings that create a regulatory obligation, a notice of error, a complaint under the bank's policy, a misstated fee or hold, are the ones where a disagreement between reviewers means one of them is leaving a violation in place.

They are also the ones a rule can settle. A notice of error under Regulation E can be spoken or written, and has to identify the customer and account and say why the customer believes there is an error; the reviewers' disagreement is usually about whether a given sentence meets that test, and reading the rule together resolves it. A complaint is defined by the bank's own policy, and when reviewers split on complaints the cause is often the definition rather than the reviewers. In 2014 CFPB examiners criticised at least one consumer reporting agency whose complaint program "lacked a formal definition for direct consumer complaints, did not provide training on how to handle such complaints, and inconsistently tracked such complaints" (CFPB Supervisory Highlights, Fall 2014); a bank's calibration session that keeps splitting on complaints may have found the same gap. Complaint vs inquiry on a bank call works through where that line usually falls.

How do you run a calibration session?

Run a calibration session in three parts: everyone scores the same calls alone, the team compares and settles each disagreement against the rule text, and someone writes down what was decided so the next session starts from it. The steps below are ours, written for findings with a legal clock.

The discussion goes faster when everyone can open the same passage. At Corvane each finding links to the source the agent relied on, highlighted, and opening it is logged, so a calibration session can settle a disagreement on the rule's own words rather than on someone's memory of training.

Settling a disagreement on the rule's own words

How often should call center calibration sessions be held?

Hold call center calibration sessions on a fixed rhythm, monthly for most teams, and add one whenever something changes what reviewers have to decide. The FDIC's examination manual ties the frequency of compliance reviews to staff turnover and to changes in regulations and products (FDIC Consumer Compliance Examination Manual, II-3.1), and calibration is the same idea applied to the reviewers.

Four triggers are worth a session of their own: new reviewers joining; a change to a rule or to the bank's procedure, such as a revised complaint definition; a new product or call type; and a change to the agents, a new prompt, model or threshold. Corvane's own example shows why the procedure trigger matters. A knowledge article written for credit card billing disputes was linked to debit card questions on September 28, and by October 21 the same wrong advice ("call the merchant first") had been given on 37 calls by 11 bankers at three sites. Reviewers who had calibrated on the old article would have scored those calls as following procedure.

How do you calibrate AI call scoring? Against the consensus, not one reviewer

Calibrate AI call scoring the same way as people, with one change: the agent is scored against the answers the calibrated team agreed on, never against a single reviewer. A team with a kappa of 0.46 on notices of error cannot tell an agent's error from a reviewer's, so human agreement comes first and the agent's test set is built from the calls the team settled.

Once the agents are live, calibration becomes continuous. At Corvane a model risk validator runs each new agent version on a golden set of 1,500 interactions before the Chief Compliance Officer approves it. Every day after that, a sampling agent draws the Compliance Monitoring Manager's check: 10% of confirmations and every High-severity dismissal, each with a pre-check that compares the stated reason with the evidence. On October 20 that was 31 of 309 confirmations and all 9 High dismissals. The pre-check never agrees or sends back on its own; the manager does, and nobody checks their own decisions.

The manager's check sample: calibration every day, not once a quarter

Two more measures keep the agents honest between sessions. A share of the verifier agent's removals, 2% at Corvane, goes to QA review, so a filter that quietly drops real findings shows up. And misses have to be tested by speaker: automatic transcription errs more for some groups of speakers than others, so a calibration set that never includes them can certify an agent that misses their disputes. Testing transcripts by speaker first covers how.

When reviewers disagree because the rule is unclear

Not every disagreement is a reviewer problem; some are a policy problem, and calibration is often where a bank first sees it. If two careful analysts split on whether "this is the second time I've called" is a complaint, the complaint policy has not said whether a repeat contact counts. If they split on whether a banker's "it should clear by Monday" was a misstatement, the hold procedure may not say what bankers are allowed to promise.

The fix is a written decision that changes the policy or the scorecard, approved by whoever owns the rule, and a record of the change. At Corvane rule changes are proposed by one person and approved by the Chief Compliance Officer, and the agents use the new version from the next call, which means a calibration decision reaches every future call rather than only the reviewers who were in the room. A team that runs a pilot on last month's calls should calibrate first, for the same reason.

Questions people ask

What is inter-rater reliability in call monitoring?

It is the extent to which different reviewers reach the same decision on the same call. It is usually reported as percent agreement, which overstates it on rare findings, or as Cohen's kappa, which corrects for the agreement two reviewers would reach by chance given how often each one flags.

Who should attend calibration sessions?

The QA analysts who decide findings, their manager, and the owner of each finding type: a dispute specialist for notices of error, the complaint officer for complaints, a supervisor for coaching items. Whoever approves rule changes should see the written decisions, because some disagreements turn out to be gaps in the policy.

How many calls should a calibration session use?

Enough positives to measure, not just enough calls. In the 50-call example, one changed decision moves kappa by as much as 0.18, and McHugh notes that a kappa "may have such a wide confidence interval (CI) that it includes anything from good to poor agreement." Pick calls so each finding type appears several times.

Should calibration scores be used in a QA analyst's performance review?

Use them to find training needs and unclear rules, not to rank people. A low kappa on complaints often means the complaint definition is unclear; scoring analysts on it pushes them to agree with each other rather than with the rule.

Does calibration apply to chats and secure messages?

Yes. A customer can give a notice of error or make a complaint in writing as easily as on the phone, and reviewers tend to read text faster and less carefully than they listen. Include chats in every calibration set.