The choice between 100% inspection and sampling comes down to cost: sampling exists because checking every unit used to be destructive, expensive or slow, and a sample can tell you whether a batch is acceptable without checking it all. In policy servicing, where an outsourcer or an in-house team keys thousands of endorsements, certificates, cancellations and refunds a month, the usual practice is a monthly quality sample, often drawn and scored by the team whose work it is. That sample answers a narrow question with a wide margin of error, and it can miss the errors that cost the most. Botminds' Policy Servicing Analytics compares every finished transaction with the request that started it and with any later change that corrected it, so the sampled accuracy and the real one can be put side by side.
The figures on this page come from a fictional sample insurer, Ashcombe Mutual, whose servicing is split between two in-house teams and two outsourcers, over two weeks from 29 September to 12 October: 15,077 finished transactions. The statistics come from the US National Institute of Standards and Technology and the NAIC's examiners' handbook, linked where quoted.
Checked on every transaction, the outsourcers' error rates are two to five times what they report
100% inspection vs sampling: the short answer for policy transactions
Sample when checking is costly and you only need to accept or reject a batch; check everything when the check is cheap and each error matters on its own. NIST's handbook of statistical methods calls acceptance sampling "the middle of the road" approach between no inspection and 100% inspection, and lists when it is used: "Testing is destructive", "The cost of 100% inspection is very high", "100% inspection takes too long" (NIST/SEMATECH e-Handbook of Statistical Methods, section 6.2.1).
None of those three holds in the old way for a policy transaction compared by software with the email, form or call note that requested it. The check destroys nothing, it costs little once the comparison is set up, and it can run overnight on the day's work. What remains expensive is a person's judgement on the differences the comparison finds, and that is where the quality team's time should go.
The same handbook adds a warning that matters more than the cost argument: "the main purpose of acceptance sampling is to decide whether or not the lot is likely to be acceptable, not to estimate the quality of the lot." A vendor's monthly sample is often used for the second purpose, as the accuracy figure in a contract, which it was never designed to give.
What is the 100% inspection method in policy servicing?
The 100% inspection method checks every unit of output against its specification, rather than inferring the quality of the whole from a part. For a servicing transaction the specification is the request: the customer's email asking to add a vehicle from a given date, the agent's portal form changing a limit, the lender's letter replacing a mortgagee. Every field the request names is compared with what was keyed.
A second test catches what a field comparison cannot. If the insurer has to change the same field on the same policy within a set window, and the customer did not ask for the change, the first transaction was wrong. At the sample insurer the window is ten business days. That test found 268 vendor transactions redone in-house over ten working days, including a car added with the wrong effective date and no lienholder, which left a customer's claim on hold for 27 hours.
How effective is 100% inspection? Only as effective as its definition of an error. A check that compares the wrong fields, or ignores what happened afterwards, will pass bad work as reliably as a sample does. That is why the definition belongs in the contract, as it does in the sample insurer's agreement with its largest outsourcer: the insurer measures, on completed work, and a transaction it must redo within ten business days is not accurate.
What a vendor's sample of 220 can and cannot tell you
A sample of 220 can tell you the error rate within a band about three points wide, and it cannot tell you why the insurer's own measure finds several times more errors. The sample vendor checked 220 of its 2,486 endorsements for the month and found 2 errors, an error rate of 0.9%. Using the Wilson interval NIST recommends for proportions (e-Handbook, section 7.2.4.1), the 95% range for the true error rate behind 2 in 220 runs from about 0.3% to 3.3%. The vendor's sample is consistent with an accuracy anywhere from 96.7% to 99.7%.
Measured on all 2,486 endorsements, 147 were wrong, a 5.9% error rate. If 5.9% of endorsements were wrong and 220 were drawn at random, the chance of finding two errors or fewer is about two in ten thousand, by our binomial arithmetic. So the gap is not sampling error. The vendor's check and the insurer's are counting different things, or the vendor's 220 were not drawn at random from the month's work, or both.
There is also a floor below which a small sample cannot work with ordinary formulas. NIST's rule for the normal approximation to a proportion is that the sample be large enough that the expected count of defectives is at least five: "if p0 = 0.1, then N should be at least 50 and if p0 = 0.01, then N should be at least 500" (e-Handbook, section 7.2.4). For a desk running at around 1% errors, a 220-file sample is too small for the textbook test.
Accuracy on every transaction in Policy Servicing Analytics, with each vendor's own figure beside it
Who draws the sample matters more than how big it is
A sample drawn by the team whose work it checks cannot prove much, whatever its size. The NAIC's examiners' handbook tells examiners not to let the company being examined choose the sample, because nobody can then show the files were not picked in advance (the examiners' sample sizes are in the market conduct exam sample size calculator). It warns that pre-selection can be innocent, files on a bottom shelf, a branch nobody wants to visit, a computer-generated pull list that leaves out a date range, and still make a sample "unrepresentative and unsuitable for making inferences about the target population" (NAIC Market Regulation Handbook, 2017 edition, Chapter 14).
The same chapter puts the stakes in one sentence that every vendor manager could pin above the desk: "If a regulator places total reliance on the company, there would be no need for regulators to be there at all—and a self-report of the results of any sample drawn would be adequate." An insurer that relies on its outsourcer's self-drawn sample is in the position the examiners refuse to be in.
Where the errors cluster makes the selection problem worse. At the sample vendor, 14 processors who started on 28 September keyed a quarter of the month's endorsements and three quarters of its endorsement errors. A sample spread evenly across the month, or weighted towards experienced staff, would barely see them.
A quarter of the endorsements, three quarters of the errors: where one vendor's mistakes sit
Endorsements keyed (2,486)
- New starters, 612 · 25%
- Tenured staff, 1,874 · 75%
Endorsements wrong (147)
- New starters, 109 · 74%
- Tenured staff, 38 · 26%
Statistical sampling plans for inspection: AQL, LTPD and whose risk
A formal sampling plan is a sample size and an acceptance number: inspect n items and accept the batch if you find c defectives or fewer. NIST defines the two quality levels that set them. The acceptable quality level, AQL, "is a percent defective that is the base line requirement for the quality of the producer's product", and the lot tolerance percent defective, LTPD, "is a designated high defect level that would be unacceptable to the consumer" (e-Handbook, section 6.2.2).
The plan then balances two risks. Rejecting a good batch is the producer's risk; accepting a bad one is the consumer's risk, which NIST describes as the case where "the consumer suffers when this occurs, because a lot with unacceptable quality was accepted." In outsourced servicing the vendor is the producer and the insurer is the consumer. A plan designed by the producer is likely to be tuned to the producer's risk rather than the insurer's.
An acceptable quality level sampling plan also answers a yes-or-no question about a batch. It does not produce the accuracy figure a service credit is priced on, and it does not say which transactions were wrong, which is the information the in-house team needs to fix them before a customer calls. Those two jobs need every transaction, or at least a sample the insurer draws itself.
How examiners combine screening every file with sampling
Examiners already screen whole populations with data and sample only what needs a person. For billing notices, the handbook's "Automation Tip" tells the examiner to obtain a data file of all cancellations for nonpayment, count for each one the days from the date the company says the notice was generated or mailed to the date the cancellation takes effect (the data fields that makes possible are listed in notice compliance software requirements), and "extract a report of all notices, which apparently fail to comply with state requirements and submit to the regulated entity for explanations." The sample is used to test the ones that appear to conform.
When a sample does find a systematic error, the bill comes from the whole population. In a 2017 target examination, North Carolina's examiners took 119 policies at random from a list of 1,388 auto policies where physical damage coverage had been cancelled, and found premium refunds calculated incorrectly on all 119. At the examiner's request the company refunded $10,497.04; "As a result of the incorrect calculation of refunds, the examiner requested that the Company conduct a self-audit. The Company refunded an additional $392,052.17" (North Carolina Department of Insurance, target examination of Titan Indemnity Company, 2017). The sample found the error; checking everything priced it, at 37 times the sample's refunds.
A monthly QA sample against checking every finished transaction
Monthly sample: Is this month's work likely to be acceptable?
Every finished transaction: Which transactions are wrong, and what is the error rate?
Monthly sample: Often the team whose work it checks.
Every finished transaction: Nobody: all work is in scope.
Monthly sample: 2 errors in 220 means a true rate somewhere between 0.3% and 3.3%.
Every finished transaction: The count itself: 147 of 2,486 endorsements.
Monthly sample: Errors visible in the file on review.
Every finished transaction: Field differences from the request, plus later corrections the customer did not ask for.
Monthly sample: After the month closes.
Every finished transaction: Overnight: 1,912 weekend transactions checked by Monday 06:03.
Monthly sample: The process, eventually.
Every finished transaction: The 23 wrong transactions found that morning, before customers call.
Sample size for quality inspection: how big is big enough?
Big enough depends on the error rate you expect and how precisely you need to know it, and at the error rates servicing contracts care about the answer is large. The NAIC handbook gives the anchor points: "a sample of size 1,067 is necessary to produce a 95 percent confidence interval of ±3 percentage points when the population proportion is 50 percent", while "A sample of only 203 files supports an estimate of the same interval at the same confidence when the proportion is reduced to from 50 percent to 5 percent."
Files needed to estimate an error rate at 95% confidence, against one vendor's monthly endorsements
| Expected error rate | Precision wanted | Files to check | Share of 2,486 endorsements |
|---|---|---|---|
| 5% | ± 3 points | 203 | 8% |
| 5% | ± 1 point | 1,825 | 73% |
| 2% | ± 1 point | 753 | 30% |
| 1% | ± 0.5 point | 1,522 | 61% |
| 2% | ± 0.5 point | 3,012 | more than the month |
A contract floor of 98.0% asks whether accuracy sits above or below a line one or two points from where the vendor operates. Measuring that within half a point at a 2% error rate needs more files than the vendor keys in a month. At that precision, a sample is no longer cheaper than checking everything, and checking everything also names the wrong transactions.
Rework moved to the team that caused it, and each vendor's reported accuracy replaced by the measured figure
Where checking everything fits, and where sampling still belongs
Checking every transaction works where there is a written request to compare against and a field to compare: dates, vehicles, drivers, limits, additional interests, holder wording, amounts. It is the right tool for the high-volume keying that outsourcers do, and for in-house teams on the same request types, which at the sample insurer run between 98.6% and 99.0%.
Sampling still belongs where the judgement is the product: whether a call was handled courteously, whether a coverage explanation was clear, whether a referral was the right call. It also stays useful as a second look at the checker itself, a person reviewing a random draw of transactions the comparison passed, so that the definition of an error is tested too. In the solution, people still decide every difference that matters: a vendor can dispute a charged rework within five business days and the quality analyst rules on it.
The point of the comparison is not to retire the quality analyst. It is to start the analyst from every transaction already checked, rather than sixty files a day picked by hand from two thousand. How to score what the comparison finds, and how examiners would score the same files, is in the transaction quality audit checklist; what the measured figure means for a vendor's invoice is in the SLA penalty calculation formula. Or start from Policy Servicing Analytics.
Questions people ask
How effective is 100% inspection?
Done by people, it runs into the cost and time that NIST lists as the reasons sampling exists. When software compares each transaction with its request, consistency is not the problem; the definition of an error is. On the sample data, a definition that also counts later corrections found 181 wrong transactions where the vendor's own check found 2 in its sample.
What is an acceptable quality level sampling plan?
It is a sampling plan, a sample size and a maximum number of defectives, chosen so that batches at or better than the acceptable quality level are very likely to pass. NIST calls the AQL the producer's baseline quality requirement. The plan decides acceptance; it does not estimate the error rate or identify which items were wrong.
Do insurance examiners accept sampling?
Examiners sample themselves, under the NAIC handbook's rules on confidence levels and selection, and they expect samples they did not let the company pick. The handbook's 2017 edition sets a minimum 95% confidence level for the first-stage sample and says the regulator's second-stage confidence "should never be less than 90 percent."
Does checking every transaction slow the work down?
It need not, because it reads finished work rather than adding a step before it. On the sample, the weekend's 1,912 transactions were compared with their requests by 06:03 on Monday, and 23 that differed were drafted for send-back before the teams started.
Is a random sample still useful once every transaction is checked?
Yes, for two things: judgement work with no written request to compare against, and a periodic test of the checker by a person on transactions it passed. Both are better drawn by the insurer than by the team being checked.