What is full population testing, and what can a 59-item sample prove?

Full population testing means checking every item in a defined population against the test's attributes: every exception hold placed in the quarter, every Regulation E claim opened, every change-in-terms notice mailed, rather than a sample of them. In consumer compliance it answers a different question from sampling. A sample estimates how often something goes wrong; a full population run says which items went wrong, where and for whom. That matters because deposit errors cluster: one branch's form, one teller platform's setting, one batch of images. Botminds' Consumer Compliance Testing exists to make full population testing practical for Regulations CC, E and DD, with agents reading every notice beside its core record and people deciding what the exceptions mean.

What is full population testing?

It is a test with no sampling step. The population is defined, counted against an independent source, and then every item in it is tested. The OCC's examiner booklet on sampling makes the difference plain in a footnote: short of testing everything, nobody can rule out that some exceptions exist (OCC Comptroller's Handbook, Sampling Methodologies, version 1.0, May 2020). Only a full run can say "no exceptions"; a sample can only say "probably fewer than".

Auditors use the term for the same idea in financial audit, where software can recompute every journal entry. In deposit compliance the obstacle has never been the computing. It is the reading: the evidence for most attributes sits in a notice, a letter or a disclosure image, and those had to be read by a person, one at a time. That is the step agents now take over.

Where does the 60-item sample come from?

From attribute sampling arithmetic, whether or not the program writes it down. The OCC's examiners pick a tolerance rate, the exception rate they want to show is not exceeded, and a confidence level, then read the sample size off a table. At 95% confidence and a 5% tolerance, the table says 59. At 90% confidence it says 45. The booklet notes that the size "is independent of the population size": 59 serves for 200 holds or 20,000 (OCC, Sampling Methodologies).

Sixty items is therefore a sound answer to a specific question: is this process failing more than about one time in twenty? It is the wrong instrument for a different question a compliance program is also asked: did any customer get a hold notice with the wrong amount, and who?

What a clean sample proves, and what it cannot

A clean 59-item sample supports one sentence: with 95% confidence, the exception rate in the population is below 5%. On a quarter of 1,512 holds, 5% is 75 holds. The sample passes, correctly, while up to 75 customers could have been held too long. The OCC warns examiners against reading more into it: "A sample exception rate below the tolerance rate is, by itself, not enough to conclude that the tolerance rate is not breached in the population" (OCC, Sampling Methodologies).

The chart below shows the same thing from the other side. It takes the sample bank's 1,512 holds and asks how likely a random 59-item sample is to contain no exceptions at all, for different numbers of exceptions hidden in the quarter.

What does a sample miss? Errors that cluster

It misses errors that are rare across the bank and concentrated in one place, which is how deposit errors usually arise. A form, a teller-platform setting or a branch's habit produces the same mistake on every item it touches and none elsewhere. At the sample bank, two of 41 branches still printed a hold notice form retired on July 1, 2025. Every large-deposit hold at those two branches held the whole deposit instead of the excess over $6,725; the other 39 branches held only the excess on all 214 of theirs. In the quarter that was 6 holds in 1,512, a rate of 0.4%. A random sample of 59 would have missed all six about 79 times in 100.

Even a sample that catches one of the six leaves the job unfinished. The analyst sees one wrong hold and cannot tell whether it is a teller's slip or a pattern; the OCC's own advice to examiners is to look for exceptions that "have a common attribute or result from a deficient practice", and to consider a look-back "when there is a potential for consumer harm". Its booklet adds that once a deficiency and its likely cause are found, "the bank should use its resources to determine the extent of the deficiency" (OCC, Sampling Methodologies). That extent is the full population, tested after the fact.

Six exceptions with one cause, found because every hold was tested

Examiners cannot rule out a non-zero population exception rate unless they have tested the whole population.

OCC Comptroller's Handbook, Sampling Methodologies

Is a 60-item test a statistical sample at all?

Only if the items were picked at random. The OCC's booklet is strict about it: "If examiners use the sample sizes in table 1 but select the items from the population judgmentally rather than randomly, they are using judgmental sampling, not statistical sampling" (OCC, Sampling Methodologies). Plenty of bank testing programs fail that test without meaning to: the first 60 holds on the report, five from each of twelve branches, the items whose images were easiest to pull. Those are judgmental samples, and a judgmental sample supports no estimate of the population at all.

A random sample is not hard to draw; the booklet describes three ways to do it with a spreadsheet's random-number function. What is hard is keeping it random once the document requests go out. A sampled item whose notice cannot be found is an exception about evidence, not an item to swap for one whose notice can.

What one exception in a sample tells you

Less than it seems. When a 59-item sample finds one exception, the honest statement is a range: with 95% confidence, the rate in the population is no higher than about 7.8%. That figure is our calculation, using the binomial model the OCC's tables are built on; the same method reproduces the booklet's own worked example, two exceptions in 29 loans giving an upper bound of 20.16%. On 1,512 holds, 7.8% is up to about 118 holds.

The single exception does not say whether the others exist, where they are, or whether this one is a teller's slip or the visible edge of a pattern. That is why a sampled program that finds one usually ends up testing the population anyway, as a separate project: expand the sample, run the look-back, find the cause. Full population testing does that work first and reports the count.

Do examiners accept sampling?

Yes, and they sample themselves. The Federal Reserve's examiners describe transaction testing as "reviewing a sample of consumer transactions to verify compliance", used for the higher-risk areas of a risk-focused exam (Consumer Compliance Outlook, 2018). The CFPB's manual leaves the depth to the exam's scope (CFPB Supervision and Examination Manual, November 2025). The FDIC's Regulation CC procedures can come down to a single item: "Review at least one account deposit on which a large-deposit hold was placed," one step reads (FDIC Consumer Compliance Examination Manual VI-1, March 2024).

That is the point. An examiner samples to decide whether the bank's own program can be relied on. The bank's program is the thing being relied on, so its job is to find the errors, not to estimate them. The FDIC's 2025 figures show where the misses surface: Regulation E rose to the second most cited consumer law, with error resolution behind roughly three in four of those citations (FDIC Consumer Compliance Supervisory Highlights, March 2026). Each of those was an error the bank's own testing had not caught first.

The human baseline: measure the reader too

Sampling risk is only half the error budget. The other half is the reading: a sampled item that is misread passes. No public study measures how often compliance testers misread a hold notice or miscount business days, and a program that never re-tests its own work cannot know its rate either. The honest baseline is the one a bank measures for itself.

The way to measure it is the same for people and agents: a second, independent reader who does not see the first answer. In Consumer Compliance Testing that is a second agent on a different model family, re-testing every exception and a random 5% of passes; on the sample bank's third quarter it agreed on 468 of 469 items, and the one disagreement went to the analyst. The same design works on a sampled program: have someone re-test a share of last quarter's items blind and count the disagreements. How the blind re-test sits among the other tasks is set out task by task.

When is full population testing possible?

When the population can be listed completely and the evidence for each item can be read. Both conditions are about data, not effort. The first is a reconciliation: the hold extract against the core's daily hold report, the claim list against the claim register, the overdraft fees against the fee line in the general ledger. The Federal Reserve's examiners encouraged internal audit years ago to use system reports "to determine the universe" of an issue before deciding what to test (Consumer Compliance Outlook, 2013); the same report is the control total for a full run.

The second condition fails more often. Notices and disclosures live in imaging, and imaging has gaps. On the sample bank, a Regulation DD disclosure test could not run because one imaging batch covering three days had failed to index, leaving 43 images missing. A full population test that quietly ran on the rest would have passed accounts nobody read. Where the images genuinely do not exist, a sample is still the right tool: the bank's monthly statement test stays a sample of 40, because the statement images allow no more.

A short population stops the test instead of passing quietly

Is full population testing better than sampling?

For finding and fixing errors, yes; for estimating a rate, a sample is enough. Most deposit compliance questions are the first kind. A quarter's Regulation CC test exists to find the holds that kept back too much and the customers who paid for it, and to find the cause before it runs another quarter. The interagency rating system weighs how long a violation lasted and how many consumers it reached; a sample that misses a cluster adds a quarter to both.

Sampling keeps three honest jobs. It is the right tool where evidence for every item does not exist. It is how examiners and internal audit check a program, including a full population one. And it is how a bank measures its own reading accuracy, by re-testing a share of passes blind. For loans the same argument runs through different rules; sample against 100% loan compliance testing covers TILA and TRID. For deposits, the Consumer Compliance Testing page shows each step on one quarter, and testing into compliance covers the way a sample can be bent to pass.

Questions people ask

How many items should a compliance test sample?

It depends on the tolerance and confidence you need, not on the population size. The OCC's table gives 59 items for 95% confidence that the exception rate is under 5%, 99 for 3% and 299 for 1%. A sample that size can bound the rate; it cannot list the exceptions.

What is judgmental sampling?

Choosing items by experience rather than at random, for example the branches with the most complaints. The OCC says its key limitation is that "the resulting conclusions cannot be extrapolated statistically to the population". It is good for finding specific problems and useless for estimating how common they are.

What confidence level do compliance tests use?

Usually between 90% and 99%. The OCC's booklet offers 90%, 95% and 99% and tells examiners to use no less than 90%, higher where risk management is weak or the potential harm to customers is greater.

Does full population testing replace internal audit?

No. Testing every item is management's monitoring; audit is an independent check on whether that monitoring works, and it still samples. A full population program makes audit's job easier because every result has its evidence, but it does not make the program independent of itself.

Can a sample still be used after a full population test?

Yes, and it should be: a random share of passes re-tested by a second, blind reader is how a bank measures the accuracy of the full run. On the sample bank, a 5% re-test of passes plus every exception came to 469 items in a quarter.

What if the full population cannot be reconciled?

Then the test should stop and say why, not run on what it has. A partial run can be justified, for example when an imaging batch is still being recovered, but it should be labelled partial and approved by the compliance manager so nobody later reads it as complete.