Automated compliance testing vs a 64-loan sample: what sampling misses

Automated compliance testing is software that applies a bank's compliance rules to every transaction, here every closed mortgage, instead of a person applying them by hand to a sample. Its main value in loan testing is coverage. A random sample is a sound way to estimate how often errors happen at random, which is what most second-line programmes were designed to do. It is a weak way to find a systemic error, the kind caused by one fee code or one system setting, which fails the same rule on a handful of loans and then on every similar loan after. A sample of 64 loans from a quarter of 3,712 closings has about a one in nine chance of including even one of seven affected loans. Botminds' Loan Compliance Testing tests every closed loan against every rule that applies to it, so the question becomes how many loans failed and why, not whether the sample happened to land on one.

This page sets the two approaches side by side, works through the probability, and argues that coverage belongs in the accuracy measures a chief compliance officer and internal audit look at. No published figure gives the error rate of manual compliance testing; the argument rests on sampling arithmetic and on how examiners grade what they find.

What is automated compliance testing, and why does coverage matter?

It is the same tests a compliance tester runs, applied by software to the whole population, with the tester reviewing what the software cannot settle. The tests themselves do not change: the APR within tolerance, the Closing Disclosure received three business days before consummation, the 10% bucket within 10%. What changes is the population and the question a result answers. A sample answers "how often do we fail, roughly". Full coverage answers "which loans failed, and what do they have in common", which is the question remediation and root cause need.

Most sampling programmes are built to bound an error rate, not to find a rare fault. A sample of 59 files with no errors gives 95% confidence that the true error rate is below 5%; 64 clean files bound it at about 4.6%. That is a useful statement about common errors. It says nothing about an error that touches 0.2% of loans, which is what a new fee-table fault looks like in its first month.

Can a sample detect a systemic error?

Usually not while it is small, and a systemic error is small at first. When a fee-table change starts producing understated finance charges, the number of affected loans grows with every lock and every closing. In the sample bank a wholesale fee table, version 14, went live on 28 July and coded the discount points and the rate-lock extension as seller-paid. Seven wholesale loans closed in August with the error, out of 1,284 closings. Had the fault run through a whole quarter, a similar error might reach a few dozen loans before anyone looked.

A random sample sees those loans in proportion to their share. With 64 files drawn from 3,712, the chance of drawing at least one of seven affected loans is about 12%. At 21 affected loans, a month's error repeated for three months, it is about 31%. The sample starts to catch the error reliably only when it is already common: at 2% of loans, about 74 affected, the chance reaches about 73%.

A sample that finds one failed loan still cannot tell you the error is systemic

Finding one loan is not the same as finding the error. A single fail among 64 files looks like what examiners call an "isolated or sporadic" violation, the FDIC's lowest level of concern, and teams often treat it that way: correct the loan, note it in the workpaper, move on. To see a pattern a sample needs two or more fails with the same cause, and the odds of that are far lower. With seven affected loans in the quarter, the chance that a 64-loan sample draws two of them is under 1 in 100. With 74 affected, it is about 37 in 100.

The sample bank's own history shows the same limit from the other side. Its second-quarter sample did find late homeownership counseling lists, 3 of 64 files, and escrow statements missing from the file, 2 of 64. Both were noted in the workpaper. Both came back in August, when every loan was tested: 31 loans for the counseling list, 27 of them from the correspondent channel and two partners, and 18 loans for the escrow statement, traced to the same three settlement agents. The sample found the symptom. It could not show where it came from, so the fix missed it.

Examiners grade systemic violations as a separate, higher level

The reason this matters is how violations are weighed once found. The FDIC grades consumer compliance violations in three levels. Level 1 covers violations that are "isolated or sporadic". Level 2 covers "systemic, recurring, or repetitive violations that represent a failure of the bank to meet a key purpose" of a rule. Level 3 covers the highest concern, typically with restitution "in excess of $10,000 (in aggregate)". Of the 462 TILA and Regulation Z violations the FDIC cited in 2025, 451 were Level 2 and 11 were Level 3 (FDIC Supervisory Highlights, March 2026).

The FFIEC rating system asks examiners to weigh "root cause, severity, duration, and pervasiveness" for every violation (FFIEC, 2016). A sample measures none of those four well. It cannot say how long an error ran, because it holds a few loans from a quarter. It cannot say how pervasive the error is without a second pull. And the root cause of a systemic disclosure error usually sits in the loan system, which the sample does not test. The CFPB's examination procedures ask directly whether "an information system coding error led to inaccuracies across a portfolio" (CFPB compliance management review), and its examiners have found exactly that: a loan origination system "not programmed to round" as the note required (Supervisory Highlights, Issue 30).

The money follows the same logic. When an APR or finance-charge error comes from "a clear and consistent pattern or practice of violations", the agencies are required to order an adjustment for the affected borrowers (15 U.S.C. 1607(e)). A pattern is defined by the population, not by the sample. What one disclosure error costs follows that cascade.

A sample estimates how often things go wrong at random. The errors examiners grade highest are not random.

Is 100% loan testing better than sampling?

For finding systemic errors and sizing remediation, yes; for estimating the accuracy of the tests themselves, a sample still has a job. Full coverage changes the question a failed test answers. In the sample bank's August run, the same night that found the Whitcombe loan's understated APR found the six others with the same cause, all wholesale, all locked after 28 July, and gave the total: $38,940 of finance charges understated. The testing manager signed the finding 19 days after the Whitcombe loan closed. A quarterly sample, if it found the error at all, would have found it months later and with more loans behind it.

Every loan closed in August in Loan Compliance Testing, against last quarter's sample

Read the coverage bar first: it is the number a sample-based workpaper never shows.

Coverage does not make each test right. A test applied to every loan with a wrong rule fails every loan wrongly, or passes them. That is why the accuracy of the individual test still matters: the calculator recomputes from the payment stream instead of reading the disclosed APR, each result cites its page so a person can check it, and results the rules cannot settle, 167 of 31,592 in August, go to a tester. Full coverage moves the risk from "we did not look" to "we looked and got it wrong", and the second is the one a reviewer can measure. Testing the rules themselves on the previous month's loans before a new library version goes live is part of the same discipline.

Sampling still has a use. A testing manager may keep a small random sample for a second, manual review of the agents' results, which is the most direct way to estimate their error rate. That is a sample used to check the tests, not to stand in for them.

A full month at a glance: which loans failed, and which rules

Do regulators require 100% compliance testing?

No. No federal consumer-compliance rule sets a sample size for a lender's own monitoring; what regulators require is coverage commensurate with risk. The CFPB tells examiners to evaluate monitoring and audit programmes "to determine whether, considered together, they are commensurate with the institution's size, complexity, and risk profile", and whether monitoring is "comprehensive, timely, and successful at identifying and measuring material compliance risk management" (CFPB CMR procedures). The FDIC's examiners decide their own transaction testing the same way, to "conduct transaction testing where risks intersect with weaknesses in the CMS" (FDIC Consumer Compliance Examination Manual II-1.1).

Two expectations push towards full coverage for disclosure calculations in particular. The OCC expects banks to complete "sufficient testing and monitoring of automated processes to be able to rely on the accuracy of the outcomes, calculations, and disclosures generated", and to re-test results "when regulatory changes occur" (OCC Comptroller's Handbook, Compliance Management Systems). And an examiner who finds a pattern will ask how far it reaches, which a sample cannot answer without a second project.

How to judge automated compliance testing on coverage

Ask what a product proves about the loans it did not flag, not only what it flags. "We test every loan" can mean every loan against every rule, or every loan against the few rules that are easy to automate. The questions below separate the two, and each has a concrete answer a vendor can show on its own screens.

How to measure what your own compliance sample misses

A chief compliance officer can put a number on a sampling programme's blind spot without changing tools. The checklist below turns the arithmetic above into five steps, using the bank's own volumes and its own findings history. The numbers will differ by lender; the method does not.

Step four is, in practice, a pilot. A pilot of mortgage compliance software on last month's loans sets one out, and Closing Disclosure errors that start in the loan system covers the class of error a sample is least likely to catch. The Loan Compliance Testing page follows the August run from the first failed test to the signed workpaper.

Questions people ask

How many loans should be sampled for compliance testing?

It depends on what the sample is for. To show with 95% confidence that an error rate is below 5%, 59 clean files are enough; to have a 90% chance of drawing even one loan from an error touching 0.5% of a 3,712-loan quarter takes about 450, and seeing it twice takes about 700, by our calculation.

Is a risk-weighted sample a good middle ground?

It is better than a simple random sample at finding known risks, such as a new product or a new partner. It is no better at finding an unknown setup fault, because nobody knew to weight for it. The solution keeps random and risk-weighted sampling as settings for teams that want them.

Does automated compliance testing still need a person to check a sample?

Yes. A small random sample re-tested by hand is the most direct way to estimate how often the software is wrong, and internal audit will want it. That sample checks the tests; it does not stand in for them.

How quickly can automated testing find a systemic error?

As quickly as it runs. A month-end run found the sample bank's fee-table fault the morning after, 19 days after the Whitcombe loan closed; a nightly run of wholesale loans would have shown it on 4 August, the day after the first affected loan closed.

What does a missed systemic error cost?

Each month it runs adds loans to the remediation. In the sample bank, seven loans in August carried $38,940 of understated finance charges, about $5,600 a loan; left for a quarter at the same pace, the bill would roughly triple before the reimbursement rules of 15 U.S.C. 1607(e) are even considered.