Testing into compliance is repeating a test, or changing its sample, method or reading, until a passing result appears, and then reporting the pass while the failure is set aside. The term comes from the U.S. Food and Drug Administration, which calls the practice "unscientific and objectionable" in drug manufacturing. In bank compliance testing it takes quieter forms: a sample redrawn after exceptions, an exception reclassified without a cause, a fix validated on a handful of friendly items. Each makes a test say something about the sample rather than about the population. Botminds' Consumer Compliance Testing removes the occasion for most of it by testing every item the data allows and keeping every result, with a person's reason on anything cleared.
The retest loop, and the test that does not need one
Testing into compliance
- A sample of 60 is drawnThe method is whatever the analyst does this quarter.
- Three exceptions turn upThe script would fail.
- Two are reclassified“Notice not found” becomes a documentation issue; one is called a teller slip, with no cause.
- A fresh sample is drawnOr the branch with the problem is left out as unrepresentative.back to step 2 if it fails again
- The script passesThe first result is not in the workpaper.
A defensible test
- Method fixed and approved firstPopulation, items or sample size, attributes and any retest rule.
- Every result keptPasses and fails, with the evidence for each.
- An exception is cleared only with a reasonA named person, a reason code, visible to whoever approves.
- Exceptions traced to a causeIsolated or systemic, and how far it reached.
- The fix is re-tested on new itemsWhere the error lived, by a method set before the re-test.
What is testing into compliance?
It is a result produced by the testing rather than by the thing tested. A test is meant to measure a population: a batch of tablets, a quarter of exception holds. When the person running it can retest, resample or reinterpret until the answer changes, and only the final answer is kept, the result measures the tester's persistence. The population is no more compliant than it was after the first test failed.
The FDA's wording is the clearest available: "FDA inspections have revealed that some firms use a strategy of repeated testing until a passing result is obtained, then disregarding the OOS results without scientific justification" (FDA, Investigating Out-of-Specification (OOS) Test Results for Pharmaceutical Production, Guidance for Industry, Revision 1, May 2022). OOS means out of specification: a failed result. Searches for the phrase usually lead to that guidance, which is why its definition is the one to start from.
This practice of "testing into compliance" is unscientific and objectionable under CGMP.
FDA, Investigating Out-of-Specification Test Results, May 2022
Where the term comes from
From drug quality control, where a failed laboratory result can mean a batch must be rejected. The FDA's guidance, in its current revision of May 2022 (FDA guidance page), sets the rules that stop a failure from being tested away, and each of them has a direct equivalent in compliance testing.
Retests are allowed, but the plan comes first: "The maximum number of retests to be performed on a sample should be specified in advance in a written standard operating procedure (SOP)," and "The number of retests should not be adjusted depending on the results obtained." A retest can replace a result only when a laboratory error is clearly identified, and even then the original data is kept. Otherwise there is "no scientific basis for invalidating initial OOS results in favor of passing retest results", and all results, passing and suspect, are reported (FDA, 2022). The guidance also recommends that a retest plan often use a different analyst from the one who ran the original test, and that an investigation of a confirmed failure extend to other batches that may share the cause.
Translate those four rules to a deposit test and they read: fix the method before the test; keep every result; let a different reader re-check; and when a failure is real, look for it elsewhere. That is most of what a sound compliance testing program does.
How it happens in bank compliance testing
Rarely as fraud and usually as a series of small, reasonable-looking choices, often under pressure to report a clean quarter. Five patterns cover most of it.
Redrawing the sample. Exceptions appear, the sample is judged "unrepresentative", and a new one is drawn. Reclassifying exceptions. A notice the branch cannot find becomes a documentation issue rather than a missing notice, or an error is called isolated with no cause found. Narrowing the population after the fact. A branch, a product or a date range is dropped once it is seen to fail. Changing the bar after seeing the items. A tolerance or an attribute is loosened once the results are in. Validating a fix on a friendly sample. A handful of items from a branch that never had the problem are tested, pass, and the issue closes.
The OCC's sampling booklet forbids the fourth outright for its own examiners: "To preserve the mathematical soundness of the statistical sampling process, examiners must not change the tolerance rate after they select the specific items for the sample" (OCC Comptroller's Handbook, Sampling Methodologies, version 1.0, May 2020). The others fail for the same reason: the conclusion stops being about the population.
Is retesting a failed sample allowed?
Yes, to investigate a failure; no, to replace it. Re-reading an item to check whether the first reading was wrong is good practice, and if it was, the corrected result stands with a note. What is not defensible is drawing a new sample because the first one failed and reporting only the second.
The OCC's rules for its examiners draw the line precisely. If more items need testing than the original sample held, "a new sample must be drawn", and it may be judgmental or statistical; the first sample's results do not go away. If a sample is stopped early because problems are clearly systemic, the results "should not be used to estimate the population exception rate" and the exceptions are documented as found (OCC, Sampling Methodologies). In both cases the failure stays on the record and the next step looks for more of it, not less.
How do you validate a fix by testing?
On new items, where the error lived, by a method set before the re-test. The FDIC expects management to come back later and check that each corrective action held (FDIC Consumer Compliance Examination Manual II-3, June 2019; the follow-up for refunds is inremediation testing in audit), and the OCC expects audit to review and validate management's actions on material weaknesses (OCC, Compliance Management Systems). Both assume the validation can fail.
Size is the trap. A fix that still fails on 5% of items will pass a 10-item re-test about 60% of the time and a 20-item re-test about 36% of the time. The chart shows the arithmetic; the cure is to re-test every relevant item after the fix, or a sample sized for the tolerance before anyone looks.
How often a small re-test passes a fix that still fails 5% of the time
At the fictional sample bank, the large-deposit hold finding came with its validation built in. Two branches had held whole deposits under a retired notice form; the issue could only close after every large-deposit hold at both branches in November was re-tested, on the same attributes, and passed.
Validating a fix in Consumer Compliance Testing: who re-tests, who decides
A person decides
Compliance managerApproves the finding and its cause, and with it the validation re-test the issue must pass before closing.
A person decides
Deposit operations supervisorPoints both branches to the current notice form and checks the form settings on every teller platform.
Agent
Issue agentAfter the fix, re-tests every large-deposit hold placed at the two branches in November, on the same five attributes.
Never: closes the issue
Agent
Reviewer agentRe-tests the re-test’s exceptions and a share of its passes, blind to the first answer.
Never: settles a disagreement
A person decides
Compliance managerCloses the issue only if the re-test passes, with a reason code on the record.
An issue that cannot close until its re-test passes
The Reg E version: which day starts the clock
Testing into compliance does not always involve a retest; sometimes the test is set up to pass. A Regulation E claim test that counts business days from the day a claim was keyed into the dispute system will pass claims that were late from the customer's side, because the rule's clock runs from the bank's receipt of a notice of error, oral or written (12 CFR 1005.11(b) and (c)). At the fictional sample bank, a $486.20 claim was provisionally credited on business day 9 counted from keying, and day 11 counted from the customer's first call. The blind re-test caught it because the second agent read the call notes; an earlier finding had already traced the same cause, and its issue named it plainly: a claim clock that started when the claim was keyed, not at the first call.
The same structure hides in other tests: a hold-notice test that reads only the notices the teller system printed, never the deposits that should have had one; a change-in-terms test that starts from the notices mailed rather than the changes made. The defence is to define each population from the obligation, not from the evidence of compliance.
Warning signs a program is testing into compliance
The program's own numbers usually show it. Exception rates that stay near zero while complaints about the same product rise. Exceptions cleared far more often than confirmed, or cleared without reasons. Samples redrawn or expanded only after failures. Populations that shrink between the request and the test. Issues closed on a re-test of a handful of items, or on the owner's word. Any one can have an innocent explanation; together they describe a program measuring its own persistence, and each is easy to check when every result and every clear carries a reason and a name.
What makes a compliance test defensible?
That someone who disagrees with the result can follow it from the method to the evidence and reach the same answer. That needs the method written before the test, every result kept, and every judgement attributable to a person with a reason. The Federal Reserve's examiners put documentation of "the scope, level of testing, deviations, file expansions" and follow-up on corrective actions at the centre of a credible audit function (Consumer Compliance Outlook, 2013); the same record defends a test.
A compliance test that cannot be tested into compliance
- The method is approved before the run
Population, items or sample size, attributes and any retest rule, with a version number.
- The population is reconciled
Against an independent count, so nothing is quietly left out.
- Every result is kept
Fails stay on the record even when a later reading corrects them.
- Clearing an exception needs a reason
A reason code and a name, shown to the person who approves the finding.
- A different reader re-checks
Blind to the first answer, on every exception and a share of passes.
- Confirmed failures are looked for elsewhere
Same cause, other branches, earlier quarters.
- Fixes are re-tested on new items
Where the error lived, by a method set before the re-test.
- The preparer does not approve
Findings, script changes and closures need a second person.
What do examiners think of resampling?
They accept a properly documented new sample and reject a result that hides the first one. The interagency rating system is the larger reason. It weighs how long violations lasted and is explicit about the worst case: "When violations are brought to the attention of an institution's management and management allows those violations to remain unaddressed, such violations are of the highest supervisory concern" (FFIEC Uniform Interagency Consumer Compliance Rating System, 81 FR 79473). A failed test that was resampled into a pass is exactly that: a violation management saw and left running.
The same system rewards the opposite. Self-identified violations that are resolved promptly can sit alongside a satisfactory rating; violations an examiner finds after the bank's own test passed them are evidence the program cannot be relied on. How examiners rate testing and audit covers that in more detail.
Full population testing removes the temptation
Most testing into compliance needs a sample to redraw or a population to trim. Testing every item removes both. In Consumer Compliance Testing each script runs on its whole reconciled population; a short count stops the test instead of running on what is left. Every item's result is kept with its evidence. An analyst can clear an exception, but only with a reason code, and the compliance manager sees the cleared exceptions when approving the finding: on the sample bank's large-deposit hold finding, the approval screen showed "Cleared exceptions: None".
Script changes follow the FDA's logic too. A change to a test is dry-run on last quarter's items and approved before it runs, and every version is kept with who changed it and why, so a loosened attribute cannot slip in after the results. What full population testing is covers the arithmetic of samples; the Consumer Compliance Testing page shows the library and the approvals.
The method is versioned: who changed a test, when and why
Questions people ask
What does testing into compliance mean?
Repeating or reshaping a test until it passes and reporting only the pass. The FDA's 2022 guidance on out-of-specification results describes it as repeated testing until a passing result is obtained, with the failed results disregarded without scientific justification.
Is testing into compliance illegal?
In drug manufacturing the FDA calls it objectionable under current good manufacturing practice. Banking has no rule that names it, but its effect, a known violation left unaddressed behind a passing test, is what the interagency rating system treats as the highest supervisory concern.
How many retests are acceptable?
As many as the written method allowed before the first test, and no more. The FDA's guidance asks for the maximum number to be set in advance and not adjusted to the results; a compliance program can use the same rule for re-reading items and expanding samples.
Who approves a test method?
Usually the compliance manager or testing manager, before the test runs. In Consumer Compliance Testing every script and every change to one needs the compliance manager's approval after a dry run on last quarter's items.
How do you record a test method?
With the population and how it was counted, the items or sample size and how they were chosen, the attributes and the rule sections behind them, any retest or expansion rule, and the version and approver. The OCC asks its own examiners to document the population, sample size, how items were chosen and why a sample was expanded or stopped.