Audit sampling methods are the rules an auditor uses to choose which items to test when testing every item is not practical: random selection, systematic selection with a random start, haphazard selection, stratification into groups, selection in proportion to size, block selection and judgmental or risk-based selection. The PCAOB's standard defines audit sampling as "the application of an audit procedure to less than 100 percent of the items within an account balance or class of transactions for the purpose of evaluating some characteristic of the balance or class" (PCAOB AS 2315, Audit Sampling). In a delegated authority audit the items are a delegate's risks and claims, often thousands of them, and the auditor tests perhaps thirty files. How accurate that is depends less on the method's name than on how rare the breaches are. Botminds' MGA & Coverholder Audit, an agentic solution for MGA and coverholder audits, tests every record against the binder first and then samples the files, so this guide also sets the baseline against which that approach should be judged.
The arithmetic below uses a fictional delegate from the solution's sample data: Bayline Coastal Underwriters, a commercial property MGA, with 2,318 risks and 412 claims in a six-month audit period, 60 of which carry an exception against its binder. The chances are our own calculations from those counts.
How often a random sample of 36 files catches each kind of breach
What are audit sampling methods?
Audit sampling methods fall into two families: those that give every item a known chance of selection, and those that rely on the auditor's choice. The PCAOB lists random sampling, stratified random sampling, sampling with probability proportional to size and systematic sampling with one or more random starts as forms of random-based selection, and says either a statistical or a nonstatistical approach "can provide sufficient evidential matter when applied properly" (AS 2315.03 and footnote 4). What separates good sampling from bad is whether the sample can be expected to represent the population, and whether the auditor can say afterwards why each item was chosen.
In a delegate audit the population is not one list but several: risks bound, premium accounted for, claims reserved and paid, each with its own rare failures. That shapes which method suits which test.
Seven audit sampling methods, and what each misses in a delegate audit
| Method | How files are picked | Strength | What it misses in a delegate audit |
|---|---|---|---|
| Simple random | Every file has an equal chance, drawn with a recorded seed | Unbiased; the result estimates the whole book | Rare, serious breaches: a few risks over a limit are unlikely to be drawn |
| Systematic | Every nth file after a random start | Easy to run on a bordereau sorted by date | Patterns that repeat on the same interval, such as month-end binding |
| Haphazard | Files chosen without a conscious reason | Quick | Hard to defend afterwards; easy to drift toward files that are easy to pull |
| Stratified | Separate draws from groups: risks and claims, states, size bands | Every group is represented; fewer files for the same confidence | Breaches that sit in a group given a small allocation |
| Proportional to size | Larger premiums or claims more likely to be picked | Money-weighted: the big files are tested | Small files with big control failures, such as a missed sanctions screen |
| Block | All files from one month or one underwriter | Shows one process end to end | Everything outside the block |
| Judgmental or risk-based | Files chosen for risk signals: near a limit, prior findings, new staff | Finds failures the auditor expects | Failures nobody expected, unless random picks are kept beside it |
How accurate is a sample-based delegated authority audit?
A sample is accurate about rates and unreliable about rare events. If 60 of Bayline's 2,730 records carry an exception, a random 36 can be expected to include 0.8 of them, and on our arithmetic it includes at least one about 55 times in a hundred. That is enough to say "there are exceptions in this book", which is useful, and not enough to say which ones, which is the auditor's actual job.
The odds fall away for the breaches that matter most. Three risks bound over the $15 million per-location limit: a random 36 includes one of them about four times in a hundred. One hail claim paid $12,400 over a $50,000 settlement authority: about once in eighty samples. A random sample that returns clean on these tests is not evidence that the book is clean; it is evidence that the sample was small. Stratifying helps a little, by guaranteeing claim files are drawn, but even a dedicated claims sample of 12 from 412 includes the one overpaid claim less than three times in a hundred.
Hand-picked samples do better only when the picker knows where to look, and the picker usually cannot see the whole book. That is why the solution's sample audit starts the other way round: tests on every record first, so the auditor knows where the exceptions are before deciding which files to open.
What is sampling risk?
Sampling risk is the chance that a sample leads the auditor to a different conclusion than testing everything would have. The PCAOB's wording: it "arises from the possibility that, when a test of controls or a substantive test is restricted to a sample, the auditor's conclusions may be different from the conclusions he would reach if the test were applied in the same way to all items in the account balance or class of transactions." It adds the rule every sample-size argument rests on: "For a sample of a specific design, sampling risk varies inversely with sample size: the smaller the sample size, the greater the sampling risk" (AS 2315.10).
For a delegate audit the practical form of sampling risk is the clean opinion that should not have been clean: a sample that happened to miss the over-limit risks and reported the delegate as satisfactory. How many files it takes to bring that risk down to a level an auditor can defend is worked through in the audit sample size calculator.
What is nonsampling risk? A 2008 case shows it
Nonsampling risk is everything that can go wrong apart from the luck of the draw: the wrong test, the wrong population, or an auditor who looks at a document and fails to see what it hides. The PCAOB puts it bluntly: "An auditor may apply a procedure to all transactions or balances and still fail to detect a material misstatement" (AS 2315.11). And it gives the trap that matters most for bordereaux: "an auditor would not be able to detect understatements of an account due to omitted items by sampling the recorded items" (AS 2315.17).
Markel v Surety Guarantee Consultants shows both risks in one audit. A surety coverholder had issued bonds in Markel's name above the binder's financial limits and reported them on the bordereaux as if they were inside. Before the insurer's audit in December 2005, the court found, the true bonds in the files were replaced with "dummy" bonds written for sums inside the limits. Then the auditor drew 14 files at random and reported no significant issues, in the words quoted below. The bonds over the limits "were not discovered. They were not on the file." (Markel International v Surety Guarantee Consultants, [2008] EWHC 1135 (Comm)).
Two ways a delegate audit can miss a breach
Sampling risk: The sample did not happen to include the bad files.
Nonsampling risk: The test, the population or the evidence was wrong, however many files were read.
Sampling risk: The judge allowed that the 14 random files might simply not have included the bonds over the limits.
Nonsampling risk: The files had been altered and the bordereaux understated the exposure, so the files and the bordereaux agreed with each other.
Sampling risk: More files, stratified draws, and tests on every record before sampling.
Nonsampling risk: Evidence from outside the delegate’s own reporting: bank statements, policy documents issued, third-party confirmations.
Sampling risk: Removes it for anything the delegate reported: every record is tested.
Nonsampling risk: Nothing on its own. A whole-book test of a false bordereau finds a clean, false book.
The lesson is not that sampling failed and something else would have succeeded. The lesson is that a test of what the delegate reports is only as good as the report, so an audit programme needs at least one test that reaches outside it: premium in the bank against premium on the bordereau, documents actually issued against risks declared. That is where the real money in the Markel case was, and why the judge found the coverholder had made a secret profit.
14 sample files were chosen at random. The auditor reported that “there would appear to be no significant issues with the reporting and submission of Risks Bound and Premium Settlement Bordereaux.”
Markel v Surety Guarantee Consultants, High Court, 2008, on the insurer's December 2005 audit
What is random sampling in audit, and why keep the random picks?
Random sampling in audit means every item in the population has a known, equal chance of selection, drawn from a recorded random seed so the draw can be repeated. Its value in a delegate audit is not that it finds the most breaches; on the arithmetic above it often finds none. Its value is independence. A risk-weighted sample finds what the risk model expects; a handful of random picks checks whether the model is blind somewhere, and lets internal audit see that the sample was not chosen to come out clean.
The PCAOB's rule for representativeness is that "all items in the population should have an opportunity to be selected" (AS 2315.24). A purely judgmental sample does not meet it; a judgmental sample with random picks beside it, each drawn from a recorded seed, does. In the fictional Bayline audit, the six random files found no exceptions, which is itself a finding: the rest of the book reads as clean as the whole-book tests said.
When should an auditor test 100 percent?
When accepting any sampling risk is not justified, and when testing everything costs little. The PCAOB expects auditors to examine in full the items for which, in their judgment, sampling risk is not acceptable, such as items whose potential misstatement could on its own equal or exceed the tolerable misstatement (AS 2315.21). In a delegate audit, a risk bound over the binder's limit or a claim paid over settlement authority is that kind of item: one is a finding.
Cost used to decide the question. When bordereaux arrived as spreadsheets in a dozen layouts, testing every row against the binder was days of work. With the bordereaux read into one shape, the field-level tests (state, class, insured value, wind aggregate, screening date, issue date, remittance date, payment against authority) run on every row in minutes. What still needs a sample is the file-level evidence: the referral log, the signed application, the claim note. That is the split the continuous auditing tools guide compares in detail.
Set the human baseline before judging an agent
Before anyone trusts an automated review, the current manual one should be measured the same way. A delegated authority team knows how many files it tests and how many findings it raises; it rarely knows how many breaches its sample missed, because a missed breach is, by definition, not in the workpapers. The regulator noticed the gap a decade ago: the FCA's 2015 review found that "In many cases, there is insufficient oversight of the performance of products and delivery of services" (FCA TR15/7, June 2015, since marked historical by the FCA). A baseline turns that from an impression into numbers.
Five numbers to record about your current sampling before comparing anything
- Population and sample, per audit
Records in the period and files tested, for risks and claims separately.
- How each file was chosen
Random with a seed, judgmental with a reason, or neither. If neither, the sample cannot be defended.
- Exceptions found in the sample
By test, with the files they came from.
- Exceptions found later in the same period
From claims that surfaced, the next audit, a bordereau correction or a whole-book re-test of the old period.
- Hours per audit
Requesting, sampling, testing, writing and chasing, from timesheets rather than memory.
The fairest comparison runs both on a closed period: the old audit's sample and findings beside a whole-book test of the same bordereaux, counting what each found and what each missed. The risk-based audit approach guide sets out how the files are then chosen and who signs.
How Botminds' MGA & Coverholder Audit chooses the sample
The solution samples after it has tested every record, so the sample is built around known exceptions rather than hoping to stumble on them. Every risk and claim is scored on weighted signals: a whole-book exception that needs the file, a value within 10% of a binder limit, a process raised at the last audit, a Tier 1 wind county, size, month-end binding, new staff, a link to another sampled record. Files with exceptions that only the file can settle go in first, then the highest scores, then six random picks from a recorded seed. The sample locks when fieldwork starts; an auditor can add or remove a file only with a reason.
The sampling plan: weights, seed and what a random sample would have found
The weights are the carrier's own settings, and the memo records each file's reasons, so internal audit can check the draw.
The method has limits the auditor should keep in view. A risk score is a model, and it can be wrong in the way all models are wrong: confident about the patterns it was told to look for. The random picks are the guard against that. And nothing in a sample, however chosen, finds a breach that the delegate left off the bordereau entirely; that takes the outside evidence the Markel case was missing.
Questions people ask
What is the most common audit sampling method?
No published survey says how delegated authority teams sample; a judgmental pick of a few dozen files, sometimes with a few random ones, is the pattern this solution was built to improve on. For audits that must project a result to the whole population, the PCAOB's standard names random-based methods: simple random, stratified, proportional to size and systematic selection with a random start.
Is judgmental sampling acceptable in an audit?
Yes, as long as the auditor can explain each choice and does not claim the result represents the whole book. The PCAOB accepts nonstatistical sampling when properly applied, but expects every item in the population to have had a chance of selection when the sample is used to conclude about the whole.
What is the difference between statistical and nonstatistical sampling?
Statistical sampling uses probability theory to size the sample and measure sampling risk; nonstatistical sampling relies on judgment for both. The PCAOB says a properly applied nonstatistical sample should ordinarily be comparable to, or larger than, an efficient statistical one for the same objective.
What is attribute sampling?
Attribute sampling tests whether a characteristic is present or absent in each item, such as "referral approved before binding: yes or no", and estimates the rate of deviations in the population. Most delegated authority tests are attributes, which is why breach rates, not monetary amounts, drive their sample sizes.
What is stratified sampling in an audit?
Stratified sampling splits the population into groups that behave alike and samples each separately: risks and claims, states, or premium bands. The PCAOB notes it can reduce the sample needed for the same assurance; in a delegate audit it also guarantees that claims and every state are represented.
Can sampling miss fraud?
Easily, if the fraud is in what the delegate reports. In the Markel case the bordereaux and the altered files agreed with each other, so testing them more thoroughly would not have revealed the bonds over the limits. Testing outside the delegate's own records, such as bank statements and documents issued, is what reaches that kind of breach.