Audit sample size calculator: how many files find a 2% breach rate?

An audit sample size calculator tells an auditor how many items to test to reach a stated level of confidence about how often something goes wrong. For a delegated authority audit the useful version answers a blunt question: if a given share of a delegate's risks or claims breach the binding authority, how many randomly chosen files must be tested to have a fair chance of seeing even one? The answer comes from one formula, n = ln(1 − confidence) ÷ ln(1 − breach rate). At a 2% breach rate it is 114 files for 90% confidence and 149 for 95%; at 1%, 230 and 299. A sample of thirty files sees at least one breach at a 2% rate only 45 times in a hundred. Botminds' MGA & Coverholder Audit is an agentic solution for MGA and coverholder audits that tests every risk and claim against the binder first, so the files an auditor samples are chosen to confirm known exceptions rather than to find them by chance; the calculator below works for either approach.

The figures on this page are our own calculations, shown so they can be checked. The example book is fictional: Bayline Coastal Underwriters, a commercial property MGA in the solution's sample data, with 2,318 risks and 412 claims in a six-month audit period.

How do you calculate audit sample size?

For finding breaches, the calculation asks how many draws it takes before the chance of missing every breach falls below what you can accept. If a share p of the files breach and each file is drawn at random, the chance that one file is clean is (1 − p), and the chance that n files are all clean is (1 − p) to the power n. Setting that equal to one minus the confidence you want and solving for n gives the formula. Each extra file buys less than the one before, which is why the numbers climb so steeply as the breach rate falls.

Estimating a rate is a different calculation with bigger answers. To say "the breach rate is 2%, give or take one point, at 95% confidence" takes about 753 files; give or take two points, about 189. Most delegate audits do not need an estimate: one confirmed breach of a binding authority is a finding, so the discovery question is the right one.

Audit sample size table: files needed by breach rate and confidence

The table gives the discovery sample for common breach rates and confidence levels, for a large book. Read a row as: "if at least this share of files breach, this many random files give this chance of including one." For books under a few thousand records the exact numbers are a little smaller, as the next section shows.

What is a good audit sample size?

A good sample size is one that matches the rate you need to detect and the confidence you need to state, and the auditing standards give the anchors. The PCAOB suggests that an auditor relying heavily on a sample "might decide that a tolerable rate of 5 percent or possibly less would be reasonable", and one relying on other evidence too might accept "10 percent or more" (PCAOB AS 2315.34). Its worked example fits the table: "if the tolerable rate for a population is 5 percent and no deviations are found in a sample of 60 items, the auditor may conclude that there is an acceptably low sampling risk that the true deviation rate in the population exceeds the tolerable rate of 5 percent" (AS 2315.41). On our arithmetic, a population with a 5% deviation rate produces a clean sample of 60 only about 5 times in 100.

The difficulty for delegate audits is that the breaches that matter are much rarer than 5%. In the Bayline book, three of 2,318 risks were bound over the per-location limit, about 0.13%. No affordable random sample is sized to find that; a test that reads every record finds it in minutes.

Does the size of the book change the sample size?

Hardly, once the book is more than a few hundred records. The PCAOB's own table of factors that move sample size lists the number of items in the population as having "Virtually no effect on sample size unless population is very small" (AS 2315, appendix, table 1). For a 2% breach rate at 95% confidence, our exact calculation gives 105 files for a book of 200 records, 138 for 1,000, 145 for 2,318 and 149 for 100,000. A delegate ten times the size does not need a sample ten times larger; a small one does not need a much smaller one.

That is counter-intuitive for many teams, who scale samples with premium. The arithmetic says to scale them with the rarity of the breach and the confidence wanted, and to spend extra effort on what sampling cannot reach.

What sample size detects a 2% error rate?

About 114 random files for 90% confidence and 149 for 95%. Smaller samples are a coin toss: at a 2% rate, 30 files include a breach 45 times in 100, and 60 files 70 times in 100. The bars show the chance of including at least one breach for the sample sizes delegate audits actually use.

Fourteen random files from a book where one bond in fifty breaches: more often than not, the sample holds none of them.

The Markel case is the cautionary tale. The insurer's audit of a surety coverholder chose 14 files at random and reported no significant issues; bonds written over the limits went undiscovered. The court found the files had been altered, and allowed a second explanation: the sample may simply not have contained them ([2008] EWHC 1135 (Comm), footnote 5; the judge's words are quoted in MGA exceeded authority limit). At breach rates of 2% or less, 14 random files miss every breach more often than not.

Why the question changes when every record is tested first

When every record on the bordereaux is tested against the binder, the sample no longer has to find the breaches the data can show; it has to confirm them and look for what the data cannot. In the fictional Bayline audit, whole-book tests listed 60 records with exceptions before any file was chosen. The sample of 36 took the 14 exceptions that needed a file to settle, 16 files with the highest risk scores, and 6 random files, and found exceptions in 12.

The six random files deserve an honest note. Six files give little statistical assurance on their own: even at a 5% breach rate in the rest of the book, six random files would all be clean about 74 times in a hundred. Their job is different. They test whether the risk model is blind somewhere and keep every file's chance of selection above zero, which internal audit can check from the seed. A carrier that wants a statistical statement about the untested remainder can raise the random count in its sampling plan and use the table above to size it.

A sample sized for confirmation, not for luck

Sizing a delegate audit sample, step by step

Size the sample after deciding what the sample is for, because a delegate audit asks several different questions of its files. Five steps keep the numbers honest and the memo short enough for internal audit to read.

1. Split the populations. Risks and claims are different books with different tests; size them separately, and stratify further by state or class where the binder's limits differ. 2. Move every field test to the whole book. Territory, class, insured value, aggregates, screening date, issue date, remittance date and payment against authority can be tested on every record, so no sample is needed to find those breaches; the sample's job for them is to open the file and see whether there is an approval, a referral or an explanation. 3. Size the file-only tests by the rate that matters. For evidence that lives only in documents, such as a signed application or a documented rating deviation, choose the lowest breach rate you need to detect and the confidence you will state, and read the number from the table. 4. Size the random component deliberately. If the random picks are there to guard the risk model, a handful will do; if they are there to support a statement about the untested remainder, size them with the calculator. 5. Write it down before fieldwork. The rate, the confidence, the seed and the reasons for any judgmental picks belong in the memo, and the sample should lock when fieldwork starts.

Done this way, the files tested in a delegate audit are rarely more than a few dozen, because the expensive question, whether a breach exists at all, has already been answered for every record by the data.

What a sample size cannot tell you

A sample size controls one risk only: the chance of missing a breach that is in the population being sampled. It says nothing about breaches that never reached the population. If a delegate leaves a risk off its bordereau, or files a document that agrees with a false bordereau, no number of files drawn from that bordereau will show it; the PCAOB's standard makes the general point that a sample of recorded items cannot reveal omitted ones. That limit applies equally to testing every record.

The remedy is a different population, not a bigger sample: premium received in the bank against premium declared, policies issued against risks reported, claim payments made against claims listed. How sampling and nonsampling risk divide, with the case that shows both, is set out in the audit sampling methods guide.

Audit sample size calculator in Excel

One cell does it. With the confidence in B1 and the breach rate in B2, both as decimals, =ROUNDUP(LN(1-B1)/LN(1-B2),0) returns the discovery sample. A second cell, =1-(1-B2)^B3 with your planned sample in B3, returns the chance that sample includes at least one breach. For a book under a thousand records, the calculator on this page gives the exact, slightly smaller figure.

Regulators use their own tables for their own tests, and those should not be mixed with a delegate audit's. Market conduct examiners size claim samples from the NAIC's acceptance tables, worked through in the market conduct exam sample size calculator; a claims compliance team testing its own files against fair claims rules has its own version in the claims audit sample size calculator. How files are chosen once the number is set is in the risk-based audit approach.

Questions people ask

Is there an audit sample size calculator I can use?

The calculator on this page sizes a discovery sample for any book size, breach rate and confidence, and shows your planned sample's chance of seeing a breach. It runs in your browser; nothing you type is sent anywhere.

What is the minimum sample size for an audit?

No rule sets one for delegate audits. The arithmetic does: below about 45 files, a random sample cannot give 90% confidence of seeing a breach even at a 5% rate. Smaller samples can still be useful for confirming exceptions already found by testing every record.

How many files should be sampled in an audit?

Enough for the rate you need to detect: 59 for a 5% breach rate at 95% confidence, 149 for 2%, 299 for 1%. If those numbers are unaffordable, the answer is to test the fields on every record and sample the files only for what the data cannot show.

What if the sample finds no breaches?

Then the breach rate is probably below the rate the sample was sized for, at the confidence used. A clean sample of 149 supports a rate below 2% at 95% confidence; it says nothing about a breach rate of 0.1%, which is where three over-limit risks in 2,318 sit.

Should risks and claims be sampled separately?

Yes. They are different populations with different tests and different breach rates; mixing them lets the larger population crowd out the smaller. In the sample setup, risks and claims are drawn two to one, with at least one file per state.

Does a bigger delegate need a bigger sample?

Barely. Above a few hundred records, the sample needed for a given breach rate and confidence hardly changes with the size of the book; the PCAOB lists population size as having virtually no effect unless the population is very small.