A premium leakage test measures how well a team or a tool finds wrong rating facts by planting known errors in closed policies and counting what comes back. You take a sample of new business that has already been worked, change some of the facts on purpose, such as a driver removed from the application, a garaging address moved or a mileage figure lowered, and run the check without telling it which files you changed. The planted errors it finds give you its recall; the clean facts it flags give you its false alarm rate; both, by type of error, are the numbers that matter. Auto Premium Leakage Detection is Botminds' software that checks every new auto policy's drivers, garaging, use, mileage, prior insurance and vehicle in its first days and re-rates any difference under the carrier's filed plan; this page explains how to test it, or any alternative, before trusting it.
The test matters because the ordinary ways of checking underwriting quality cannot see leakage. They test whether the premium matches the file. A leakage error lives in a file that matches itself perfectly. Figures about the solution come from its fictional Georgia sample book.
How a regulator samples auto underwriting files
119
auto policies sampled from a population of 350,022 in one state examination
5%
error tolerance for most areas reviewed, at a 95% confidence level
0%
tolerance for rates or forms neither filed with nor approved by the department
0
premium errors found: “All premiums charged were deemed correct.”
What a premium leakage test measures, and what an exam does not
A premium leakage test measures whether a check finds the rating facts an application got wrong. An underwriting examination measures something else: whether the carrier followed its own guidelines and filed rates on the facts in its file. North Carolina's examiners, for example, describe reviewing each sampled policy "for adherence to underwriting guidelines, file documentation, and premium determination" (NC DOI, 2015). In the examination summarised above, a random selection of 119 auto policies from 350,022 found that "all premiums charged were deemed correct."
Both can be true at once: every premium correct on its file, and some files wrong about the household. That is not a criticism of the examiners. Their test answers a compliance question. A carrier asking how much premium it is missing needs a different instrument.
Why a regulator's file review cannot see leakage
A file review compares the premium with the facts on file, and leakage is a fact that never reached the file. If the application says two drivers and the rating uses two drivers, the premium determination is correct. The third driver who lives at the address appears only in records the file review does not consult. The review's sample is also small by design. North Carolina's method states that "sample sizes were generated via an Audit Command Language (ACL) program with a random sample taken from a given population" and that the Department "utilized a 95% Confidence Level to determine the error tolerance level" (NC DOI); the tolerance it applied was "0 percent ... for the use of forms and rates/rules that were neither filed with nor approved by the Department and 5 percent for all other areas reviewed."
The same department's 2017 examination of another carrier shows what file reviews do catch: premiums on consent-to-rate forms that did not match what was charged, traced to "changes made by the producing agents after submitting the new business applications to the Companies" (NC DOI, 2017). Those are internal inconsistencies, visible on paper. Leakage is an external inconsistency, between the paper and the household, and it needs outside records and a different kind of test.
Plant known errors: building the seeded set
Build the test from closed new business your team has already worked, with the reports it already obtained, and plant errors of each kind you care about. Closed files matter because the truth is known and settled. Reusing the reports already obtained matters for a second reason: the FCRA lets a company obtain a consumer report when it "intends to use the information in connection with the underwriting of insurance involving the consumer" (FCRA §604(a)(3)(C)), and ordering fresh reports on closed policies purely to test a tool is a different purpose. Check that with counsel; where in doubt, plant errors in fictional households built on your real rating plan.
Plant each kind of error in the way it actually happens, not in the way that is easiest to detect. A hidden driver should be a teenager whose licence predates the effective date, sometimes listed on the prior policy, sometimes not. A garaging error should sometimes move one county and sometimes another state, sometimes with matching registration. A mileage error should be a believable estimate against readings far enough apart to annualise. And leave most files clean, because false alarms are measured on the clean facts.
A 200-error seeded set, proportioned to what the sample book found
| Planted error | How many | What you change | What a good check should use |
|---|---|---|---|
| Hidden driver | 53 | Remove a licensed household member from the application | Household report, driving record, prior policy’s drivers |
| Garaging address | 38 | Replace the address with a cheaper one, in-county and out-of-state | Registration, licence, billing, prior policy |
| Annual mileage | 34 | Lower the declared estimate by one or two bands | Two dated odometer readings at least 90 days apart |
| Business or for-hire use | 25 | Change business use to commute | Registered owner and class |
| Prior insurance | 25 | Hide a lapse or overstate prior limits | Prior-carrier report |
| Vehicle | 25 | Record a base trim for a higher-rated one | VIN decode |
The four outcomes, with personal auto examples
Every fact in the test ends in one of four cells, and each cell has a cost. A missed error costs premium and, if a claim comes first, a dispute. A false alarm costs an agency's time, a household's goodwill and, if acted on, a wrong charge. The test exists to put numbers in all four.
Four outcomes for every fact in a premium leakage test
Check flagged it
Check stayed silent
Planted error
Clean fact
True positive
Found
A planted error the check flagged
A 17-year-old added to the household data and missing from the application: flagged as a hidden driver, priced, request drafted.
False negative
Missed
A planted error the check did not flag
A garaging address moved one county over, to an address the registration and licence also carried: no disagreement, no flag.
False positive
False alarm
A clean fact the check flagged
An adult son who moved out three years ago, still linked to the address in the household report: flagged, and closed by the agency’s reply.
True negative
Correct silence
A clean fact left alone
Four records agree on the garaging address, so no plate sightings are ordered and nothing is asked.
The false alarm cell deserves its own design. On the solution's sample book, 29 of 263 people flagged as hidden drivers were closed as not household drivers. A test that plants only errors, and never measures how often clean households are flagged, would score a check that flags everyone as perfect.
How many planted errors make a fair test?
Enough to measure each kind of error within a margin you can act on. With 25 planted errors of a type and an observed recall of 90%, the 95% margin of error is about 12 percentage points either way; with 100 it is about 6. Those are normal-approximation figures, our arithmetic; for small counts, or recall near 100%, use an exact binomial interval, which is wider.
The margin of error on a 90% recall, by errors planted of one type
That arithmetic sets the shape of a fair test. A single headline recall across 200 planted errors looks precise and hides the type the check is weak on. Twenty-five per type is the least that tells a strong check from a weak one; 100 per type for the kinds that carry most of the premium is better. Examiners make the same trade-off in reverse, sampling small and accepting a 5% tolerance because their question is narrower.
Four numbers to report from an auto premium leakage test
Report four numbers by type of error, and never one number for the whole test. A recall without a false alarm rate is meaningless, and a count without premium hides whether the check finds the errors that matter.
What to report, by type of planted error
| Measure | How it is calculated | Why it matters |
|---|---|---|
| Recall | Planted errors flagged ÷ planted errors | The share of real leakage the check would find |
| False alarm rate | Clean facts flagged ÷ clean facts checked | The load on agencies and the risk to households |
| Premium recall | Premium on planted errors found ÷ premium on all planted errors | Whether it finds the expensive errors or only the easy ones |
| Days to finding | Median days from the effective date to the flag, replayed | Whether it finds them while the first 60 days are still open |
The last measure ties accuracy to money. A check that finds 95% of hidden drivers on day 45 is worth less than one that finds 90% on day 2, because the early finding is still inside the window where the carrier's options are widest; what premium leakage is works through that arithmetic for one policy.
How to reduce premium leakage rate without raising false alarms
Reduce leakage by checking every policy rather than a sample, early, with sources that contradict each other before anyone is asked, and keep false alarms down by requiring agreement between sources and asking before charging. The test above is how you prove a change helped.
Levers that lower leakage, and what each does to false alarms
- Check every new policy, not a sample
The largest single gain: on the sample book’s modelled split, 32 of the 41 days to a sampled finding were waiting to be picked.
- Require two sources before a hidden driver badge
A household report plus a licence, a prior policy or a registration date should cut false alarms; the seeded test shows what it costs in recall.
- Annualise mileage only from two readings 90 days apart
One reading, or two close together, produces confident wrong answers.
- Order intrusive evidence only on disagreement
Plate sightings when addresses disagree, not on every policy.
- Set an ask threshold
Below it, note the difference for renewal. On the sample book, $75 a term kept 503 small differences away from agencies.
- Ask before charging
A false alarm that the agency closes costs a reply. A false alarm that is re-rated costs a complaint.
One thing a leakage test does not measure is fairness. Whether a check's findings fall unevenly on protected groups is a separate question with its own tests. In Colorado, auto insurers' use of external consumer data and algorithms has been subject to the governance and risk management requirements of amended Regulation 10-1-1 since 15 October 2025 (Colorado Division of Insurance). Run both kinds of test.
Testing Auto Premium Leakage Detection on your own closed files
The solution is built to be tested this way. Each of its six checks can be switched on or off and replayed on past weeks, every finding cites the records behind it, and the premium re-rater prices each difference on its own and together under the plan in force on the effective date, so a planted error's premium is computed the same way a real one would be. Nothing it does changes a policy or reaches a household, so a test touches no one.
Six checks, each with its sources, owner and a replay on last week
On its own fictional sample book, the solution found differences on 814 of 6,412 new Georgia policies with a median of 2.6 days from bind. Those are sample results, not a test result, and the right response to them is the test on this page. A pilot built around it, with the pass marks set before it starts, is laid out in a 30-day pilot on your own book; the mileage check's own logic, which a seeded test should probe hardest, is in declared vs driven miles. The solution is at Auto Premium Leakage Detection.
Questions people ask
What does premium leakage mean in insurance?
Premium the insurer's own filed rates say a policy should carry but does not, because a rating fact was wrong or missing when it was written. A premium leakage test measures how much of it a check would find.
Is there a premium audit for personal auto insurance?
Not in the way commercial lines rated on payroll or sales are audited at the end of the term; the CAS ratemaking text discusses premium audits for the lines "subject to premium audits". In personal auto, checking the declared facts after bind does the job an audit would.
What recall should a premium leakage check achieve?
There is no regulatory standard; the carrier sets the bar before the test starts, by type of error. Set it with the false alarm rate beside it, because a check can reach any recall by flagging everything.
Can we use real policyholders' consumer reports in a test?
Use the reports already obtained when the policies were underwritten, and take advice before ordering new ones. The FCRA permits ordering a consumer report for the underwriting of insurance involving that consumer, which a test of a tool may not be.
How long does a premium leakage test take?
Building the seeded set takes most of the effort: choosing closed files, planting errors realistically and pricing each under the filed plan. Running the check on a few hundred files is quick for software; scoring by type and reviewing the false alarms takes the rest.
What is an auto premium leakage test, in one sentence?
A controlled test in which known errors in drivers, garaging, use, mileage, prior insurance and vehicle are planted in closed auto policies, and a check is scored on what it finds, what it misses and what it falsely flags.