AI bias and fairness testing for every pricing, underwriting and claims model, with the written explanation a regulator expects
Fairness tests and the written explanation a regulator expects, together.
See one case, screen by screen ↓Testing a pricing, underwriting or claims model for unfair discrimination: measuring how its outcomes differ by protected class, finding which inputs drive the difference, searching for a less discriminatory alternative and writing down why each input relates to risk. AI Bias & Fairness Testing does all four on every model version and drafts the test record for the model risk validator to sign.
How to choose AI solutions for insurance →
Updated
New York, Colorado and the NAIC bulletin now expect pricing, underwriting and claims models to be tested for unfair discrimination before they go live, after every material change and on a regular cadence after that — with several quantitative metrics, a search for less discriminatory alternatives and a written explanation of why each input relates to risk. Carriers do not collect race or ethnicity, so the groups have to be estimated first, from surname, first name and census block group, and used as probabilities.
Then the hard part starts. A failing adverse impact ratio says nothing about which input caused it, whether that input relates to risk, or whether a fairer version keeps the model’s accuracy. That analysis is done by hand, one refit at a time, in notebooks and spreadsheets. It takes most of a quarter, the results often arrive after the model is filed, and by the time it is done the data scientists have changed the model again.
Estimated split of the 8–12 weeks a test takes by notebook and spreadsheet; the one-day total with the solution is the target.
Six specialist agents estimate groups, run the tests, find the proxies, refit fairer alternatives and draft the record; the validator signs.
Kestrel Point Mutual’s Personal Auto Rating Model v7.2 adds a telematics driving score and rebases the credit-based insurance score tiers. It is due in the New York rate filing on Tue 1 Dec. Here is its test, screen by screen, in the working solution.
Eight models are in scope — pricing, underwriting and claims — each with its last test, its result and when it is next due. v7.2 is on the bench because of a material change: a new telematics score and rebased credit tiers. The fraud referral model has a finding open, and the homeowners pricing model is due its annual retest in 18 days.
“100% of 2026 changes tested before production.”
184,260 New York quotes from April to June 2026, reference group White. Black-probable drivers reach the preferred tier at 30.1%, White drivers at 41.8% — an outcome gap of 11.7 points against a threshold of 8.4. Four of the six tests fail: adverse impact ratio 0.72, odds ratio 1.72, standardized difference 0.24 and marginal effect 4.6% for Black-probable drivers.
“3.3 points over the threshold — Black-probable drivers reach the preferred tier at 30.1%, White drivers at 41.8%.”
The Model reader reads the 148-page model documentation and finds two inputs changed since v7.1. The Group estimator places 97.6% of quotes in a census block group. The Metric runner runs six tests across three group pairs and female vs male. The Proxy detector makes 11 refits, the Alternative searcher refits five alternatives, and the Explanation writer drafts the record.
The last step is never an agent: “Grace Liang · model risk validator — reads the record and signs.”
The gap is decomposed by input: ZIP code territory carries 6.0 points, credit tier 2.8, vehicle age 1.1, prior insurance lapse 0.7. The telematics score narrows the gap. Five territories carry the territory share, and in T07 the 5-digit factor charges 31% more than the state where losses run 8% higher.
Each of the 11 rating inputs is removed in turn and the portfolio re-scored. ZIP code territory predicts Black-probable membership with an AUC of 0.81 on its own; removing it closes 51% of the gap but costs 3.6% of the lift. Telematics, at 0.51, is not a proxy — removing it widens the gap slightly.
“Removing it closes 51% of the gap but calibration for Black-probable drivers falls to 0.93, so the input should be replaced, not removed.”
On the bench, ZIP code territory is switched off and all 184,260 quotes re-scored. The gap falls to 5.7 points, under the 8.4 line, and five of six tests pass. But calibration for Black-probable drivers drops to 0.93 and the model keeps only 96.4% of its lift — below the 98% the testing standard requires.
Five candidates are refit on the same quotes and put through the same six tests. Removing ZIP and credit together closes the most but keeps only 91.5% of the lift. The density-banded territory — county by five population-density bands — passes six of six with a 7.3-point gap and 99.3% of the lift. Mean premium for Black-probable drivers falls $71; 9,412 quotes change tier; the overall rate level is unchanged.
Kept only if it holds at least 98% of the model’s lift. The agent proposes; it never adopts.
Nadia adopts Personal Auto Rating Model v7.2-DT for the New York filing on Tue 1 Dec 2026, effective 1 Mar 2027. The test record is redrafted on the spot — sections 6 and 7 — and goes to Grace to sign.
Fourteen pages and 63 citations: why the test, the data and group estimates, the results as filed and density-banded, what drives the gap, the alternatives, how each input relates to risk and the decision. Every number points to its run, data cut and code version in the margin — the circular letter, the model documentation, the loss study, the testing standard.
“Territory: claim frequency rises with traffic density and theft; loss cost by density band is monotone, 0.86 to 1.18 — replace with density bands.”
Grace confirms she has read every section and its sources, that the tests ran on the version being filed, that the alternatives search is documented and that group estimates stayed in the test environment — then signs. The record is sealed with its data cut, code version and run log, and one more click sends it to Marcus Odell in compliance for the rate filing packet. Helen Ashworth’s annual attestation picks it up on 1 Dec.
The same test, seen by the five people who carry it — the validator, the model owner, the data scientist, the CRO and compliance.
Six specialist agents estimate groups, run the tests, find the proxies, refit fairer alternatives and draft the record; the validator signs.
Reads the model documentation, variable list and score file; lists the inputs and what changed since the last tested version.
Estimates group probabilities for each quote with BIFSG from surname, first name and the census block group of the garaging address, and reports coverage.
Runs the six tests for every group pair against the thresholds in the testing standard — adverse impact ratio, odds ratio, standardized mean difference, marginal effect holding expected loss and calibration — probability-weighted, with significance.
Removes each input in turn, re-scores the portfolio and measures the gap closed, the lift lost and how well the input predicts group membership; decomposes the gap by input.
Refits less discriminatory alternatives — rebanding, coarsening, capping, replacement — and tests each, reporting lift kept and premium effect.
Drafts the test record and the plain explanation of each input’s relation to risk, citing loss studies and results.
Signs; the model owner adopts; the CRO attests. The agents propose; a named person decides.
Ask in plain words — or try a what-if on the bench. Every answer cites the run, loss study or standard behind it.
Why does ZIP code fail but density bands pass?
At the 5-digit level the territory factor charges up to 1.31× where losses run 1.08× (T07), and ZIP alone predicts Black-probable membership with AUC 0.81. Density bands keep the part of territory that is risk — traffic density and theft — so the gap falls to 7.3 points (threshold 8.4) with 99.3% of the lift. Removing ZIP outright drops calibration for Black-probable drivers to 0.93, so it fails test 5.
What if we also remove the credit tier?
With ZIP territory and credit tier both off, the gap falls to 3.6 points and five of six tests pass, but calibration for Black-probable drivers is 0.92 and the model keeps only 91.5% of its lift — under the 98% the standard requires. Credit tier relates to risk: loss ratio runs from 0.58 in tier A to 0.91 in tier F.
Which rules is this test record written for?
New York: the metrics named in Circular Letter 2024-7 — adverse impact ratio, odds ratio, marginal effect, standardized mean difference and drivers of disparity — run before production, with the less discriminatory alternative search and the written relationship to risk. Colorado: the risk management framework and the chief risk officer’s attestation on 1 Dec. And the NAIC AI bulletin’s testing expectation.
Show me the quotes behind the gap in Territory 07
Territory 07: 41 ZIP codes, 18,420 quotes, 46% Black-probable. Preferred-tier rate 22.4%; v7.2 charges a territory factor of 1.31 where losses run 1.08. It carries 2.3 of the 11.7 points. Click any quote for its rating as filed and with density bands.
13 screens from the working solution, on its sample data. Pick one to see it large.
Thresholds, groups, retest triggers and sign-off are settings — a versioned standard your chief actuary approves.
| Setting | Default | Choose from |
|---|---|---|
| Adverse impact ratio fails below | 0.80 · four-fifths rule | Step up or down; proposed as a new standard |
| Odds ratio, surcharge tier, fails above | 1.25 | Step up or down; proposed as a new standard |
| Standardized mean difference fails above | 0.20 | Step up or down; proposed as a new standard |
| Marginal effect, % of premium, fails above | 3.0 | Step up or down; proposed as a new standard |
| Calibration, actual ÷ expected loss by group | 0.95–1.05 | Set in the standard |
| Lift an alternative must keep | 98% of the Gini as filed | Set in the standard |
| Groups tested against the reference | Black, Hispanic, Asian vs White; female vs male | Each on or off · age 65 and over for Colorado life |
| When a model is retested | Before production · material change · every 12 months · new external data source | Each on or off · outcome drift of 1.0 point in a month |
Every figure is labelled: a target the solution is built to, an estimate, a typical published result, or a proven one.
“demo” = seen in the working solution, on its sample data · “target” = the design goal, measured in the live solution · “estimated” = our estimate · “typical” = published requirement (NY DFS Insurance Circular Letter No. 7, 2024, names the quantitative metrics). Other sources: Colorado SB21-169 · NAIC Model Bulletin on the Use of AI Systems by Insurers (2023) · the four-fifths rule, EEOC Uniform Guidelines, 29 CFR 1607.4(D). People, companies and models named on this page are fictional — characters and sample data in the working solution.
Testing a pricing, underwriting or claims model for unfair discrimination: measuring how its outcomes differ by protected class, finding which inputs drive the difference, searching for a less discriminatory alternative and writing down why each input relates to risk. AI Bias & Fairness Testing does all four on every model version and drafts the test record for the model risk validator to sign.
Each quote carries a probability for each group, estimated with BIFSG from surname, first name and the census block group of the garaging address, and every result is probability-weighted. No quote is assigned to one group. The estimates stay in the test environment, are never joined to a quote, policy, claim or rate, and are deleted after the record is signed.
Six, against the thresholds in your testing standard: the adverse impact ratio on the preferred tier, the odds ratio on the surcharge tier, the standardized mean difference in indicated premium, the marginal effect of group on premium holding expected loss, calibration of actual to expected loss by group, and the adverse impact ratio for female vs male. It also decomposes the gap into drivers of disparity.
The proxy detector removes each rating input in turn, re-scores the portfolio and measures how much of the gap disappears, how much lift is lost and how well the input alone predicts group membership. In the working solution, ZIP code territory scored an AUC of 0.81 and carried 51% of the gap; the telematics score, at 0.51, was not a proxy.
No. The agents never change a filed model or move a threshold. The Alternative searcher proposes fairer refits that keep the lift floor and the overall rate level; the model owner decides whether to adopt one, and the model risk validator signs the record.
Yes. Thresholds, the lift an alternative must keep, the groups tested, the retest triggers and who signs are settings. They form a versioned testing standard, and a change is proposed as the next version for the chief actuary to approve.
Yes. Models in scope covers pricing, underwriting and claims — in the working solution, homeowners underwriting, term life accelerated underwriting, auto claims fraud referral and bodily injury severity triage sit beside the auto rating model, each with its own retest schedule.
The Agentic Solution Engine builds and deploys it from your requirements — your testing standard, model documentation and a scored test portfolio — and it goes live once every quality gate has passed. We will walk you through it on one of your own models first.
Each new rule mapped to its policy and owner the day it is published.
1-min demo50-State Regulatory ResearchEvery state answered side by side, each with the statute that says so.
1-min demoMarket Conduct Exam ManagementEvery examiner request tracked to its due date, answered from the file.
1-min demoWe’ll run AI Bias & Fairness Testing on one of your own rating or claims models.