A model bias testing tool is software that runs fairness tests on a pricing, underwriting or claims model: it estimates protected-class membership, measures outcome gaps by group, finds which inputs drive them and tests fairer versions. The right way to compare it with an in-house notebook or a consulting engagement is cost per model per year, not cost per test, because the rules make testing recurring. New York expects testing before an AI system goes into production, on a regular cadence after that, and whenever the model or its data change materially (NY DFS, 2024). Botminds' AI Bias & Fairness Testing is built for that recurrence: it runs the whole test again on every model version, so the third test of the year takes the same agent run as the first.
Why the bill for fairness testing recurs
3triggers
for testing in New York: before production, on a regular cadence, and after every material change
Circular Letter No. 7, 2024
8–12weeks
of actuarial and data-science time per model test by notebook and spreadsheet
Botminds estimate
1day
for a complete test with its written explanation
Design target of AI Bias & Fairness Testing
What does a model bias testing tool need to do, and how often?
It needs to run a full test whenever a rule says a test is due, and for a model that changes often that is several times a year. New York sets three triggers and asks for a written description of output testing "conducted at least annually." When no less discriminatory alternative exists, the search for one is repeated every year (NY DFS, 2024). The American Academy of Actuaries recommends scanning algorithms, data and results "on a regular, scheduled basis—at least annually" (Academy, 2024).
Colorado adds a reporting rhythm. Life insurers that use external data file an annual report every 1 December; private passenger auto and health benefit plan insurers file from 1 July 2026; each report must be signed by an officer attesting to compliance and must address the governance components, including a documented description of quantitative testing once the Division adopts its testing rule (Colorado Regulation 10-1-1). A tool that cannot repeat a test cheaply turns each of those dates into a project.
Examiners are starting to ask for the date. The NAIC's draft AI Systems Evaluation Tool, piloted by 12 states from March to September 2026 and due for consideration at the NAIC's November 2026 meeting, asks insurers about each high-risk AI system to "discuss testing model outputs (e.g. model drift, accuracy, unfair trade practices, unfair discrimination, performance degradation, etc.)" and to give the "Last date of model testing" (NAIC, Exhibit C, draft). An answer measured in quarters, for a model retrained in the meantime, is a finding waiting to happen.
A full test is more than a metric. It is a group estimate, the metrics for every group pair, the inputs that drive any gap, a search for fairer alternatives, and the written reasoning. Any price comparison that covers only the metric leaves out most of the work.
Why the unit is the model-year, not the test
Because the number of tests per year is set by your model inventory and your change rate, not by the vendor. A model that is retrained twice a year and tested annually needs at least three tests; one that adds a new external data source needs another. The cost of each approach is a sum of terms, and only some of them scale with the number of tests.
What one model-year of fairness testing costs, term by term
Tests in the year
Before production + material changes + annual retest + data-source changes
Count from your own change log; a model retrained twice and retested once a year is three or four tests.
Model risk
Effort per test
Days per test × loaded day rate
Group estimation, metrics, drivers, alternatives, write-up and sign-off; use your own actuarial and data-science rates.
Pricing and data science
Independent review
Annual method review by someone who did not build the test
Internal audit, a second-line validator or an outside reviewer, once a year rather than per test.
Second line
Late finding
Probability a test lands after filing × cost of refiling or remediation
A test that takes a quarter often reports after the model is filed; the remedy is then a new filing.
Compliance
A worked illustration shows the scale. Take an insurer with eight models in scope across pricing, underwriting and claims, each tested once a year plus once after a material change: sixteen tests. At Botminds' estimate of 8–12 weeks of actuarial and data-science time per test by notebook, sixteen tests are roughly 640 to 960 working days a year, several full-time people doing nothing else. That is our arithmetic on our own estimate; replace both numbers with yours.
Where a test's days go by notebook and spreadsheet
Most of a test's time goes on finding why a metric fails and on refitting alternatives, not on computing the metric. The split below is Botminds' estimate of a typical 55-working-day test of one rating model done by hand.
Most of a notebook test is driver hunting and refitting
One model test by notebook, 55 working days
- Estimating groups · 15%
- Running metrics · 15%
- Finding drivers · 22%
- Refitting alternatives · 25%
- Writing it up · 18%
- Sign-off · 5%
The split explains why free metric libraries change the economics less than they seem to. They compress the second slice, which is about a seventh of the work. Finding drivers and refitting alternatives is almost half; each refit is a model run, and New York's Step 3 asks for a documented search, not a single try.
Notebook, consultant or tool: how each scales with the number of tests
Each approach has a place. A notebook is flexible and cheap to start; a consultant brings independence and a method; a tool makes the hundredth test cost what the first did. The table compares them on the terms that decide cost per model-year.
Three ways to run fairness testing, compared per model-year
| In-house notebook | Consulting engagement | Testing tool | |
|---|---|---|---|
| Cost of the first test | Low: analyst time | High: a scoped engagement | Set-up plus subscription |
| Cost of each later test | Close to the first; partly re-built each time | Close to the first; a new engagement or change order | A re-run on the new version |
| Speed after a model change | Weeks, if the analyst is free | Weeks to a quarter, booking included | Within the testing cycle |
| Independence | None by default | High | Method fixed and versioned; still needs a human challenger |
| Group estimates and drivers | As built | Usually included | Included in every run |
| Who holds the method | One analyst | The consultant | Your versioned standard |
| What the examiner sees | What was saved | A report per engagement | A signed report per model version |
Eight models, each with its next test due
Is free AI bias testing software enough?
For the arithmetic, often yes; for the test, no. Open-source toolkits compute standard bias metrics well, and their authors are frank about what they do not settle: in the Aequitas paper's words, "there is no consensus on which metric/definition should be used" (Saleiro et al., 2018). The Academy of Actuaries says prebuilt tools "can be helpful but may not catch all instances of discrimination and should not replace the experience and judgment" of data scientists and actuaries (Academy, 2024).
Use free software where it fits: to compute metrics in a notebook, to cross-check a commercial tool, or to teach a team the measures. Budget for the rest of the test separately. A free library does not estimate race with a documented method, decompose a gap into inputs, refit a rating plan, hold thresholds under change control or write the explanation an examiner reads.
When a consultant is the right purchase
Buy outside help for independence and for method, not for repetition. NIST's AI Risk Management Framework expects regular assessments to involve people who did not develop the system, inside the firm or outside it (NIST AI RMF, MEASURE 1.3). New York wants independent review and challenge of model validation and testing, and asks that internal audit assess "potential biases in the ECDIS or other data" (NY DFS, 2024). The Academy notes that "third-party monitoring and scanning" may catch issues an internal review missed (Academy, 2024).
Those are annual needs: a review of the method, the thresholds and a sample of tests. Paying for an engagement every time a model is retrained buys repetition at consulting rates. The more economical pattern is a tool that runs every test the same way, with an outside reviewer who checks the method once a year and can re-run any test.
The cost nobody budgets: a finding that arrives after the filing
The most expensive outcome is a failed test discovered after a model has been filed or deployed. By hand, a test takes most of a quarter, and the model often changes again before the result arrives. When a finding lands late, the remedy is a new version, a new test and often a new filing.
By notebook, the result arrives a quarter after the model is ready
- Groups and metrics16 d
- Drivers12 d
- Alternatives14 d
- Write-up and sign-off13 d
- Agents run, validator signs1 d
The legal cost depends on the state. Colorado's statute requires its rules to give insurers time to remedy an unfairly discriminatory impact (C.R.S. § 10-3-1104.9(3)(c)(I)). Under the NAIC's Unfair Trade Practices Act, a state may impose penalties of up to $1,000 per violation and $100,000 in aggregate, rising to $25,000 per violation and $250,000 in aggregate for flagrant violations (NAIC Model 880, § 8). In New York, where no less discriminatory alternative exists, the insurer keeps the model under ongoing risk management and repeats the search every year; where one exists, it changes the model and tests again.
Fairness testing tools for AI: what to look for when you price them
Ask every option the same questions, framed per model-year. A quote that prices a single test, or excludes the driver and alternative steps, is not comparable to one that prices the whole cycle.
Questions to price a model bias testing tool against a notebook or a consultant
- How many tests a year does the price cover?
Per model, including re-runs after material changes and annual retests.
- Is group estimation included and documented?
Method, data vintage, coverage report, and probabilities rather than labels.
- Are drivers and alternatives part of every run?
Decomposition and refits on the same data, not an extra engagement.
- Whose thresholds, and under what change control?
Your standard, versioned, approved by a named person.
- What does a re-run cost after a model change?
In money and in elapsed days; this is the term that recurs.
- Who can reproduce a number?
Each figure tied to its run, data cut and code version, so a reviewer can check it.
- Where do the race estimates live?
In a test environment, logged, deleted on schedule, never joined to a decision.
- What happens to the method if you leave?
The testing standard, reports and method notes should stay yours.
The insurer's own testing standard, as settings
Where AI Bias & Fairness Testing fits, and where it does not
It fits an insurer with several models in scope that change more than once a year, where each change today waits weeks for a test. It runs the whole cycle on every version: estimate, six tests in the insurer's standard, drivers, alternatives and a drafted report for the validator to sign. Six agents do the work; they never change a filed model or move a threshold.
It fits less well where there is one stable model tested once a year by an actuary who already has a working method; the saving there is small. It does not replace the actuary's judgment on whether an input relates to risk, or the validator's challenge. It needs a scored test portfolio and the model's documentation to start. And it cannot test groups that names and addresses do not reveal well. Piloting it on a model you filed last year is the way to measure the saving on your own inventory before committing. The agents and the signed report are on the AI Bias & Fairness Testing page; what the agents do and what people decide sets out the division of labour.
Questions people ask
Is there free AI bias testing software?
Yes: open-source toolkits compute common bias metrics in Python or R at no licence cost. They cover the metric step, which in Botminds' estimate is about a seventh of a full model test; group estimation, drivers, alternatives and the written explanation remain your team's work.
What is an AI fairness toolkit?
A software library that computes fairness metrics, such as rate ratios and error-rate differences between groups, and sometimes offers methods to reduce a measured bias. It is a component of a test, not the test: it assumes you already know each person's group and does not produce the documentation insurance regulators ask for.
How much does model bias testing cost?
Price it per model per year. Count the tests your triggers require, multiply by the days each takes and your loaded rate, then add an annual independent review. Botminds estimates 8–12 weeks of specialist time per test by notebook, which is why the number of tests a year matters more than the price of one.
Do we still need a consultant if we buy a tool?
Possibly, for independence rather than for running tests. NIST's framework and New York's letter both expect review by people who did not build the test; an annual outside review of the method and a sample of reports meets that need at a fraction of the cost of outsourcing every test.
Can a vendor's testing tool satisfy New York's circular letter?
It can do the work, but the responsibility stays with the insurer. New York keeps compliance with the insurer whatever a vendor claims, so the method must be visible and the conclusions signed by the insurer's own people.
What tools exist for AI model fairness in insurance?
Four kinds: open-source metric libraries, in-house notebooks built on them, consulting engagements that run a test and write it up, and testing tools that run the full cycle on each model version. Most insurers end up combining two: one for every test and one for independent review.