Claim severity modeling for litigation scores: what to validate

Claim severity modeling estimates the cost of a claim: the dollars of loss per claim, predicted from what is known about it. In actuarial practice the workhorse is a generalized linear model with a gamma distribution, because claim costs are right-skewed with a long tail; the Casualty Actuarial Society's monograph on GLMs calls the gamma "a natural fit (and indeed the most widely used distribution) for modeling severity in a GLM". On an open bodily injury file the biggest driver of severity is often an event, not a characteristic: the claimant hires a lawyer and sues. Botminds' Litigation Propensity Prediction scores that event. Six agents read every open injury file each night and give it a 0–100 score with the phrase behind each point, and a next-step writer prices the options from similar closed files. For a second line, both halves are models to validate. This guide lists the questions, then works through them on the solution's test.

What is claim severity modeling?

Claim severity modeling predicts the average cost per claim, or the expected cost of one claim, from its characteristics. Pricing actuaries model frequency and severity separately and multiply them; claims and reserving teams use severity estimates on individual open files to set reserves, route large losses and decide where to spend investigation money. The usual form is a GLM with a log link and a gamma or inverse Gaussian error; tree-based machine learning models are also used.

Severity models assume the claim's path. A severity estimate made at first notice for a rear-end collision with a back injury carries an average mix of represented and unrepresented outcomes. When the claimant hires a lawyer, the expected cost of that one file moves to a different distribution. That is why carriers pair severity models with propensity models: one asks how much, the other asks which road the file is on.

Where a litigation score sits beside a severity model

A litigation score is a classifier: it ranks open files by the chance of a defined event within a window. In Litigation Propensity Prediction the score is additive and inspectable. Each signal adds points within one of seven families, each family has a maximum, and the total runs from 0 to 100. A supervisor can see that a file is at 78 because of +19 for a lawyer mentioned before any letter, +17 for escalating treatment, +14 for the venue, +12 for liability, +9 for claimant facts and +7 for an unreturned call.

The severity half sits in the next step. For each routed file, the next-step writer finds similar closed files and prices the options: resolving now, waiting for a demand, or assigning counsel. On the sample file that is $38,000–$55,000 to resolve early against $120,000–$190,000 if the claimant is represented and sues, plus $25,000–$40,000 of defense. Those ranges are severity estimates conditional on a route, and they need the same scrutiny as the score: how similar is similar, how many files stand behind a range, and how old they are.

How do you validate a claims prediction model?

Out of time, without leakage, against the base rate, and on your own files. The CAS monograph recommends splitting by year rather than at random where events cluster, because the model is then tested on a period it never saw, "giving us a more accurate view into how the model will perform on unseen years". The NAIC bulletin describes validation in similar terms: comparing performance on unseen data at development with "the performance observed on data post-implementation".

Leakage is the subtler failure in claims. A closed file contains its own ending: the summons, the defense invoices, the settlement memo. A score rebuilt from the whole closed file will find the suit because the suit is in it. The solution's test was run on the sample carrier's 2025 closed injury files with each score computed, in the test's own words, only from what was in the file on the scoring day, 30 days before suit or closing. Ask any vendor to show how that was enforced.

The closed-file test: what happened, by score band

Reading a closed-file test band by band

Start with the base rate. In the sample test, 286 of 6,924 closed injury files were sued: 4.1%. Every band is read against that. A file in the 75–100 band was sued 59% of the time, about fourteen times the base rate; the 0–39 band, 0.4%, a tenth of it. The two upper bands together hold 508 files and 221 of the suits, so 43.5% of everything on the heat list was eventually sued, and 77% of all suits were on the list at least 30 days before service.

Then read the misses. Of the 65 suits the test missed, 27 were served with no warning in the notes, and 21 were on files opened after a letter of representation had already arrived. Neither is fixed by better weights: the first needs better notes, the second is a file that was represented before the carrier ever saw it. A validation report that separates misses by cause tells the business what the model can and cannot be asked to do.

Finally, read warning time by line. In the sample test, commercial auto suits were flagged a median 44 days ahead, personal auto 37 and general liability 40. If one line's warning is too short for the specialist desk to act, a single company-wide threshold is the wrong design; how thresholds are set and owned is covered in a claims triage model for injury files.

Can a litigation model use ZIP code?

It can be built to, but where a law like Colorado's applies, the insurer has to be able to show the result does not unfairly discriminate; the solution excludes it outright. Colorado's SB21-169 bars insurers from using external data, algorithms or predictive models that unfairly discriminate by race, color, national or ethnic origin, religion, sex, sexual orientation, disability, gender identity or gender expression, and requires each insurer to "Establish and maintain a risk management framework that is reasonably designed to determine, to the extent practicable, whether the insurer's use of external data sources unfairly discriminates". The rule is about outcomes, so leaving a field out does not settle the question.

Litigation Propensity Prediction never reads or uses race, ethnicity, religion, national origin, sex, income or ZIP code, as a fixed setting. Two of the inputs it does use deserve a second line's test. Venue is the county a suit would be filed in, and counties differ in their populations. And one claimant signal adds points for an occupation a jury will relate to, with teacher, nurse and bus aide as the examples; in the Bureau of Labor Statistics' 2025 figures, women held 87.3% of registered nurse jobs and 79.2% of elementary and middle school teacher jobs, against 47.1% of all employment.

The aim is not to drop every input that correlates with something. Venue genuinely changes what a suit costs, and claims counsel will want it. The aim is to measure whether the score's errors fall unevenly, and to record the decision either way. The fairness testing that sits across all of a carrier's models is the work of AI bias and fairness testing; a claims desk brings it the score, the inputs and the closed-file outcomes.

What does the NAIC AI bulletin require for claims models?

A written program, proportionate to the risk, covering every model that makes or supports regulated decisions, claims included. The bulletin had been adopted by 29 jurisdictions as of 8 October 2026. For predictive models it asks for "Inventories and descriptions of the Predictive Models", documentation of development and use, assessments of interpretability, repeatability, robustness and drift, and validation before and after implementation. Its list of factors for sizing controls includes how far people are involved in the final decision.

From the claim desk, that means four artefacts per model: what it predicts and why, the test it passed, who can act on its output, and the change history. In the solution these exist as screens and settings: the Signals screen with the weights version and its approvers, the Results screen with the closed-file test, the authority table, and the rule that weights change only after a test on closed files and a second approver. The carrier-wide inventory belongs to AI governance and model inventory.

The signal weights, with their version and approvers

How do you monitor model drift in claims?

By watching the inputs, the outcomes and the law. The NAIC bulletin defines model drift as "the decay of a model’s performance over time" arising from changes in the data. Claims data drifts for three reasons a second line can name in advance.

The law changes the outcome. Since 2023, a Florida plaintiff found more than 50% at fault may not recover any damages outside medical negligence; a model trained on earlier Florida files has learned a different relationship between liability and suit. Claims practice changes the inputs: a new note template, a new call-centre script, a new medical bill feed. And the model changes its own outcomes. Once files at 75 or above go to a specialist the same day, some of the suits the score predicts never happen. In the sample carrier's first four and a half months, 97 of 168 files that crossed 75 were taken before a lawyer's letter and 61 resolved without counsel. Those files are no longer clean test labels, and the next validation has to account for the intervention.

A score that works changes the outcomes it was tested on. Validation after go-live has to separate files the model touched from files it did not.

Overrides and dismissals are validation evidence

Every time a supervisor dismisses a signal, the model has been challenged by someone who read the file. The bank regulators' 2011 model risk guidance, SR 11-7, never bound insurers and was replaced for banks in April 2026 by SR 26-2, but it set out the practice in detail, and on overrides it was blunt: "Such overrides are an indication that, in some respect, the model is not performing as intended or has limitations." It went further: "If the rate of overrides is high, or if the override process consistently improves model performance, it is often a sign that the underlying model needs revision or redevelopment."

In the solution, any signal can be dismissed, but only with a reason: read out of context, already handled in a later note, or a duplicate. The score drops by that signal's points at once and the reason is saved for the next weights review. A second line should ask for the dismissal counts by family and reason each quarter; the supervisor's side of the same edit is described in a litigation risk assessment supervisors trust. A family dismissed often as "read out of context" needs a better reader; one dismissed as "already handled" needs a later read of the file, not new weights.

Who can change what: thresholds, authority and exclusions

Questions people ask

What is claim severity modeling?

Predicting the cost of a claim, or the average cost per claim, from what is known about it. Actuaries usually use a GLM with a gamma distribution, which the CAS monograph calls the most widely used distribution for severity; claims teams apply the estimates to reserves and routing.

How do you validate a claims prediction model?

Test it out of time on years it never saw, rebuild each score from only what was in the file on the scoring day, compare every band with the base rate, and break the misses down by cause. In the sample test, the top band's 59% suit rate meant something because the book's rate was 4.1%.

Can a litigation model use ZIP code?

It can, but where laws like Colorado's SB21-169 apply, the insurer must be able to show its models do not unfairly discriminate, whatever their inputs. Litigation Propensity Prediction excludes ZIP code and income as fixed settings, and still needs venue tested for proxy effects.

What is proxy discrimination in insurance?

An input that is allowed standing in for one that is not, so that outcomes differ by a protected characteristic even though the characteristic was never used. An occupation signal built on teacher and nurse examples can act on sex, since women held 87.3% of registered nurse jobs in 2025 according to the BLS.

How do you monitor model drift in claims?

Track input mixes, band outcomes and overrides over time, and mark legal changes, such as Florida's 2023 move to modified comparative negligence, as known breakpoints. Separate files the model routed early from the rest, because an effective score prevents some of the suits it predicts.

What does the NAIC AI bulletin require for claims models?

An inventory and description of each predictive model, documentation of its development and use, validation before and after implementation, drift monitoring and controls proportionate to the harm a wrong decision can do. It covers models bought from vendors as well as models built in house.