Fraud model validation: replaying last year's closed new accounts

Fraud model validation is the test of whether a set of rules, scores or agents would have caught fraud it was never shown. For new deposit accounts, the target is an account opened with a stolen identity, rented as a mule or seasoned for a bust-out, and the aim is to stop it before money leaves. Whether a given set actually does that can be tested before it goes live: replay it over last year's accounts that were closed for fraud and over a matched sample of accounts that stayed good, feeding it only what the bank knew on each day. The two numbers that come out, days earlier for the bad accounts and restrictions per thousand for the good ones, are what a fraud committee should approve. Botminds' New Account Fraud Detection runs on that discipline: every threshold and agent version is tested on past outcomes and approved by the model risk manager before it runs.

The examples use the solution's sample data on fictional people and companies, at a fictional sponsor bank, Tessaly Community Bank, and its fintech partners. Arithmetic marked as illustrative is ours, shown to explain the method, not a result.

How do you validate a new account fraud model? Replay it

You validate it on accounts whose ending you already know, whether the "model" is a score, a rule set or a team of agents. A control that fires on a pattern is a claim about the future; the past is the only place where the claim can be checked against what actually happened. Federal model risk guidance calls this outcomes analysis: it "compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use", and it can take the form of "standalone activities such as back-testing or outlier analysis" (Revised Guidance on Model Risk Management, SR 26-2, April 2026).

For new accounts the replay has an unusual advantage: the window is short. An account opened in March has, by June, either been closed for fraud, been restricted and released, or graduated as an ordinary customer. A year of openings therefore gives a year of finished stories. A team that already closes accounts for fraud already holds the answer key; the replay just asks the new detection set to take the exam with the answers hidden.

Can you backtest fraud rules on closed accounts?

Yes, and closed accounts are the best material a bank has, provided they are not the only material. Replaying only the accounts closed for fraud tells you what share of last year's fraud the set would have caught and how early; it tells you nothing about the price. The price lives in the accounts that stayed good, and a replay that leaves them out will always recommend a lower threshold.

So a replay has two samples. The first is every account closed or restricted for confirmed fraud in the period, whatever its type: stolen identity, mule pass-through, bust-out, kiting, synthetic-looking. The second is a sample of accounts opened in the same months, through the same channels and partners, that were never restricted. Matching matters more than size. If the fraud came mostly through one partner's app and the good sample mostly through branches, the replay measures the difference between channels, not the detection set.

The Federal Reserve's synthetic identity toolkit suggests the questions to ask of each closed deposit account, among them "How long was the account open?", "Was the transaction activity minimal?" and "Did any suspicious transactions occur prior to the overdraft?" (Federal Reserve, Identifying Existing Synthetics Post-Loss). Those questions become the replay's columns: they tell you what the account looked like on each day before the loss, which is exactly what the detection set will be shown.

What data do you need for a fraud replay test?

You need the account's life as it was known at the time, and the ending as it is known now, kept apart. Most banks hold the first in four or five systems and the second in a case tool, which is why replays are rarer than they should be.

How do you avoid hindsight bias in a fraud backtest?

By never letting the replay see the future. The detection set may only use what was known on the day it is judging. That sounds obvious and is broken constantly, because the data a bank keeps today has been cleaned by the outcome: a customer record updated after the fraud investigation, a device later added to a negative list, a linked account discovered in the case work. Each makes the replay look cleverer than the live system could ever be.

"At the very least leakage leads to overestimation of the model's performance."

Kaufman, Rosset and Perlich, Leakage in Data Mining, KDD 2011

Data scientists call this leakage, which Kaufman, Rosset and Perlich define as "the introduction of information about the data mining target, which should not be legitimately available to mine from", and their remedy is what they call a "learn-predict separation" (KDD 2011). The model risk guidance names the same discipline for banks: model testing may include "out-of-sample and out-of-time testing" (SR 26-2). Out of time is the one that matters here. Tune on last spring's accounts, test on last autumn's.

How do you measure time to detection?

In days of the account's life, against the day the bank actually acted. For each closed fraud account the replay records the first day the detection set would have proposed a restriction, the day the bank really did, and the difference. A positive difference is days gained; zero means the set adds nothing for that account; a negative one, rare but possible, means the live team was faster.

Days are only half the measure. What the bank cares about is money, and in a new account the money does not leave evenly. A bust-out sits quiet for weeks and then pushes one large check through, so a detection set that is two days earlier but still after the check has been withdrawn against saves nothing. The better replay metric is exposure: what was still in the account, or still pending, on the replay's restriction day. Why the loss lands in the last two days of a bust-out is worked through on its own page.

How many good customers get flagged by fraud rules?

No public figure answers that for new accounts, and any figure quoted without a source should be treated as a vendor's. It depends on the bank's channels, its partners and where it sets its thresholds, which is precisely why the replay has a good sample. A new account review costs a good customer something real: a held paycheck, a blocked card, a call to a support line. The replay should therefore separate two kinds of flag. A review that ends with no action costs analyst time. A restriction on a good customer costs that customer, and is the number to put first. More on why published false positive rates rarely transfer from one bank to another is in the guide to false positive rates in fraud detection.

A replay also measures something no single number does: whether the proposals are right for the right reasons. In the solution every proposal carries reason codes and the evidence behind them, and the sample bank's evaluation of the action agent shows analysts overriding it on 7 of 298 reviewed accounts in October. An override rate that low is only reassuring if the replay also shows the reasons match the outcomes; otherwise it may only mean analysts are agreeing too easily.

Each agent version with its last evaluation, approved before it runs

Reading one replay: a lower threshold for one partner

The sample bank's partner oversight agent proposed lowering the review threshold for its largest partner, Fernway, from a score of 640 to 600. Replayed on August and September outcomes, the change would have sent 9 more Fernway accounts a month to review, of which 3 were later restricted for fraud, with $11,200 held. Extended to a second partner, Tidewell, at 620, the change added 14 reviews a month, 5 later restricted, $17,900 held.

That is a readable trade. Roughly one in three of the extra reviews found fraud, and the rest cost analyst time but, because a review is not a restriction, no customer harm. The committee's question is not whether the number is high or low in general; it is whether the bank wants to buy three restrictions a month with six extra clean reviews. The proposal does not run until the model risk manager approves it, and the previous version stays available for rollback.

A threshold proposal with its replay, waiting for model risk

What model risk will ask, and where the guidance stops

A model risk manager will ask what the replay was tuned on, what it was tested on, and whether the test period came after the tuning period. They will ask for the good-customer cost, not just the catch rate, and for the version of every setting used. They will also notice a gap in the current guidance. The April 2026 revision leaves generative and agentic AI out of its scope altogether, on the ground that they are new and changing fast, and its definition of a model excludes "deterministic rule-based processes" (SR 26-2).

That leaves a rules-only detection set and an agent-based one both largely outside the formal model definition, which is a reason for more testing, not less. The same footnote says a bank's "risk management and governance practices should guide the determination of appropriate governance and controls" for anything the guidance does not cover. A versioned replay, approved before use and rerun when fraud patterns move, is the practical form that takes.

Where a replay cannot help

A replay only knows the fraud the bank found. Accounts that left with the money and were written off as ordinary overdraft losses, or mule accounts that passed funds and closed quietly, sit in the good sample by mistake. Dal Pozzolo and colleagues name the same limit in card fraud: investigators confirm only a small share of transactions in time, so most outcomes arrive late or never (IEEE TNNLS, 2018). The fix is partial: before the replay, review overdraft charge-offs on young accounts for fraud that was booked as credit loss, as the Federal Reserve's post-loss toolkit suggests for synthetic identities.

The second limit is change. Fraud rings adapt, partners change their onboarding, and a replay on last year proves least about next year's newest pattern. That is why the replay is a recurring control, not a launch event: in the solution, a threshold change arrives with its backtest and waits for the model risk manager, and every analyst override is recorded with its reason, ready for the next review. The account-by-account trace shows what those agents propose day to day; New Account Fraud Detection shows the replay on your own accounts.

Questions people ask

How do you prevent new account fraud?

By combining checks at opening with watching the account's first weeks: the Red Flags Rule requires a written program to detect, prevent and mitigate identity theft both at the opening of an account and in existing accounts. In practice that means verifying identity, re-reading the opening file against what the account does, linking accounts that share devices or addresses, and acting before a large deposit can be withdrawn against.

What is a good detection rate for new account fraud?

There is no published benchmark from a regulator or researcher, and detection rate alone is the wrong target. A useful replay reports the share of fraud accounts restricted before money left, the median days earlier than the bank's real decision, and good customers restricted per 1,000. The solution's own design target is to restrict 80% of bad new accounts before their first large deposit clears.

How long a period should a replay cover?

At least two periods: one to tune on and a later one to test on, in line with the out-of-time testing the 2026 model risk guidance describes. For new accounts, which mostly reach an ending within 90 days, six to twelve months of openings usually gives enough closed fraud cases to see each common shape more than once.

Should synthetic identities be in the replay?

Yes, but many are hidden among credit and overdraft losses rather than fraud closures. The Federal Reserve's toolkit notes that from a deposit account perspective "there may be no indication of suspicious activity or behavior prior to intentionally overdrawing the account", so a replay that only uses fraud-coded closures will miss them.

Who should sign off a replay?

Someone independent of the team that tuned the settings. In the solution that is the model risk manager, who approves each threshold and agent version before it runs, with the previous version kept for rollback.