Synthetic identity fraud detection software can be tested properly before you buy it, on files you already own and already know the ending of, and the test is cheaper than the first month of any contract. The design is a replay: take twenty-four months of closed applications, feed them to the system in the order they actually arrived, and compare what it would have done with what happened. The reason it works is that a synthetic identity eventually declares itself — a booked account charges off and the collections team cannot find a person to call — so your own back book is a labelled data set that nobody else has. Botminds' Synthetic Identity Fraud Detection is offered on exactly that basis: a sample of past applications and confirmed fraud cases, replayed.
The honest version of this test is harder than the one usually proposed, in two ways. It has to score each application only against the applications that preceded it, or the result is flattering nonsense. And it has to produce four numbers rather than one, because a catch rate with no false-positive count beside it is a half-reported result.
The worked figures and screens below are the solution's sample data on a fictional issuer and fictional applicants, marked as such. They are an illustration of the method, not a benchmark — the only numbers that matter in this exercise are the ones that come off your own files.
The sample ring, as a back-test would score it
14
applications from 10 people, arriving over eight months
11of 14
that would have scored under 300 and passed, judged one application at a time
4
already booked as cards before the cluster crossed the hold line
1
genuine applicant linked to the ring, cleared on her own evidence
The only test that means anything
Three kinds of evidence get offered in this market, and only one of them is worth anything to you. A vendor's own catch rate, measured on a data set you cannot see, tells you about their data set. A live pilot on new applications tells you something real but takes a year to produce enough confirmed fraud to read, and costs you the fraud you take in the meantime. A replay on closed files tells you about your book, in a fortnight, with the answers already known.
The case for the third is strongest precisely where the category is weakest. The Federal Reserve recorded one credit union's machine learning tool that "flagged approximately 85% of credit applications originating from synthetic identities" in beta tests, and at the same time noted an estimate that "traditional fraud models were ineffective at catching 85% to 95% of likely synthetic identities" (Federal Reserve, July 2020). Two numbers in the same neighbourhood, measuring opposite things, on populations neither of which is yours. Your own replay is the only way to tell which one describes you.
What is a replay, and why does the order matter?
A replay feeds historical applications to the system one at a time, in the order they were received, and lets it score each one against only the applications that came before. The output is a decision history: what the control would have done on each date, with what it knew on that date.
Order is not a detail; it is the whole experiment. A link-based control gets stronger as applications accumulate, so a system shown all fourteen files at once will find the ring instantly and tell you nothing. The question you are asking is when it would have found it — on the fourth application, the seventh, or two months after the first card was booked. Only date-ordered scoring answers that, and the difference between the two designs can be the entire business case.
In the solution's sample ring the shape is visible. Through to the fifth of October the cluster carries a recycled voice-over-internet number and little else, and the score sits at 540 against a hold line of 700. On the eleventh, a seventh application arrives on the same Android handset, the credit files are re-read, one borrowed authorised-user card turns up on seven of them, and the score goes to 904. A replay shows you both the 540 and the 904. A bulk scoring run shows you only the 904.
The same fourteen applications, scored two ways
- Each applicant has a real number and a credit file; 11 of the 14 score under 300Feb – Oct
- Four booked as cards between February and AprilFeb – Apr
- Faint links on one recycled telephone line; score 540, under the 700 hold lineto 5 Oct
- Seventh application on one handset; one borrowed card on 7 credit files; score 90411 Oct
- Six applications held, three limit increases pausedfrom 11 Oct
Step one: the files you already have
Five sets, all of which exist somewhere in your institution already, and none of which require a vendor to assemble.
Twenty-four months of closed applications, including the ones you declined and the ones you booked. Declines matter: a synthetic identity's first contact with a lender is usually a refusal, and the file it builds afterwards starts there.
The booked accounts that charged off, with the collection notes. This is where your unlabelled positives live, and the Federal Reserve describes exactly the signature to look for — the collections team that cannot find a real person to pay.
Your confirmed fraud cases, which are your known positives. These are the ones you already investigated and called synthetic, and they anchor the whole exercise.
The contact and device data as it was captured at the time — telephone number, device session, internet address, email address, address type. If this was never stored, say so now rather than halfway through: a link model without it is being tested with one hand tied, and the result will understate what it can do live.
The decisions your current control actually made, so the comparison is against what you run today rather than against nothing. A replay that beats an empty baseline has proved only that something is better than nothing.
Step two: what counts as a hit
This is where most back-tests quietly fail, because "caught a synthetic identity" means four different things and the one you pick changes the answer by a wide margin. Fix the definition before you see any results, write it down, and hold the vendor to the same one.
Four ways to count a catch, and what each one hides
| The definition | What it counts | What it misses | When to use it |
|---|---|---|---|
| Held before booking | Applications the system would have stopped before any credit was extended, that you later confirmed as fraud. | Rings you never confirmed because they never charged off, and identities still sleeping in the portfolio. | The primary number. It is the only one that maps directly to money not lent. |
| Re-classified charge-off | Accounts booked as credit losses where the replay links them to a confirmed ring. | Charge-offs with no surviving contact data, and genuine hardship cases that look linked by coincidence. | The number that changes how your loss is coded — and often the largest one. |
| Linked to a confirmed case | Any application the system joins to a case your investigators already proved. | Nothing about timing. An application linked after the ring was found is not a catch. | A sanity check on the link model, not a performance measure. |
| Flagged at any score | Every application the system raised above zero. | Everything useful. A control that flags a quarter of your book catches every ring and is unusable. | Never, on its own. Ask for it only alongside the volume it implies. |
The same point applies one level up, to the word itself. There is no settled industry definition of a synthetic identity, which is why two honest measurements can disagree.
"Fraud industry experts advise that these detection models could be improved if they utilize a standard definition of synthetic identity fraud. This would allow for broader comparison and analysis of the data and results."
Federal Reserve, Mitigating Synthetic Identity Fraud in the U.S. Payment System, July 2020, page 9
Step three: the four numbers
Four, reported together, every time. Any one of them alone can be made to look good by moving a threshold, and the set cannot.
Rings found before booking. How many confirmed rings the replay would have stopped at application, and how much credit that represents. On the sample ring the recommended actions keep $102,500 — five declines worth $72,000, three limit increases worth $16,000 and four open card lines worth $14,500.
Charge-offs re-classified. How many accounts you booked as credit losses the replay ties to a ring. This is the number that most often surprises a credit committee, because it moves loss out of one column and into another without any new money being involved.
Genuine applicants who would have been held. The cost side, in people rather than percentages. Count them, then read a sample of their files, because the aggregate rate tells you much less than four actual applicants do.
Days of warning gained. For each confirmed ring, the gap between the date the replay would have crossed your hold line and the date you actually found it. This is the number that tells you whether a control changes outcomes or merely confirms them, and it is the one most often missing from a vendor's pack.
The four numbers, on the sample ring
A replay, mid-ring, still under the hold line
The number vendors leave out, worked through on one cleared link
A link-based control will reach genuine people, because genuine people share elements with fraudsters: an employer, a building, an internet provider, a telephone exchange. The question is not whether that happens but what the control does next, and a back-test is where you find out.
The sample ring contains exactly one such case, and it is worth working through because it is the shape of every complaint you will receive. An applicant is pulled into the cluster only because three ring members claim to work at her real employer. Her own evidence is unambiguous: the government number check matches, her credit file goes back to 1998, the telephone number is the company's main line rather than the ring's voice-over-internet "HR" number, and her licence and selfie pass. Her score is 212 and she is cleared to underwriting for $11,000 — and left out of the case file, because she did nothing.
Two things in that paragraph are testable on your own files. First, whether the control can separate a shared employer from a shared handset at all — that is what the link weights are for, and a control that treats every shared element as equal evidence will produce false positives you cannot defend. Second, whether a cleared applicant leaves a record of why they were cleared, because a control with no false-positive history has nothing to answer a complaint with and nothing to tune against.
It is also the point where federal identity guidance stops being abstract: whoever runs a fraud check has to monitor how it is performing and give applicants a documented way to resolve problems it causes them (NIST SP 800-63A-4, §3.2.3). A back-test is the first measurement in that obligation, not a separate exercise.
One genuine applicant, and the evidence that cleared her
What a hold-out design looks like
If the system is being configured by anyone who can see your outcomes, the result is contaminated — not through bad faith, but because knowing which accounts charged off makes it very easy to choose weights that find them. The fix is ordinary experimental hygiene.
Pick a cut date somewhere around two thirds of the way through the period. Everything before it is available for configuration: thresholds, link weights, which checks run. Everything after it is sealed, and nobody tuning the system sees it until the configuration is frozen and written down. Then the replay runs across the whole period and you read the results on the sealed part only.
Two refinements are worth the effort. Keep the confirmed-fraud labels out of the configuration set entirely where you can, so the system is being asked to find rings rather than to recognise ones you have already named. And have one person outside the evaluation hold the cut date and the sealed outcomes, so "we moved the date after we saw the first run" never becomes a thing that quietly happened.
The federal identity guidelines make this a disclosure matter as well as a methodological one: any supplier using a model is expected to document what it was trained on, how often it is updated, and the results of the testing it has already completed (NIST SP 800-63-4, §3.8). If a vendor configures against your outcome data, that configuration is part of the model's history and should be written down as such.
Why your definition of a synthetic identity is part of the test
Before the first number is produced, somebody has to decide which of your charge-offs count as synthetic. That decision is not neutral, it is not obvious, and it will move your results more than any threshold will.
The useful marker is the one the Federal Reserve gives for the moment the truth emerges: an institution often does not identify an account holder as synthetic until after the bust-out, when the collections team cannot find a real person to pay the debt. A practical rule built on it: a charged-off account is a candidate if collections could not reach a living person at the address or telephone number on file, and it is confirmed if the application's identity elements fail on re-examination — a number that does not match, a file first seen shortly before the application at an age that does not fit, an address that is a mail-receiving agency.
Write the rule down, apply it to every candidate before anyone sees a replay result, and keep the list of accounts you could not decide about. That undecided pile is itself a finding: it is the measure of how much your loss data cannot currently tell you, and it is usually the argument for capturing better contact data at application whatever you buy.
What should a fraud vendor prove before you buy?
Eight things, all of which are answerable on your files rather than theirs. These are deliberately different from the model-disclosure questions the federal identity guidelines hand you, which are about how a system was built; those are set out alongside the honest division of a fraud analyst's work. The eight below are about evidence.
Eight things to make a vendor prove on your own closed files
- Run the replay in date order, and show the score history
Not a bulk scoring run. You want the score on each date, so you can see when the control would have acted rather than that it eventually would.
- Report all four numbers, not the catch rate alone
Rings found before booking, charge-offs re-classified, genuine applicants held, days of warning gained. Any single number can be moved by a threshold.
- Agree the hit definition in writing first
Before any result is produced. Four reasonable definitions give four different answers, and the one chosen after the fact is always the flattering one.
- Accept a hold-out, and name who holds it
A cut date, sealed outcomes, a frozen configuration and a person outside the evaluation who owns all three.
- Show the hold volume behind the catch rate
How many applications in total would have been held each month. A control that stops two per cent of your book is not deployable whatever it catches.
- Explain one catch and one miss, by name
Which element linked it, which application it was found on, which document it was read from. A score with no explanation cannot be defended to an applicant or an examiner.
- Show what it does with a shared employer
The commonest innocent link. If every shared element carries the same weight, the false-positive number will tell you so expensively.
- State what it cannot see in your data
If device or contact data was not captured historically, the replay understates live performance — and the vendor should say so before the result, not after.
To run the replay on your own past applications and confirmed fraud cases, start from Synthetic Identity Fraud Detection. For what the whole control actually consists of before you test it, the sequence of checks and the data each one needs is the companion page.
Questions people ask
How do you test a fraud detection system?
On closed files whose outcome you already know. Take twenty-four months of applications — declined and booked — together with the accounts that charged off and your confirmed fraud cases, seal the last third as a hold-out, replay everything in the order it arrived so each application is scored only against those before it, and report four numbers: rings stopped before booking, charge-offs re-classified, genuine applicants who would have been held, and days of warning gained.
What counts as a hit in a fraud back-test?
Whatever you wrote down before you saw a result. The strongest definition is an application the system would have held before any credit was extended that you later confirmed as fraud. A weaker but valuable one is a charged-off account the replay links to a confirmed ring. "Linked to a case" ignores timing, and "flagged at any score" means nothing without the hold volume beside it.
What is a good false positive rate for fraud detection?
There is no published benchmark worth quoting, and anyone offering one is describing a different book from yours. The answerable version is comparative: how many genuine applicants would the new control have held that your current control did not, on the same files, for the catch it adds. Then read a sample of those applicants' files, because four real cases tell you more about whether the rate is tolerable than the rate does.
How do you avoid testing on data the model has already seen?
With a hold-out. Choose a cut date, make everything after it unavailable to anyone configuring the system, freeze and document the configuration, then replay the whole period and read only the sealed part. Keep the confirmed-fraud labels out of the configuration set where you can, and put the cut date and the sealed outcomes in the hands of someone outside the evaluation.
How many of our charge-offs were really fraud?
Nobody can tell you without doing the work, and that is the point of the exercise. The candidate set is charge-offs where collections could not reach a living person; the confirmed set is those whose application identity elements fail on re-examination. The accounts you cannot decide about are a result in their own right — they measure how much your current loss data cannot say.
How do you measure a link model's accuracy?
Not as a single accuracy figure, because the unit of a link model is a ring rather than an application. Measure it on rings found before booking, the warning gained on each, and the genuine applicants pulled in — and look at the weights behind the links, because a model that treats a shared employer the way it treats a shared handset will score well on catches and badly on the people it was wrong about.