Fraud investigation software: how to measure what it proposes

Fraud investigation software, often called a fraud investigation case management system, is where a bank's fraud investigators gather evidence, record decisions and close cases. Once that software starts proposing decisions, by merging alerts, drafting the memo and suggesting a decision code, it has to be measured the way a supervisor measures a junior investigator: is the evidence complete, is every claim in the write-up supported, does the experienced investigator keep the proposal, and is the case decided on time. A system that cannot show those four numbers on your own cases cannot be trusted with proposals, however good its demonstration looks. Measurement is hard because fraud labels arrive late and only for the cases someone checked. Botminds' Fraud Case Management records each measure on every case, so a fraud operations manager can sample, compare and challenge them.

The figures about the product on this page are sample data from a fictional bank, Corvane National Bank, and its invented customers and staff, or design targets where they are called targets. The sampling arithmetic is ours and is shown so you can redo it on your own volumes.

What is fraud investigation software?

It is the system of record for fraud investigations: every alert, every piece of evidence, every decision and who made it, every dollar that moved because of it. Older systems were filing cabinets with workflows. Newer ones do part of the investigator's work: they merge related alerts, pull evidence from the core and the payments systems, draft a memo and propose a decision. That second kind is the subject here, because a proposal is a claim about the truth, and claims have to be tested.

The test matters more than the feature. A system whose proposals are almost always kept on easy cases and often rejected on hard ones will look excellent in an average. A system whose memos cite sources that do not support the sentence will look rigorous until someone clicks. The four measures in the table above are designed to catch both, and each one is countable from the case record if the system records it.

Is the evidence complete? What makes a fraud investigation reasonable

For consumer claims the bar is set in law. Regulation E requires a bank that receives a notice of error to investigate and decide within 10 business days, or up to 45 days with a provisional credit, and the CFPB's compliance FAQs spell out what the investigation must be: reasonable, which includes a reasonable review of the relevant information already in the bank's own records. The evidence a bank holds and did not look at counts against it.

The same FAQ gives the failure case: the Bureau found an institution did not investigate reasonably when it summarily denied disputes because the consumer had prior transactions with the same merchant, without considering other relevant information such as the consumer's own assertion. That is an evidence-completeness failure, and it is measurable. For each decision code, list the sources a reasonable investigation of that type needs; then count, case by case, whether they are present.

On the sample bank's account-takeover code, the binder gathers nine: the alerts, core transactions, the digital banking device log, the check image, the call transcript, prior cases, recovery records from the payments hub and the bank's own standard for new-payee holds. When a source is unavailable, the evidence agent says so rather than guessing, and the gap is visible on the case. A completeness rate below 100 percent is not automatically a failure; an unexplained gap is.

The case binder: nine sources, each with what was taken from it

Is every claim supported? Scoring sentences against their sources

Claim support asks whether each sentence is true to the passage it cites, which is a stricter test than whether it has a citation. On the sample case all 17 sentences were cited. One still failed: sentence 4 stated that the owner "clicked a phishing link", citing a call transcript in which the owner said only what she believed. The quality agent scored its support at 0.58, below the bank's 0.70 threshold, and proposed wording that kept the belief and dropped the fact. The supervisor accepted the rewording.

Count claim support at two moments. Before approval, the share of sentences challenged tells you how often the drafting needs correcting. After approval, a quality analyst's sample tells you how many weak sentences got through anyway. The second number is the one that matters to an examiner; the first tells you whether the drafting is improving. A system that only reports the first is marking its own homework.

Keep the human edits in the count. On the sample bank, any sentence a person types is marked as theirs and the agent's wording is kept in the history. That makes it possible to tell an agent error from a reviewer's style preference, which matters when the number is used to approve a new agent version.

What is decision agreement in fraud investigations, and why is it not accuracy?

Decision agreement is the share of the system's proposed decisions that the investigator keeps. It is not accuracy, because the investigator can be wrong too, and because on most fraud cases nobody learns the true answer quickly. Researchers who tested fraud models on a stream of more than 75 million card transactions named the problem plainly: verification latency, their term for the fact that investigators check only a small share of transactions in time for the answer to be of use. The labels that would prove accuracy arrive late, and only for the cases someone examined.

So measure agreement, and then make the disagreements informative. On the sample bank, overriding an agent's proposal needs a reason code, and the codes go to fraud strategy monthly. The sample quarter's override reasons mix two things worth separating: overrides of the detection engines, where a customer confirmed the activity, and overrides of the agents themselves.

Two of those codes measure the agents directly. AO-3 counts merges the intake agent missed; AO-4 counts decision proposals the evidence contradicted. Track each as a rate against the proposals of that kind, by decision code and by month. A rising AO-4 on one fraud type is the earliest sign that the drafting has drifted or that the fraud has changed.

How do you QA fraud investigations? Sampling closed cases

Sample closed cases at random, by decision code, and big enough to see the error rate you care about. The sample bank's quality analyst reviews a 6 percent weekly sample of closed cases with a checklist. If about 6,910 cases close in a quarter, that is roughly 415 reviews a quarter (our arithmetic). Whether that is enough depends on what you are trying to detect.

The last two numbers explain a design choice. A golden set of about 300 closed cases is roughly what it takes to have a good chance of seeing a 1-in-100 failure at least once. The sample bank replays such a set of 300 closed cases before any new agent version goes live, and the Chief Fraud Officer approves the result. Stratify the set: if account takeovers are a fifth of your cases but most of your losses, give them more than a fifth of the sample.

Random sampling must also include the cases nobody escalated. A QA program that reviews only overrides and complaints measures the system's visible failures and misses the quiet ones, such as a well-written memo that closed a case as no fraud when the evidence was thin.

Time to decision, measured from the right event

Time to decision is the time from a complete case file to a signed decision. Start the clock at the wrong event and the number measures something else: from the alert, it mostly measures queue length; from case assignment, it measures staffing. From a complete binder, it measures how long people take to judge and sign, which is what better drafting should shorten.

On the sample bank's third quarter, the median from a complete binder to a decision was 38 minutes; the design target is the same day. Report the slowest tenth alongside the median, because the cases that sit are the ones that breach clocks. The regulatory clocks run from other events entirely: Regulation E from the consumer's notice, and the SAR filing deadline from the bank's initial detection of facts that may justify one. Those belong on the queue report as separate measures, covered in fraud KPIs for fraud operations.

The supervisor's view: maker, checker and what the approval will do

How accurate are AI case summaries?

Nobody can answer that in general, and a vendor who gives you one number is answering a different question. We know of no published regulatory or peer-reviewed benchmark for systems that draft fraud case memos. Accuracy depends on the bank's sources, its decision codes and its fraud mix, so it has to be measured on the bank's own closed cases, with the four measures above.

A practical protocol: take 300 closed cases across your decision codes, with their final decisions. Run the system on the evidence as it stood at the time of decision. Score evidence completeness and claim support for every case, compare the proposed decision with the final one, and have a senior investigator adjudicate every disagreement, blind to which side proposed what. The adjudicated disagreements are where you learn the most. Running that pilot on cases you already closed is how a bank can test the solution before buying it.

What should a fraud case file contain?

Enough for a stranger to reach the same decision a year later. In practice that means the triggering alerts and how they were merged, every source consulted with the record itself, the classification of the fraud, the decision with its code and reasons, who made it and who approved it, the money that moved, any recovery steps still open, and, where one was made, the referral reference without its contents. If the case becomes the basis of a SAR, its documents become supporting documentation the bank must keep for five years from filing and produce on request under 31 CFR 1020.320(d). How to write the memo itself is set out in how to write a fraud investigation report.

What the sample quarter's measures look like at the top

Questions people ask

How do you measure fraud case quality?

With measures you can count from the case record: evidence completeness, claim support, decision agreement and time to decision, plus a random sample of closed cases reviewed against a checklist. On the sample bank, a quality analyst reviews 6 percent of closed cases each week, and challenges raised before approval are kept with their outcomes.

What is a good decision agreement rate?

There is no published benchmark, and a very high rate can mean investigators have stopped reading. Watch the trend by decision code and read every case where the evidence contradicted the proposal; on the sample bank those carry reason code AO-4, 61 in one quarter.

How do you sample closed fraud cases?

At random, stratified by decision code and loss size, and sized for what you want to detect. Estimating an error rate within five points at 95 percent confidence takes about 364 cases out of 6,910; having a 95 percent chance of seeing a 1 percent failure at least once takes about 299 (our arithmetic).

Should QA review cases before or after approval?

Both, for different reasons. Before approval, a challenge on a weak sentence fixes the case while it is open; on the sample bank the quality agent does this on every memo. After approval, an independent sample measures what got through, which is the number an examiner will care about.

Does a case management system need its own validation?

The proposals it makes should be tested before each change goes live. The sample bank replays 300 closed cases for every new agent version and every material rule change, and the Chief Fraud Officer approves the result.