APP fraud detection means finding, among a bank's outbound payments, the ones a customer is being tricked into sending, early enough to stop them. The most honest way to test it is on scams the bank already knows it missed: take last quarter's reimbursed authorised push payment (APP) claims, replay the payments behind them as they looked on the day, and count how many would have been held with a stated reason by the end of the next business day, which is the UK deadline for establishing grounds to delay. Then count the other error, the genuine payments the same settings would have held. A pilot built that way measures the thing the bank pays for under reimbursement and the thing its customers feel. Botminds' APP Fraud Detection, which names the scam behind each held payment, writes the questions for it and flags coaching on the call, is the worked example on this page, and its fictional sample bank, Corvane Bank UK, supplies the illustrations; every figure about it is sample data, not a customer's result.
Pass marks for an APP fraud detection pilot, written before the first replay
- Caught in time
Of the payments behind last quarter’s reimbursed claims, the share that would have been held with a stated reason by the end of the business day after the bank received them. The bank sets the bar; the rule sets the deadline.
- Named correctly
Of those, the share where the likely scam named on the day matches the scam type the claims team later recorded, and how often the tool said “unclear” rather than guessing.
- Genuine payments held
On a sample of payments that were not scams, how many the same settings would have held, and for how long. FG24/6 expects technology calibrated to minimise the impact on legitimate payments.
- Reasons that open
Every reason given for a hold opens its source: the payee check response, the transcript line, the account history. No reason that rests on a score alone.
- Notices that pass
Every drafted delay notice states the delay, the reasons and what the bank needs, by the end of D+1, and mentions no internal referral or report.
- Flags people agree with
In the live phase, the share of coached-answer flags that a reviewer, listening to the call, agrees with.
- No decision without a person
Every hold, release, refusal and protective block in the live phase made by a named colleague, with a second person where the bank’s policy requires one.
How do you test APP fraud detection? Replay the payments behind your reimbursed claims
You test APP fraud detection by running it on payments whose outcome you already know, with nothing it could not have known at the time. The reimbursed claims are the right set because the bank has already accepted that each one was an in-scope scam, already borne its share of the cost, and already knows which scam type it was. The question for each is simple and unforgiving: given the payment, the payee check, the account history and the customer's contacts with the bank up to the moment the payment order arrived, would the tool have produced reasonable grounds for a hold by the end of the next business day?
That deadline is not arbitrary. Under the Payment Services (Amendment) Regulations 2024, a bank may delay a payment only if the grounds are in place by the end of the business day after the payment order arrives, and FG24/6 adds that a bank that misses it loses the power to delay that payment under that power. A detection that arrives on day three is, for this purpose, no detection. A replay that does not timestamp what the tool knew, and when, cannot answer the question a regulator would ask.
How one reimbursed claim is replayed in an APP fraud detection pilot
A person decides
Pilot ownerSelects the claim and freezes the inputs as they stood when the payment order arrived: payment, payee check, history, contacts.
Agent
Payment signals and typology agentsRead only those inputs; name the likely scam with a confidence and the signals behind it, or say “unclear”.
Never: See the claim outcome
Agent
Script and notice agentsDraft the questions the colleague would have asked and the D+1 notice the bank would have sent.
Never: Send anything
A person decides
Fraud strategy reviewerScores the result against the pass marks: in time or not, scam named right or wrong, reason opens its source or not.
Can you test on reimbursed claims? What the claim file holds, and what it lacks
Yes, and they are the best starting set a bank has, with three gaps to fill before the replay is fair. A claim file records the scam after the fact; a replay needs the payment as it looked before anyone knew. The table below lists what a pilot has to rebuild.
What the claim file holds, against what the replay needs
| Input | In a typical claim file | What the replay needs |
|---|---|---|
| The payment | Amount, date, payee details | The payment order as received, with the time of receipt, so D+1 can be counted |
| The payee check | Sometimes noted | The result the payee check returned at the time: match, close match, no match or unavailable |
| The customer’s contacts | Rarely | Contact centre transcripts and telephony records for the days before the payment |
| The fraud engine | The score, if it held the payment | The score it gave, whether or not it held the payment |
| The scam type | Recorded by the claims handler | Kept back from the tool, used only to score the typology |
| The call, if there was one | Notes | The recording, so coached answers can be checked |
The second gap is definitional. The PSR's own reporting shows that sending and receiving banks do not always agree on what is an APP scam: its 2024 report notes "disagreement between sending and receiving firms regarding whether certain claims meet the criteria for an APP scam", and singles out payments between a customer's own accounts and payments to crypto exchanges as hard cases. A pilot should decide in advance how it will score those claims, or exclude them, before seeing a single result.
The third gap is the claims the bank never received. Reimbursed claims are a biased sample of scams: they over-represent customers who knew the policy existed and chose to claim. The PSR's survey of victims a year into the policy found that nearly half had not tried to claim. A pilot that scores only reimbursed claims will overstate how much of the bank's scam exposure the tool sees; it should say so in its conclusions.
What do you need for an APP fraud pilot?
You need a quarter of reimbursed claims with their payment orders, a matching sample of genuine payments, the payee check results, the customer contact history for the days before each payment, the fraud engine's scores, and the bank's own scripts, reason codes and hold threshold. You also need two people with time: a pilot owner from fraud strategy who freezes the data and writes the pass marks, and a reviewer independent of the vendor who scores the results.
The genuine-payment sample is the input most often forgotten and the one that decides whether the tool can go live. Without it, a pilot measures only half of the trade-off. FCA FG24/6 records that, when it consulted, respondents asked for data on "true and false positive transaction delays to help assess the impact on legitimate payments", and the guidance tells banks to keep, for every delayed payment, the grounds, the length of the delay and "whether the transaction was ultimately completed or refused". A pilot should produce exactly those fields, for both sets of payments.
The bank's scripts and reason codes matter because the tool should be tested on the questions it will actually ask. In Botminds' APP Fraud Detection the script agent writes only from the bank's approved library and the typology agent names scams from the bank's own list, so a pilot run on the vendor's default scripts measures the wrong thing. The solution's own offer is to run on a sample of the bank's held payments and intervention scripts.
How many legitimate payments get delayed? Measuring the other error
The answer depends on the threshold, so a pilot should report it at the threshold the bank intends to use and at one either side. Every hold of a genuine payment costs the customer time, the bank a colleague's call, and, under the delay rules, any interest and charges the delay directly causes. FG24/6 asks banks to calibrate their technology so that legitimate payments are affected as little as possible and reminds them that an unusual payment is not, on that ground alone, a suspicious one.
The sample bank shows the kind of answer a pilot should produce. In its September, 2,614 of 1.08 million outbound consumer Faster Payments were held for an intervention, 0.24%, and 418 were stopped. When the typology agent proposed lowering the hold threshold from 700 to 680, it replayed the month's payments and estimated the effect: about 210 more payments held, about 31 more scams stopped, and about 1.6 more full-time colleagues to make the calls. That is one sample month and the agent's own estimate, not a measured result; the point is the shape of the report. A threshold change should arrive with its catch, its cost in genuine payments held and its cost in people, and in the solution it cannot go live without the model risk approver.
What a pilot report should look like: stops, scripts and a threshold change priced in advance
How do you measure APP fraud detection once it is live? The shadow phase
A replay can tell you whether the tool would have held the right payments; only live use can tell you whether its questions and flags help a colleague on a real call. The second phase of a pilot therefore runs the tool alongside the intervention line on live held payments, with colleagues deciding as they do today and the tool's typology, questions, flags and notices recorded for comparison. Nothing the tool produces changes a decision in this phase unless the bank chooses to let colleagues use it.
Four live measures matter. The share of held customers called within the bank's own target, which at the sample bank was 97.9% within 30 minutes. The share of coached-answer flags a reviewer agrees with on listening to the call: 93% of 611 flags in 1,904 calls at the sample bank. The quality of the notices, checked against the regulation's content and the bank's reading-age target. And outcomes for customers, tested the way the sample bank tests them, by having an auditor review a sample of 40 interventions with transcripts for the right questions, a plain-words warning and support offered; in its September, 38 were good and 2 needed remediation, one because a warning was read before the questions rather than after.
Every intervention leaves a file a reviewer can score: questions, flags, decisions and the standard of caution
Which claims to replay first, and why value beats count
Score the replay by value as well as by count, and report each scam type separately, because the two can point in opposite directions. In the PSR's data for the largest UK banking groups before mandatory reimbursement, purchase scams were almost three in four cases but just over a quarter of the money, while investment scams were one case in twenty-five and a quarter of the money (PSR, 2024 performance report). A tool that catches most small purchase scams and few investment scams can post a high catch rate and still leave most of the bank's half of the refunds unpaid for.
The scam types should be the ones the bank's claims team already uses, so that "named correctly" has a fixed meaning. The PSR's report defines seven, from advance fee and CEO fraud to impersonation, investment, invoice and mandate, purchase and romance. The sample bank's own list is finer, splitting impersonation into safe-account, police and government callers, because each needs different questions; a pilot can score at either level as long as it says which. For each type, the report should show the number of claims, their value, the share held in time, and the share named correctly, with "unclear" counted as neither right nor wrong but reported.
How to report a replay: one row per scam type, by count and by value
| Scam type | Claims replayed | Value | Held with reasons by end of D+1 | Named correctly | Said “unclear” |
|---|---|---|---|---|---|
| Impersonation (bank, police, other) | n | £ | % of claims · % of value | % | % |
| Investment | n | £ | % of claims · % of value | % | % |
| Romance | n | £ | % of claims · % of value | % | % |
| Purchase | n | £ | % of claims · % of value | % | % |
| Invoice and mandate | n | £ | % of claims · % of value | % | % |
| Advance fee and other | n | £ | % of claims · % of value | % | % |
How long should an APP fraud pilot run?
Long enough to cover a full quarter of claims in the replay and a full month of live payments in the shadow phase, which in our recommended plan is about ten weeks. The replay length is set by the claims: a quarter gives enough of each scam type to score the typology, and claims can arrive long after the payment, since a customer is expected to report within 13 months, which is why a pilot cannot be built on last week's data. Research on card fraud calls the same problem verification latency: investigators confirm only a small share of transactions in time for the labels to be useful, and a bank's scam claims share the problem. The live phase is set by the calendar: a month includes paydays, weekends and the bank's own busy days.
An APP fraud detection pilot in four phases
- Data and settings2 weeks
- Replay of last quarter3 weeks
- Live shadow phase4 weeks
- Review and decision1 week
What an APP fraud detection pilot cannot tell you
A pilot cannot tell you what a customer would have said on a call that never happened. The replay can show that a payment would have been held with good reasons, but whether the colleague's questions would have broken the scam depends on the customer, the scammer and the call. That is why the replay and the live phase measure different things and should not be added together into one success rate.
It also cannot replace model governance. For banks in its scope, the Prudential Regulation Authority's model risk principles, SS1/23, include "Principle 4 – Independent model validation"; a vendor's pilot report is evidence for that validation, not a substitute for it. And it cannot fix weak operations around the tool. The FCA found in its review of firms that management information "focused primarily on reporting against commercial risk appetite and financials" in many cases; a pilot that reports only money saved repeats the mistake. Report customer outcomes alongside it.
Finally, a pilot does not decide reimbursement claims, and should not be scored on them. The PSR's guidance on the consumer standard of caution says a bank that engages with a customer cannot hand the job of judging the payment's risk entirely to the customer. Detection improves what a claims handler has to read; it does not change the rules they apply, which belong to the separate APP Fraud Reimbursement solution. Why a stopped payment is worth more than a refund is set out in APP fraud prevention, what the vendor should be able to show in scam detection software, and the case behind every example here in APP Fraud Detection.
Questions people ask
What is a good detection rate for APP fraud?
We know of no benchmark published by a regulator or researcher, and a rate means little without the deadline and the cost behind it. Set the pass mark as the share of reimbursed-claim payments held with stated reasons by the end of D+1, alongside the share of genuine payments held at the same threshold, and agree both before the pilot starts.
Should a pilot include payments the bank already held?
Yes. Replaying held payments that colleagues released, and that later became claims, shows whether the tool's questions and flags would have changed the decision. At the sample bank, every claim that followed a release was handed to the claims team with the full intervention file, which is the material such a review needs.
Can an APP fraud pilot run without call recordings?
The replay can, because it measures detection before the call. The live phase is weaker without recordings: the listening agent can work on answers a colleague types, but a reviewer cannot check its flags against the call itself.
Who should own an APP fraud detection pilot?
A fraud strategy owner who writes the pass marks and freezes the data, with an independent reviewer scoring the results. The vendor should run the tool but not score it.
What rules apply to a pilot that touches customers?
The same as live use. Any live phase that delays a payment must follow SI 2024/1013 and FG24/6 on grounds, timing and notices, and the bank's Consumer Duty, data protection and model risk obligations apply; a replay on last quarter's payments delays no one.