AI fraud detection companies sell a wide range of things under one label — a scoring model bolted onto an existing alert queue, a device-intelligence feed, or, in Botminds' case, a full agentic re-read of every early default with a proposed cause a credit risk analyst confirms. Before comparing what any of them claim to detect, the more useful first question is what a buyer is actually being asked to trust: a model's output, a vendor's data, or a vendor's judgment about where a human decision still belongs. This guide sets out twelve questions worth putting to any AI fraud detection vendor, grouped by the kind of risk each answers, illustrated against how Early Payment Default Review answers its own.
The examples below come from the product's own fictional sample vintage: Lanternfield Bank, August, 87 early defaults, including Ring R-0926, a 19-loan fraud ring the link analyst found. The regulatory context comes from the OCC, the Federal Reserve and NIST, linked where it appears.
AI fraud detection companies: what they actually sell, and what stays yours
Three things are usually on offer, and they are not the same purchase. A detection model proposes what is likely wrong with a case — a score, a flag, or, in an agentic design, a proposed cause with evidence. A workflow wraps that output in a queue, an approval screen and an audit trail. And a data feed, device intelligence or identity verification, often licensed from a third party behind the vendor's own product, supplies some of what the model reasons over. A buyer who evaluates only the detection model's claimed accuracy is evaluating one of three purchases, and often not the one that determines whether the tool is safe to run in production.
What should stay the bank's, regardless of which vendor is chosen, is the decision. Early Payment Default Review's own agents are explicit about this in their limits: the cause and fix writer "never confirms a cause" and "never sends a fix without the analyst"; the link analyst "never blocks an account or declines an application" and "never files a report — prepares the referral"; the credit rule tracer "never changes a rule." Those are not generic reassurances — each is a specific, checkable sentence, and the first thing worth doing with any vendor's own limits list is checking whether it reads the same way.
The twelve questions, grouped: data, accuracy, limits, governance
Twelve questions to put to any AI fraud detection vendor
- 1. What data does the model actually read, and where does it come from?
Data — your own loan files, or a shared pool across the vendor’s other customers? The two have very different privacy and competitive implications.
- 2. Does it read our documents directly, or a summary someone else produced?
Data — a cited finding traceable to the original pay stub or device log is verifiable; a pre-digested summary is not.
- 3. What is the measured false positive rate, on whose files?
Accuracy — a vendor’s own demonstration data answers a different question than your closed files do. Ask specifically which.
- 4. What is the measured false negative rate — what does it miss?
Accuracy — the number vendors are least eager to volunteer, and the one that matters most for a fraud ring you have not found yet.
- 5. How is a confidence score different from an accuracy rate here?
Accuracy — see this series’ guide on false positive rates for the distinction and why it matters for any bulk-confirm setting.
- 6. What can the agent never do, in exact words?
Limits — ask for the literal sentence, not a paraphrase. A vendor that cannot produce one has not written one down.
- 7. Who confirms a proposed cause, and is that step enforced in the product, not just the policy?
Limits — a limit that lives only in a sales deck is not a limit; it has to be built into what the software lets a user click.
- 8. Can the system contact a borrower, block an account, or send a fix on its own?
Limits — for Early Payment Default Review, the answer to all three is no; every fix is sent only when the analyst sends it.
- 9. What happens to a confirmed fraud ring — does the vendor’s product decide to file a SAR?
Governance — filing a suspicious activity report is the bank’s own BSA team’s decision, on a 30-day clock from detection; a vendor that files on your behalf is a red flag on its own.
- 10. Is every proposed credit rule change reviewed for disparate impact before anyone sees it?
Governance — ask whether fair-lending review is a configurable toggle or an unconditional step before a committee ever sees a proposal.
- 11. Is the vendor a regulated third party under your existing risk framework?
Governance — if it is not already in your third-party risk inventory, it needs to be before it touches credit decisions.
- 12. What does “live” mean, and which quality checks does it pass before then?
Governance — ask for that checklist in writing, and whether it includes a walkthrough on your own files before full production.
What a good answer to "what can the agent never do" sounds like
A good answer to the limits question is specific enough to fail a test case. "The link analyst never blocks an account or declines an application — it links the loans and prepares the referral for your BSA team to decide" names the exact action withheld and exactly who holds it instead. "The credit rule tracer never changes a rule — every proposal goes through fair-lending review before the credit risk committee" is checkable: a buyer can ask to see the fair-lending review step in the product, not just hear that it exists. These are the kind of sentences that can be tested in a demo by trying to make the system do the forbidden thing and watching it refuse, or by checking an audit trail for whether every fix really is unsent until a named person sends it.
What a red flag sounds like, and why it should end the conversation
A red flag is the same question answered in a way that cannot be checked. "The system is designed with human oversight in mind" says nothing about which decisions a human must make versus may make — NIST's own framework draws exactly that distinction, noting that "human-AI configurations can span from fully autonomous to fully manual" and that "AI systems can autonomously make decisions, defer decision making to a human expert, or be used by a human decision maker as an additional opinion" (NIST AI 100-1). A vendor who cannot say which of those three their own product does, for which specific action, has not actually built the limit — they have built a feature they are prepared to describe flexibly.
A good answer, against an evasive one, on the same four questions
A good answer: “Never blocks an account, never files a report, never confirms a cause — here is the audit trail entry that shows a person did.”
A red flag: “It’s designed to keep humans in the loop.” No named action, no way to check it.
A good answer: “We have not published one because it depends on your own files — here is how we would build it with you in a pilot.”
A red flag: “Our accuracy is 98%,” with no file set, no cause breakdown, and no offer to replicate it on your data.
A good answer: “We prepare the referral; your BSA team decides whether to file and when.”
A red flag: “We handle SAR filing for you,” without naming who at the bank signs it.
A good answer: “Yes — here is our documentation for your third-party risk assessment.”
A red flag: “We’re a tool, not a vendor you need to assess that way.”
Why an AI vendor is a third party under the same rules as any other
An AI fraud detection vendor does not get a lighter review because its output is a model score rather than a loan serviced by an outside shop. The OCC's own third-party risk guidance "describes sound risk management principles to consider when developing and implementing third-party risk management practices, commensurate with the bank's risk profile and complexity as well as the criticality of the activity supported by the third party" (OCC Bulletin 2023-17). A vendor reading applications, verification documents and credit decisions for 87 loans a month is supporting a criticality tier most banks would place well above a low-risk administrative service — which should drive the depth of diligence applied, not the vendor's own description of itself as "just software."
"Commensurate with the bank's risk profile and complexity as well as the criticality of the activity supported by the third party."
OCC Bulletin 2023-17, Third-Party Relationships: Interagency Guidance on Risk Management
Model-risk guidance adds a second layer specifically for the model itself, separate from the vendor relationship. The current text is the Revised Guidance on Model Risk Management that the Federal Reserve, the OCC and the FDIC issued on April 17, 2026, which supersedes the 2011 guidance in SR 11-7. Its cover letter, SR 26-2, says it is "expected to be most relevant to banking organizations with over $30 billion in total assets regulated by the Federal Reserve," and that models at smaller banks "typically are subject to internal risk management and governance practices appropriate for the size and risk profile" of the bank — while adding that "in some situations, this guidance also may be relevant" to them (SR 26-2). A smaller institution should confirm with its own risk function which applies, rather than assuming the large-bank guidance applies wholesale, or not at all.
What to ask about the systems it needs to read
A credible agentic fraud-cause tool needs to read widely to do the job this guide is describing, and what it needs to connect to is itself a question worth putting to a vendor plainly, before any pilot begins.
The kind of systems-integration inventory to ask any vendor for
| System | What it needs to read | Why it matters for the diligence |
|---|---|---|
| Loan servicing | Payment history and the day a loan crosses the early-default line | Determines how fast a case opens after the missed payment — ask for the exact trigger logic. |
| Loan origination | The application, uploaded documents and verification results | The core evidence every proposed cause cites; ask whether it reads the original document or a summary. |
| Decision engine | The decision record and every credit policy version | Needed to replay a decision under the policy in force at funding — ask how far back its version history goes. |
| Fraud checks | Fraud scores and device intelligence | Often itself a third-party feed behind the vendor’s product — ask who that sub-processor is. |
| Channel and partner data | Lead partners, sub-sources, merchants and dealers | Needed to trace a pattern like a ring back to the channel it came through. |
| App release log | What changed in onboarding, and when | Needed to catch the bank’s own errors, like autopay a release never saved correctly. |
How to run a short pilot on your own closed files before committing
The twelve questions above get a vendor's answers on paper; a pilot tests whether the product does what the answers claim. The shortest credible version runs a vendor's tool against a set of closed, confirmed early defaults — cases a bank already knows the real cause of from a completed investigation — and compares its proposed causes against the known ones, case by case, including the ones the bank's own review previously missed. Ring R-0926, in the sample vintage, is exactly the kind of case a pilot should include: a pattern that would have been easy to miss in a sampled review and straightforward to verify once the agent's link map is laid out for a person to check against the loan files directly.
A ring, laid out for a person to verify — not just to trust
A pilot is also where the "never" list gets its real test. Try to find a path in the product where a fix sends without the analyst clicking send, or a rule changes without fair-lending review, or a ring gets an account blocked without a referral to the BSA team. If every one of those paths is genuinely absent, the vendor's limits list was not marketing; if even one of them has a workaround, that belongs in the risk assessment before a production decision, not after.
What "live" should mean before you sign off
"Live" should mean the same thing for an AI fraud detection vendor that it means for any system touching a credit decision: validated on the bank's own closed files, reviewed by the appropriate model-risk or credit-risk function, with fair-lending review built into any path that touches a credit rule, and a named owner inside the bank for every category of fix the tool proposes. The product's own description of going live ties it to a list of quality checks completing, with a walkthrough run on the bank's own early defaults first — ask for that checklist in writing, with named owners, rather than accepting "live" as a single milestone a vendor declares on its own schedule.
The twelve questions in this guide line up closely with what a buyer needs to judge any of the approaches in fraud detection software for banks, compared on one vintage, and where Early Payment Default Review sits next to a bank's existing credit risk stack is set out in credit risk management software: where EPD review actually fits. To see the ring, the referral and the audit trail together, start from Early Payment Default Review.
Questions people ask
What questions should I ask an AI fraud detection vendor?
Group them by risk: what data it reads and from where, what its measured false positive and false negative rates are and on whose files, what it can never do in exact words, and whether fair-lending review, SAR referral and third-party risk assessment are built in rather than assumed. The twelve questions in this guide give one grouped starting list.
Is an AI fraud detection tool a third party my bank has to assess?
Generally, yes. The OCC's interagency guidance on third-party risk applies "commensurate with the bank's risk profile and complexity as well as the criticality of the activity supported by the third party," and a tool reading loan applications and proposing credit-relevant causes sits well above a low-criticality activity.
What is a red flag when evaluating an AI fraud detection vendor?
An answer that cannot be checked — "designed with human oversight in mind" instead of naming the specific action a human must take, or a published accuracy figure with no file set or cause breakdown attached. A vendor who cannot produce the literal sentence describing what their system never does has likely not built that limit into the product.
Does SR 26-2 apply to every bank evaluating an AI fraud detection vendor?
SR 26-2 states it is "expected to be most relevant to banking organizations with over $30 billion in total assets regulated by the Federal Reserve." Smaller institutions should confirm with their own risk function which model-risk framework applies, rather than assuming the large-bank guidance covers them in full or not at all.
Can an AI fraud detection vendor file a suspicious activity report for my bank?
In a well-governed design, no. A vendor's system may prepare the evidence and draft a referral, but the decision to file, and the filing itself, belongs to the bank's own BSA team — and under the Bank Secrecy Act's reporting rule, the bank is the filer of record.
How should a pilot with an AI fraud detection vendor be run?
On closed, confirmed cases from your own files, not a vendor's demonstration vintage, comparing the system's proposed causes against known outcomes — including cases, like a hidden fraud ring, that your existing process previously missed. Time the whole path from proposal to a confirmed, routed fix, not just the model's scoring step.