Piloting card dispute automation on your own closed claims

The risk in buying chargeback automation software is not that it fails loudly; it is that it fails politely, agreeing with your analysts on the easy claims and drifting on the hard ones, quietly, for a quarter. The pilot design that finds that drift before it costs anything has three moves: replay claims you have already closed, score the system’s findings against the decisions your own analysts signed at the time, and only then run it live beside the team for a week against pass bars agreed before the pilot starts. Everything in this method is done with the bank’s own files, on its own clocks, and nothing in it requires trusting a vendor’s demo data. Card Dispute Management is built to be tested exactly this way — its own sample claim runs a replay of its central rule, which is the first test a pilot should repeat on your claims.

The replay a pilot repeats: the standing rule tested against the bank’s own calendar

How to test chargeback automation software before you buy it

Three principles keep the test honest. First, your claims, not demo claims: the drift that matters is the drift on your book’s shapes — the three-charge-two-card claim, the new-account claim, the claim where the merchant answers with a tracking number. Second, your decisions as the answer key: what your analysts decided and signed at the time is the only standard that exists; the pilot measures agreement with it, not with the vendor’s expectations. Third, measured in advance: the pass bars are written down before the replay runs, so the verdict at the end is arithmetic, not a debrief. A vendor who resists any of the three is telling you something useful.

Start on claims you have already closed

Closed claims are the perfect test corpus: the notice, the evidence as it stood, the decision, the letter and the clocks are all final, and nobody’s money or deadline moves if the system disagrees with the file. The replay is also the only test of the parts a live pilot cannot touch safely — the standing rules. On the sample claim, the rule under test is the one the operations manager approved: post Regulation E provisional credit one business day before the Day 10 line. Replay it against a book that includes a federal holiday and the question answers itself — did the credit stage on the right business day, through the calendar, every time? That is a question no sales demo can answer and a replay answers in an afternoon. The legal frame underneath is unchanged: the system is rehearsing a duty that remains the bank’s, to investigate promptly and determine whether an error occurred within 10 business days (12 CFR 1005.11).

Which claims to replay: a representative quarter, not the easy ones

Sixty to a hundred and twenty claims is enough, chosen the way an examiner chooses: a quarter of the book, stratified so every shape the desk actually sees is in the sample. Include the multi-charge claims, the ones that crossed a holiday, the ones where a merchant replied and the ones where nobody did, the denials with a second signature and the small claims credited without a chargeback. Resist two temptations: cherry-picking the clean claims, which manufactures agreement, and seeding only the horror stories, which manufactures disagreement. What the pilot is measuring is the ordinary Tuesday — because that is what the book is mostly made of.

Scoring agreement between the agents and your own analysts

Score per transaction, not per claim — a claim with three charges can agree on two and drift on one, and the drift is the finding. For each replayed transaction the comparison has four cells: your analyst credited and the system recommends credit (agreement); both deny (agreement, and check the reasons match); your analyst credited and the system recommends denial (a would-have-been complaint — read it first); your analyst denied and the system recommends credit (a would-have-been loss or a would-have-been wrong — read it second). Count the first two against the total and you have the agreement rate. Then read every cell three and four individually, because the disagreements are the entire value of the pilot: they are the list of what the system sees that your desk does not, and vice versa.

When the agents disagree with the old file — and when the old file was wrong

A disagreement is not automatically the system’s error, and treating it as one is how pilots lie. Two of the three disagreement cells can mean the old file was wrong: the analyst who credited a charge the evidence did not support, or the denial that rested on a merchant’s reply nobody checked. The enforcement record shows which direction regulators punish: the CFPB’s 2019 order against USAA Federal Savings Bank — approximately $12 million in restitution and a $3.5 million penalty — faulted the bank for failing to initiate and complete reasonable error resolution investigations, not for being too generous (CFPB enforcement action, January 3, 2019). So the review of each disagreement asks two questions symmetrically: what record supports the system’s recommendation, and what record supported the original decision? Where the old file cites nothing, the disagreement is a finding about the old file. Where both sides cite records, the pilot has surfaced a real judgment call — and those are the cases to hand to your best analyst, because they are the cases the desk has always decided hardest.

How long a pilot runs: replay, then one live week

Four weeks in the shape above, and two reasons not to stretch it. The replay answers the load-bearing questions — agreement, explanations, the standing rules — inside the first fortnight, and a pilot that runs for months stops being a test and starts being an undocumented deployment. The live week beside the team is the second half because some things only show up live: how the queue feels at 08:40 on a Monday, whether the escalation a day before a clock expires reaches a named person, and how much analyst time the review actually takes. That last number is the one to hold the vendor to afterwards. Plan the borrowed hours deliberately — the people whose judgment you are borrowing are fraud examiners and investigators, part of a BLS occupational group of about 137,800 people nationally with some 9,400 openings projected a year to 2035 (O*NET OnLine, BLS 2025 data) — which is another way of saying their hours are scarce, and the pilot is partly a test of how many of them the system gives back.

Live beside the team: the agents’ run on the morning’s claims, waiting on the analysts’ signatures

What counts as a pass

The pass bars belong on paper before the first claim is replayed, and they belong to you, not to the vendor. The shape that works: an agreement threshold with the answer key, a completeness rule that every disagreement carries cited records on both sides, an accountability rule that nothing moves without a person’s signature, and a clock rule that passes at 100% or not at all. The accountability rule has a framework behind it: the National Institute of Standards and Technology’s AI risk framework makes it a category of its own, Govern 2: “Accountability structures are in place so that the appropriate teams and individuals are empowered, responsible, and trained for mapping, measuring, and managing AI risks” (NIST AI RMF 1.0, Govern). In a dispute desk that structure already has names: the analyst who signs, the second signature above the limit, the operations manager who owns the standing rules. The pilot passes only if the system strengthens that structure rather than routing around it.

What data you need to start a pilot

Less than the vendor’s integration questionnaire suggests, and all of it you already keep. The pilot needs the notices as they arrived — call notes, app forms, emails, scanned letters; the authorization log for the disputed charges; the merchant replies and representments as they stood; the account activity and dispute history the analysts actually saw; the decisions, letters and signatures as filed; and the clock log. Anonymise for the replay if your policy asks it; keep the dates real, because the calendar is half the test. If a vendor’s pilot cannot start on an export of those, the pilot is an integration project in disguise — which is a reason to negotiate, not to quit, but know which you are signing.

The risk of going straight live

Skipping the pilot does not skip the test; it just runs the test on customers, on clocks, and without an answer key. The failure modes are quiet ones: standing rules that fire on the wrong business day through a holiday; agreement that was real on the demo book and is not on yours; recommendations that read well and cite nothing. Each is invisible until the first examiner sample or the first complaint letter, and each was discoverable in a two-week replay. The comparison below is the honest scope of what a pilot does and does not prove — and why the go decision belongs after it.

The same comparison, priced, is in what a dispute costs and the ROI of automating it, and the questions to put to any vendor — before the pilot, not after — in what to evaluate before you buy.

Questions people ask

Can I pilot dispute automation on old, already-closed claims?

Yes — and it is the safest first step. Closed claims give you a final answer key: the evidence as it stood, the decisions your analysts signed, the letters that went out. Nothing moves if the system disagrees, and the standing rules can be rehearsed against the real calendar, holidays included.

How long should a dispute software pilot run?

About four weeks: two to replay and score sixty to a hundred and twenty closed claims, one live week beside the team where analysts decide as usual, and a final week for the verdict against the pass bars. Longer pilots stop being tests and become undocumented deployments.

What counts as a pass in a dispute automation pilot?

Bars agreed before the pilot starts: an agreement threshold with your signed decisions per transaction, every disagreement carrying cited records on both sides, zero sends or filings without a signature, and a 100% clock record in the live week. Set the numbers yourself and write them into the pilot plan.

How do you score agreement between the agents and your own analysts?

Per transaction, in four cells: both credit, both deny, the system denies where your analyst credited, and the system credits where your analyst denied. The first two over the total is the agreement rate; the last two are read individually, because they are the list of what each side sees that the other does not.

What data do I need to start a pilot?

The notices as they arrived, the authorization log, the merchant replies as they stood, the account activity and dispute history the analysts saw, the decisions and letters as filed, the clock log, the standing rules, and the holiday calendar — eight exports, none exotic, all kept already. Keep the dates real; the calendar is half the test.

What’s the risk of skipping a pilot and going straight live?

That the test still happens, but on customers and live clocks: standing rules firing on the wrong business day, agreement that held on demo data and not on your book, and recommendations that cite nothing. Each is invisible until an examiner sample or a complaint letter — and each was discoverable in a two-week replay.