To test trade finance document automation, replay it on presentations your bank has already decided: last quarter's refusals, waived discrepancies and clean payments, with your examiners' final calls as the answer key. Score what the system finds against those calls one discrepancy type at a time, count its misses separately from its false alarms, and send every disagreement to a senior examiner before you believe either side. The test matters because no regulator or researcher publishes an accuracy benchmark for letter of credit checking, and because a demonstration on a vendor's tidy samples says nothing about your presentations, your credits and your examiners' practice. Botminds' Letter of Credit Document Checking is built to be judged on that replay, and to run it again whenever a model or a prompt changes.
The research figures below are from a measured study of 500 US letter of credit files and from published peer-reviewed work, linked where they appear. The product examples come from a fictional sample bank, Corvane National Bank, as it runs through the solution.
A 91% score can hide a system that never catches one rule
How do you test trade finance document automation?
You test it the way an examiner would audit a colleague: give it presentations whose outcome you already know and compare its findings with the decisions your bank made. In practice that is five steps. Pull a quarter's closed presentations, including the credit and every amendment as they stood on the day of presentation. Record the final decision on each: complied, discrepant and waived, discrepant and refused. Run the system on the documents alone, without the decisions. Compare finding by finding. Then have a senior examiner adjudicate every case where the system and your file disagree, because sometimes the file is the one that is wrong.
The last step is the one most tests skip. Your examiners' decisions are the best answer key you have, but they are not ground truth: they were made under a five-day clock, and in a waived file nobody had a reason to argue about whether each discrepancy was real. Mann's study faced the same problem and was candid about it: "I relied entirely on the banks' internal documentation of those issues" (Mann, 2000). A test that treats every disagreement as a system error will under-score a system that is right, and one that treats every disagreement as a file error will flatter it.
Why your closed files are the gold set
A gold set is a collection of cases with agreed answers, against which a system is scored. Closed presentations make the best one for three reasons. They carry your credits, with your amendments, your 47A habits and your customers' document styles. They carry decisions made by your examiners under your procedures, which is the standard the system will be held to after go-live. And they carry consequences: which discrepancies were waived, which led to a refusal, which payment was released on which day.
They also carry the mix. The only measured breakdown of discrepancy types in the public record is Mann's: 554 discrepancies in the 343 discrepant files examined, collected at five US banks in 1999 under the previous rules, UCP 500. It shows why a test built from a vendor's samples misleads. More than four in five discrepant files had a defective document, the vaguest category and the one where judgment matters most; the objective failures, such as late shipment and late presentation, were each under a fifth. A system tuned on crisp date errors can look excellent and still struggle on the category that fills most of your queue.
How much of a real test set will be discrepant: import files more than export
All 500 files
- Discrepant · 73%
- Conformed · 27%
Import side: 253 files
- Discrepant · 77%
- Conformed · 23%
Export side: 247 files
- Discrepant · 68%
- Conformed · 32%
The split matters for test design. An issuing bank's import desk should expect most of its test presentations to be discrepant, so a test set needs enough clean presentations, deliberately added, to measure false alarms. The variation across banks matters too: conformity ran from 17% at one bank to 36% at another, so a test on another bank's files tells you little about yours.
Score by discrepancy type, not one accuracy number
A single accuracy figure hides the failure that hurts. Take Mann's 554 discrepancies as a test set, as the chart at the top of this guide does: a system that found every one except the 48 late presentations would score 506 of 554, or 91%, while being blind to an entire rule. The scoring sheet below keeps each discrepancy type on its own line, and keeps the two kinds of error apart.
A scoring sheet for a replay, one row per discrepancy type
| Measure | How to count it | Why it matters |
|---|---|---|
| Found | Discrepancies the system raised that the final, adjudicated decision also holds | The share of real discrepancies caught, per type. The number a misses-averse desk cares about most. |
| Missed | Discrepancies in the final decision that the system did not raise | Each miss is a payment the bank might have made against documents that did not comply. |
| False calls | Findings the system raised that the adjudicated decision rejects | Each one costs examiner time, and a false call that reaches a notice is a wrongful refusal. |
| Sent to a person | Lanes the system routed for a human decision instead of calling them | Honest routing is a strength, not a failure, until it routes so much that nothing is saved. |
| Cited correctly | Findings whose page, line and article match what the examiner relied on | A right answer on the wrong page does not survive a senior checker or an auditor. |
| Clock | Banking days from presentation to a decision-ready file | Five banking days is a maximum. Time saved early is time for a waiver later. |
The 91% example is arithmetic on a 1999 mix, not a claim about any system. Its point is that the per-type rows are what a buyer should read. If late presentation is 14% of your discrepant files and the system misses it, you will not see that in one overall score; you will see it the day an applicant declines to reimburse a payment made against a late presentation.
Should you test on refused or waived presentations?
Test on both, and on clean presentations as well, because each answers a different question. Refused presentations are rare: in Mann's 500 files the applicant waived and allowed full payment in every discrepant file but one, and one bank estimated three refusals a year out of 10,000 presentations. A test built only from refusals would be tiny and skewed toward the most serious defects.
Waived presentations are where the discrepancy calls live. Each one carries an examiner's list of discrepancies, sent to the applicant and accepted, which is a reasonably reliable answer key for what was found. Clean, complying presentations are the only way to measure false calls: a system that raises a discrepancy on one of them is proposing a refusal the bank would have had to defend. Why applicants waive almost every time explains what the waiver data mean for the desk.
Count the misses separately from the false calls
The two errors cost different amounts and land on different people, so one combined error rate is the wrong number. A missed discrepancy means the bank honored a presentation that did not comply; its right to reimbursement from the applicant is then open to challenge, and in practice rests on the applicant's willingness to waive. A false call means the bank may refuse a complying presentation, and a bank that wrongfully dishonors owes the amount and can face litigation costs. What a missed discrepancy costs sets the two side by side.
A vendor demonstration and a replay on your files answer different questions
Vendor demonstration: Prepared samples, often clean scans with one planted error each.
Replay on last quarter's files: Your scans as they arrived, including faint stamps, handwritten notations and missing pages.
Vendor demonstration: A generic MT700 with standard fields.
Replay on last quarter's files: Your credits as amended, with your 46A wording and your 47A conditions.
Vendor demonstration: Whatever the vendor planted.
Replay on last quarter's files: Your examiners' decisions, adjudicated where the system disagrees.
Vendor demonstration: Chosen to show the product well.
Replay on last quarter's files: Whatever your applicants and beneficiaries actually got wrong.
Vendor demonstration: Whether the system can work.
Replay on last quarter's files: Whether it works here, for which discrepancy types, and how often it routes to a person.
How accurate is AI document checking for letters of credit?
Nobody can tell you from public evidence. No regulator has published an accuracy benchmark for AI in letter of credit examination, and the peer-reviewed work that exists measures something else. A 2025 study of AI-assisted LC examination modelled how to assign AI-flagged documents to human checkers, and reported risk reductions for that assignment model, not detection accuracy (Khalil et al., Decision Analytics Journal, 2025).
The closest published measurement is an analogy. A 2026 Bundesbank study applied large language models to examining securities prospectuses for collateral eligibility and reported "high precision (up to 91%) in document-level eligibility, exhibiting a conservative operating profile that minimizes false acceptance" (Hamotskyi, Gautam and Hänig, FNP 2026). Prospectuses are not letters of credit, so the number does not transfer. What does transfer is the shape of the result: a system designed to lean toward caution, where errors fall on the side of sending a case to a person. That is the profile a trade desk should test for, and the one the solution's 0.90 confidence threshold is built around.
How many past presentations do you need for a test?
Enough of each discrepancy type to see a miss if there is one, which is more than most pilots use. A plain statistical rule helps: if a system makes no misses in n cases, the true miss rate could still be as high as about 3 in n at 95% confidence. To show a miss rate below 5% on a type, you need about 60 cases of that type with no misses.
Presentations needed to see 60 cases of each type, on the 1999 mix
| Discrepancy type | Share of discrepant files | Discrepant files needed | Presentations needed at 73% discrepant |
|---|---|---|---|
| Defective documents | 85% | about 71 | about 97 |
| Missing documents | 22% | about 273 | about 374 |
| Late shipment | 18% | about 333 | about 457 |
| Late presentation | 14% | about 429 | about 587 |
| Credit expired | 11% | about 546 | about 747 |
The table shows why a pilot of 50 presentations proves little about the rarer types. A quarter's closed files at a mid-sized desk, the sample bank's 1,920 presentations for instance, are enough for the common types; the rare ones may need a year, or a deliberately enriched set in which known discrepancies of those types are overrepresented and reported as such.
Re-run the test whenever a model or prompt changes
A test result describes one version of a system, so it has to be repeated whenever that version changes. In the solution the models and prompts behind each agent are pinned, and a new version runs against last quarter's presentations first; a change is written to the audit trail with the person who made it.
The discipline is the bank's own. The federal banking agencies' revised model risk guidance of April 2026 replaced SR 11-7, but it places generative and agentic AI models outside its scope (interagency Revised Guidance on Model Risk Management), and the agencies have said they plan a request for information on AI (OCC Bulletin 2026-13). Until there is supervisory guidance specific to these systems, the replay is how a bank shows itself, its auditors and its examiners that a change was tested before it reached a live presentation. Agentic AI for trade finance covers the governance side in full.
The thresholds, the calendar and the pinned versions a test is run against
What should a trade finance automation pilot report?
A pilot report should let a head of trade operations, an auditor and an examiner each find their answer without asking the vendor. The checklist below is what we would expect to see in one, whoever's system it tests.
What a replay report on closed presentations should contain
- The test set, described
How many presentations, from which period, how many complying, waived and refused, and whether any type was enriched.
- Found, missed and false calls per discrepancy type
In your own taxonomy, with clean presentations as their own row.
- Every disagreement, adjudicated
Who decided, on what ground, and how many times the file rather than the system turned out to be wrong.
- The routing rate
How many lanes went to a person, by type. Too few is overconfidence; too many is no saving.
- Citation accuracy
Whether each finding pointed to the right page, line and article, checked by a senior examiner on a sample.
- Time to a decision-ready file
In banking days and in examiner minutes, on the same presentations, by hand and with the system.
- The versions tested
Model, prompt and threshold settings, so the test can be repeated after a change.
- What the system never did
Confirmation from the trail that no notice was sent, no payment released and no screening hit cleared without a named person.
Every agent step and every human decision on one trail, which is what a replay is scored from
The design choice that makes the solution testable is that it never mixes the two. Each examiner agent's findings, the disagreement between them and the examiner's decision are kept as separate entries, so a replay can score the agents against the person without reconstructing anything. To see that on a sample of your own past presentations, start from Letter of Credit Document Checking.
Questions people ask
What is a gold set in trade finance testing?
A gold set is a collection of past presentations with agreed outcomes, used to score a system before and after it goes live. For letter of credit checking it should include complying, waived and refused presentations, the credits as they stood on each day of presentation, and the final decision on each discrepancy after disagreements are adjudicated.
What error rate should we expect from LC automation?
There is no published benchmark, so any number offered without your own files behind it is a sales figure. Expect the rate to differ sharply by discrepancy type: objective date and amount checks are easier to automate than judging whether a document's wording fulfils its function. Set acceptance criteria per type before the test, not after.
Is a confidence score the same as accuracy?
No. A confidence score is the system's own estimate on one finding; accuracy is measured against your decisions across many. Confidence is useful for routing, such as sending everything below 0.90 to a person, and a replay tells you whether the threshold is set in the right place: if findings above it are often wrong, raise it.
Who should adjudicate disagreements in a test?
A senior examiner who did not make the original decision, working from the documents and the credit alone, the same standard art. 14(a) sets. Record how often the file turns out to be wrong; that number tells you as much about your desk as about the system.
Can we test on presentations from another bank?
You can, but the result transfers poorly. In Mann's study conformity ranged from 17% to 36% across five banks, and Mann found the identity of the bank had significant explanatory power for discrepancies. Credits, customers and examination habits differ, and the test should reflect the desk the system will serve.