An LLM in trade finance can be trusted with a rules answer only when five things can be measured on every answer: each sentence matches the passage it cites, the passage comes from the edition in force, the conditions in the rule survive into the answer, superseded text is never used for a new answer, and the model says "not settled" instead of guessing when the rule set is unknown or the sources disagree. A single accuracy percentage hides all five. Trade operations are a hard test for generative AI because the rules are private, partly licensed and versioned, and because a refusal under UCP 600 must be right inside five banking days. Botminds' UCP 600 & ICC Rules Search uses agents to research and cite the rules, practice and the bank's own rulings; this guide sets out how to test it, or any other tool, on the bank's own questions.
The research and rule texts are linked where they appear. The examples of the solution come from a fictional sample bank, Corvane National Bank, and its sample data; its figures are not a customer's measured results.
Five tests for an LLM rules answer in trade finance
- 1. Every sentence matches its passage
Score sentence by sentence: does the cited passage say this? An answer with one unsupported sentence fails, however right the verdict.
- 2. The passage is from the edition in force
UCP 600, ISBP 821 (not 745), eUCP 2.1, and the bank’s current procedure version. Seed old-edition traps into the test set.
- 3. The rule’s conditions survive
“Discrepant” is wrong if the rule says “unless the document shows cover from an earlier date”. Score the qualifier, not just the verdict.
- 4. Superseded text is never the basis of a new answer
It may be shown, marked as superseded. Count any answer that rests on it as a failure.
- 5. It says “not settled” when it should
When the credit names no rules, or rule, practice and opinions disagree, the right output is a question or a referral, not a verdict.
Can an LLM answer UCP 600 questions?
Yes, the common ones, because the answers sit in a short public text: UCP 600's 39 articles are served free on ICC's Digital Library. The difficulty is everything around the text: whether the credit applies it, which edition, what the credit's own terms change, the standard practice and opinions that are sold rather than published, and the questions on which the sources disagree.
A general-purpose chat assistant meets each of those difficulties with the same fluency it uses for the easy part. It may answer a question about an electronic presentation from UCP 600 alone, when an eUCP credit carries the UCP with it but the eUCP wins wherever the two differ, as eUCP version 2.1 explains. It may quote standard practice it has seen second-hand, since ISBP 821 is sold by ICC. Neither error looks like an error on the page.
The ICC has its own reason to worry. The ICC Digital Library reports that the Banking Commission's 2026–27 plan aims "to counter the growing volume of inaccurate interpretations and unofficial guidance circulating online and across social media platforms" (ICC Digital Library, August 2026), with the caveat that the article is the author's view. An LLM trained on that material learns the inaccuracies too.
What the research says about AI in letter of credit work
The research says AI helps in letter of credit work when a human checker stays in the loop, and that nobody has yet published a rules-answer benchmark a bank could borrow. That is a reason to test on your own files, not a reason to wait.
What published research can and cannot tell a bank about LLM accuracy here
91%
peak precision of LLM document-eligibility checks at the Bundesbank, on securities prospectuses, not credits
Hybrid
the AI-human model peer-reviewed LC research proposes; full automation “remains impractical”
None
regulator benchmarks we found for AI accuracy on letter of credit rules
Peer-reviewed work on letter of credit examination finds the work "remains complex, labor-intensive, and error-prone", and that integrating AI "requires human checkers to verify AI-generated decisions" (Khalil et al., Decision Analytics Journal, 2025). A companion paper concludes that "full automation remains impractical due to regulatory and trust concerns" and proposes "a hybrid AI-human approach" (Khalil et al., Digital Business, 2025).
The nearest published accuracy figure comes from a central bank, on a different document type. Deutsche Bundesbank researchers report precision of up to 91% for LLM-based eligibility checks of securities prospectuses, from a system that leaned towards rejecting rather than wrongly accepting (FNP Workshop, 2026); how to test trade finance document automation quotes the finding and asks what it means for credits. The useful lesson is not the number but the profile: a system tuned to avoid false acceptance, measured at the level of the document's decision.
US examiners already expect technology in trade operations. The OCC's handbook lists electronic documents, discrepancy detection and name screening against sanctions lists among the jobs trade systems already do (OCC, Trade Finance and Services), a passage trade finance automation software compared quotes in full. What it does not do is tell a bank how to measure a generative answer; that is the bank's own job.
Should a rules assistant decide compliance? The 2019 statement, answered
No, and a rules assistant should not claim to. The Wolfsberg Group, ICC and BAFT wrote that some elements of the process can be automated, such as screening against sanctions lists, but that reviewing trade documents as a whole cannot be, by its nature, and that controls will combine the automated and the manual (Trade Finance Principles, 2019 amendment). Their exact words, and what they mean for a financial crime alert, are in how technology can assist in combating TBML.
The statement sits in the principles' section on financial crime controls, where banks review each transaction for fraud, sanctions and suspicious activity as well as for the ICC rules. Its honest reading draws a line between two kinds of work. Researching what the rule, practice and the bank's rulings say, citing them and drafting wording is an element that can be automated and checked. Deciding whether this presentation complies, with these documents and this credit, is the review the statement describes, and it stays with an examiner. UCP 600 & ICC Rules Search sits on the first side of that line by design: the agents research, write and check; they never pay or refuse a presentation, send a notice or approve a ruling.
How do you test an AI rules answer? Build the test from your own questions
Test it on a set of your own past rule questions with known answers, scored on the five tests separately, before it answers a live question. A vendor's headline accuracy, measured on someone else's questions, says little about your credits, your procedures or your grey areas.
Three sources make a good test set. Past questions your senior checkers answered, with the answer they gave and the article they relied on. The bank's own rulings, each turned into two or three questions whose right answer depends on the ruling, not the rule. And traps: a question on a credit that names no rules, one whose answer changed between ISBP 745 and ISBP 821, one where the credit's terms override the article, and one where the sources genuinely disagree, whose only right answer is "not settled".
At the sample bank, the verdict writer's current prompt scores 97.4% on a 420-question evaluation set, and any change to a prompt or model needs the Rules Desk lead's approval before it answers anyone. That figure is sample data and says nothing about another bank's questions; the point is that the number exists, is versioned, and is tied to who approved it.
Building a rules-answer test set: what to include, and what each part catches
| Part of the set | Where it comes from | Catches | Pass mark |
|---|---|---|---|
| Settled past questions | Senior checkers’ answers, with the article | Plain rule errors, missing citations | Verdict and citation both right |
| Bank-ruling questions | Each approved ruling, as 2–3 questions | Answers that ignore the bank’s own position | Names the ruling, its version and approver |
| Edition traps | Points that changed from ISBP 745 to 821, or eUCP 1.1 to 2.1 | Superseded text used as current | Current edition cited; old one marked superseded |
| Credit-term traps | Credits that modify an article (art. 1) | Rule answers that ignore the credit | Asks for or reads the credit’s terms |
| No-rules traps | A credit silent on which rules apply | A guessed rule set | Asks which rules apply |
| Genuine disagreements | Questions your desk ruled on after debate | False confidence | “Not settled”, with each source shown |
What is the risk of a superseded edition?
The risk is an answer that is correct for a text no longer in force, cited confidently to a page that still exists. ICC's own publishing shows how easily it happens. ISBP 821, published on 3 July 2023, replaced ISBP 745 and aligned practice with opinions approved from October 2013 to January 2023 (ICC Documentary Credit World, July 2023). eUCP Version 2.1 was published on 29 June 2023 (ICC).
Yet in October 2026, ICC's free Digital Library page for the eUCP still displayed a text that begins "This version is Version 1.1" (ICC Digital Library, eUCP). A model trained on the open web, or a search tool that indexes it, can quote that page as the eUCP. The difference matters: Version 2.1 added a definition of an electronic transferable record and aligned the rules with the UNCITRAL Model Law.
Opinions carry the same risk in a quieter form. When ICC published its 1995–2001 collection, it said opinions overtaken by later ones are marked as such in the text (ICC, 5 July 2002), a status the guide to ICC Banking Commission opinions explains. A tool that retrieves opinions by similarity, without their status, can surface the cancelled one. Test two in the checklist exists for this reason.
Should an AI cite the article? The citation is the unit of accuracy
Yes, and to the passage, not just the article. An answer that cites "UCP 600 art. 28" for a point that turns on 28(e)'s exception has cited correctly and still lost the condition that decides the case. The test is sentence by sentence: can each sentence be found, in substance, in the passage it cites?
A fluent answer and a checked answer to the same insurance-date question
Fluent, uncited: “Yes, the documents are discrepant.”
Cited and checked: “Yes, discrepant, unless the certificate shows cover effective no later than the shipment date.”
Fluent, uncited: Refers to “UCP rules on insurance”.
Cited and checked: UCP 600 art. 28(e), with the passage opened and the edition shown.
Fluent, uncited: Blended into the rule.
Cited and checked: Labelled separately from the rule and from the bank’s procedure.
Fluent, uncited: Absent.
Cited and checked: The bank’s ruling, its version and its approver.
Fluent, uncited: Unknown.
Cited and checked: Each sentence checked against its passage; any removed sentence written to the audit trail.
UCP 600 & ICC Rules Search opens the passage behind each citation, with licence controls
The bank owns every AI rules answer, whoever built the tool
Accountability stays with the bank, whoever built the tool. The Wolfsberg Group's 2022 principles on AI in financial crime compliance hold a bank responsible for its use of AI, and for decisions that rely on it, whether it built the system or bought it (Wolfsberg Group, 2022; quoted in AI adverse media screening). The same principles ask that results can be explained from the data that went in, and for a programme of ongoing testing, validation and re-configuration, which the model risk view of a TBML tool applies to a second-line review.
Those principles were written for financial crime compliance, not document examination, but they translate directly. Explainable means a citation per sentence and the version of every source. Ongoing testing means re-running the test set whenever a prompt, a model or a source changes. And accountability means a named person behind every decision the answer feeds: in the sample bank, the examiner who adds a discrepancy cannot confirm it, and the author of a ruling cannot approve it.
The trail an auditor reads: question, verdict, prompt version, citation check, decision
What should you ask an LLM vendor in trade finance?
Ask how the five tests are measured, how ICC's licence is respected, and what the tool will refuse to do. A vendor who answers with a single accuracy figure has not measured the things that cause a wrong refusal.
Ten questions to put to any LLM vendor answering ICC rules questions
| Ask | A good answer shows |
|---|---|
| Which texts does it answer from, and which editions? | A source list with versions: UCP 600, ISBP 821, eUCP 2.1, your licensed opinions, your procedures. |
| How is ICC’s licensed text handled? | Entitled users only, an excerpt limit, references only in exports. ISBP 821 and opinions are sold, not free. |
| What happens when the credit names no rules? | It asks; it does not assume UCP 600. |
| How is each sentence checked against its source? | A sentence-level check, with removals logged. |
| How are superseded texts marked? | Shown as superseded, never used for a new answer. |
| What does it do below its confidence threshold? | “Not settled”, routed to a named desk with the conflicting passages. |
| Can it apply our own rulings, and who approves them? | Rulings versioned, with author and approver kept apart. |
| What can it never do? | Decide compliance, send a notice, approve a ruling, reply to a client unchecked. |
| How do we test it on our own questions? | A replay on your past questions and rulings, scored on each test separately. |
| What does an auditor see? | Question, sources and versions, verdict, prompt version, who decided and confirmed. |
Where UCP 600 & ICC Rules Search fits, and where it does not
It fits where a trade operation spends its senior checkers' time on rule research. Eight agents read the question and the credit's rules, search the rules, practice, licensed opinions and the bank's rulings by passage, write a verdict with rule, practice and bank procedure labelled apart, check every sentence, look for the deciding condition on the presented document, draft the approved refusal wording, and route what is not settled to the Trade Rules Desk.
It does not examine a whole presentation; that is document checking, and Botminds has a separate Letter of Credit Document Checking solution for it. It does not replace ISBP 821 or the bank's licence for ICC opinions; it enforces them. And it never decides. In the sample bank's first three weeks of October, examiners overrode 1.8% of verdicts, each with a reason code, which is what a tool that leaves the decision with people should produce. The design target is that every answer carries its passage, its source version and any ruling's approver. How the clock turns a slow rules answer into a real cost is set out in UCP 600's five banking days.
Questions people ask
How accurate is ChatGPT on letter of credit rules?
No regulator or standard-setter has published a benchmark, so any figure would be anecdote. The risks are specific: answers from superseded texts, from second-hand summaries of ISBP practice, or without the credit's own terms. Test any assistant on your own past questions, scored sentence by sentence against the passages.
Who is responsible if an AI rules answer is wrong?
The bank. The Wolfsberg Group's 2022 principles say a bank is responsible for decisions that rely on AI analysis whether it built the system or bought it, and ask for ongoing testing and validation. Version the prompts, models and sources, and keep a named approver for each change.
Can AI decide whether documents comply with a letter of credit?
It should not. The Wolfsberg Group, ICC and BAFT said in 2019 that the review of trade documents as a whole is not automatable, and peer-reviewed LC research proposes a hybrid in which human checkers verify AI output. An AI can research, cite and draft; the examiner decides.
Can a bank put ISBP 821 into an AI tool?
Only within its licence from ICC, which sells ISBP 821 and the Banking Commission opinions. A well-built tool shows licensed text only to entitled users, within an excerpt limit, and puts references rather than text into exports. UCP 600 itself is served free on ICC's Digital Library.
What is a good confidence threshold for a rules answer?
There is no industry standard; set one and test it. The fictional sample bank uses 0.85: below it, or when sources disagree, the answer is "not settled" and goes to the Trade Rules Desk. Measure how often the desk agrees with the threshold's calls and adjust with an approver's sign-off.