A credit support annex automation tool, the AI that reads each annex and turns its terms into data, should be judged on your own agreements, because accuracy on general contracts says little about ISDA credit support annexes, and no regulator or independent study publishes accuracy for collateral terms. The test that settles it is a replay: take agreements your team has already abstracted, run the tool on the executed documents without showing it your answers, compare every extracted term with your abstract and with what your collateral system holds, and have a lawyer read the clause wherever they differ. Scored field by field, the replay shows how often the tool is wrong, on which terms, and how often your existing abstracts were wrong too. Botminds' ISDA and CSA Data Extraction runs exactly this back-test at every new firm before anything reaches a system, and keeps a fixed set of hand-checked agreements to test every new version of its agents.
A clean result on a small test proves little: the most an error-free replay rules out
The examples come from a fictional sample firm, Quillfield Capital, with 2,400 executed ISDA, CSA, GMRA and GMSLA agreements, as it runs through the solution. Research is linked where it appears; preprints are labelled as not peer reviewed.
How accurate is a credit support annex automation tool? Nobody can tell you for your CSAs
No one can quote an accuracy figure that applies to your annexes, and a vendor that does is quoting a different test. Accuracy depends on the documents: their forms and governing laws, their scan quality, the languages of local-law annexes, and above all how many amendments and protocol letters sit on top of each base agreement. A tool that reads a clean 2016 variation margin annex perfectly can still return the wrong threshold for an agreement amended three times, because the error is not in the reading but in which document governs.
That is why a review tool's headline number is the wrong thing to compare. The questions that matter are per field (how often is the threshold wrong, the rounding direction, the eligible collateral list), per document type, and as of a date (did it return the term in force, or the term in the base annex). A broader comparison of AI tools for searching and answering questions over derivatives documents is in legal document search AI tools and their accuracy; this guide is about testing extraction of collateral terms.
What does research say about expert-labelled contract datasets?
It says reliable answers on contracts are expensive to produce and models read them unevenly. CUAD needed dozens of legal experts to create more than 13,000 annotations for contract review, and found model performance at an early stage (Hendrycks et al., NeurIPS 2021). MAUD, built on merger agreements, assembled over 39,000 examples and found models well above chance on most questions but with much room for improvement on a large subset (Wang et al., EMNLP 2023). ContractNLI found existing models struggled to tell whether a contract supports, contradicts or is silent on a statement (Findings of EMNLP 2021).
LegalBench, a benchmark of 162 legal reasoning tasks designed by legal professionals, evaluated 20 language models and set out to give lawyers and developers a shared vocabulary for what kind of legal reasoning a task requires (Guha et al., arXiv 2308.11462, not peer reviewed in this version). Two lessons carry over to a credit support annex. A good test needs an answer key that experts have checked, and the result depends on the task: finding the clause is different from extracting the number, which is different again from deciding which of two documents governs.
Why a replay on CSAs you already abstracted is the test that counts
Because you already own the answer key, the documents and the stakes. Your abstracts were made by people who knew the agreements, on the forms and languages you actually hold, and your collateral system shows which values are in use. A replay measures the tool on that, and it measures your current process at the same time: every mismatch is either the tool's error or the abstract's, and finding the second kind is often the most valuable result of the test.
The replay also tests what a demonstration cannot. It shows whether the tool applies amendments in effective-date order, whether it stops on conflicting documents or picks one, whether it reads a German or French annex in the original, and whether each value carries a citation a lawyer can check in seconds. The design that makes those checks routine rather than a one-off is described in a maker-checker design for AI on legal agreements.
How do you design the replay?
Agree the rules before the tool runs, so that nothing is adjusted to fit the result. The six steps below are the design we use and recommend.
A replay test in six steps: blind, field by field, settled against the page
1
Choose the agreements
Fifty or more already abstracted by your team, spread across forms, governing laws, languages and amendment-heavy relationships, not the cleanest fifty.
2
Freeze the answer key
Export the abstracts and the values your collateral system holds today, before the tool runs, so nothing is adjusted after the fact.
3
Run the tool blind
Give it the executed documents and every amendment, nothing else: not your abstracts, not your system values.
4
Compare field by field
Every term, every party, every currency, on the scoring rules agreed in advance.
5
Adjudicate against the page
For each mismatch a lawyer reads the clause and records who was wrong: the tool, the abstract, or both because the clause is ambiguous.
6
Report rates with bounds
Per field, with how many terms were checked, so a clean result on a small sample is not read as proof.
Two choices decide whether the result means anything. The sample must include the hard agreements: long amendment chains, protocol letters filed in the wrong folder, local-law annexes, rating-linked thresholds. And the adjudication must be done by someone who did not make the original abstract, for the same reason the checker in a maker-checker design must not see the maker's answer.
Which fields should be scored, and how strictly?
Every field that a system computes with, each on a rule written in advance. A threshold is either right or wrong, including its currency and the party it belongs to; a notification time is wrong if the city is wrong; an eligible collateral schedule is scored row by row, not as one field.
Scoring rules for a CSA replay, field by field
| Field | Scored as correct when | Common partial errors to count as wrong |
|---|---|---|
| Threshold | Amount, currency, party and any rating table all match the term in force | Base-annex value when an amendment changed it; one value for a rating-linked grid |
| Minimum transfer amount | Amount per party, and whether it is split between initial and variation margin | Treated as one combined figure when the agreement splits it |
| Independent amount | Amount or formula, trigger and cap | Trigger omitted, cap omitted |
| Rounding | Multiple and direction, for delivery and return separately | Direction missing; one rule applied to both |
| Eligible collateral and haircuts | Each row: asset, maturity band, currency, percentage, exclusions | Paragraph captured as text; exclusions added by amendment missed |
| Valuation agent and timing | Party or rule; valuation dates; notification time with its city | Time without city; “dealer” assumed |
| Interest on cash | Index, spread, day count, floor, payment dates | Index updated by amendment or protocol not applied |
| Mechanism and governing law | Transfer or security; law of the master and of the annex | Form identified, mechanism not |
| Term in force | The value as of the test date, with the document that set it | Right value, wrong source document |
What accuracy do you need on thresholds and MTAs?
None of the errors can reach a system, which is a different requirement from a target accuracy. A threshold read wrong by one amendment can mean a call of millions not made: in the sample estate, a dealer threshold cut to zero in 2023 but still held as USD 10 million in the collateral system left a $6,820,000 delivery amount uncalled on a single Friday. No accuracy rate makes that acceptable, so the test should not ask whether the tool's error rate is low enough to trust; it should ask whether its errors are caught before a system uses them.
That changes what to measure. Count the tool's errors, and then count how many of them its own controls caught: a disagreement between readers, a confidence below the re-read level, a conflict between documents. An error caught by the controls costs a reviewer a few minutes. An error that passes them is the one that matters, and the replay should report it separately.
Score your hand abstraction too
Every replay audits the abstracts it is scored against. When the tool and the abstract disagree and a lawyer reads the clause, the abstract is sometimes the one that is wrong: a value keyed from the base annex after an amendment, a currency dropped, a rating table flattened. The sample firm's back-test found more errors in its hand abstraction than in the agents' readings, and every agent error was flagged before it could reach a system.
The sample firm's back-test: every agent error flagged, more errors found in the hand abstracts
50
agreements the sample firm had already abstracted by hand, back-tested
14
agent errors, all caught by a disagreement between the two readers
31
errors in the hand abstraction, shown by the cited pages
Count the abstract's errors in the business case, because they are already in your systems. The cost side of that case, from reading time to the cost of a wrong number, is worked through in what one CSA costs to abstract.
How many agreements are enough for a test?
Enough terms to make a clean result informative, which is more than most pilots use. The rule of three gives a quick bound: if a tool makes no errors in n independently checked terms, the true error rate is still plausibly as high as 3 in n. Ten agreements of 26 terms each is 260 checks, which cannot rule out an error rate above about 1%; fifty agreements is 1,300 checks and brings the bound to about 0.23%. The bound is per field type only if you count per field type, so a test that reads 50 thresholds rules out much less than one that reads 1,300 terms of all kinds.
Spread matters as much as size. Fifty agreements with the same dealer on the same form measure that form. Choose by the dimensions that change the reading: form and governing law, language, number of amendments, protocol adherence, scan quality. In the sample firm, the last back-catalog batch covered 312 legacy agreements and 8,112 terms, and the two readers agreed on 96.8% of them; the rest went to a person.
The evidence for each term: source, readers' agreement, and the systems

What a replay on your own CSAs cannot tell you
It cannot tell you how the tool behaves on documents unlike the sample, or after a model changes. Two practices cover the gap. Keep a fixed set of hand-checked agreements and rerun it before any new version is used: in ISDA and CSA Data Extraction a new agent version runs on a golden set of 400 hand-checked agreements before the Head of Investment Operations approves it. And keep every reading on an append-only record, with the model version, so an error found later can be traced to the version that made it; FINRA lists tracking which model version was used, and logging prompts and outputs, among the effective practices it observed (FINRA, 2026 oversight report).
The audit trail: every reading, question, confirmation and load, with its version

A replay also cannot settle a clause that is genuinely ambiguous. When the Court of Appeal read a 1995 annex in 2019, it accepted that each party's reading was available on the words before deciding which was right ([2019] EWCA Civ 771). A test that scores such a clause as simply right or wrong is measuring the answer key's opinion. Record those clauses as ambiguous, and make sure the tool sends them to a lawyer rather than choosing.
Questions people ask
What is the best credit support annex automation tool?
The one that does best on a blind replay of your own agreements, scored field by field, with its errors caught by its own controls before they reach a system. General benchmarks measure other documents and other tasks; they are a starting point for a shortlist, not a decision.
How do you test an AI contract review tool?
Run it blind on documents your team has already abstracted, compare every field with the abstract and the system value, and have a lawyer settle each mismatch against the clause. Report per-field error rates with how many terms were checked, and count the abstract's errors as well as the tool's.
How many documents do you need for an AI pilot?
Enough terms that a clean result means something: about 1,300 independently checked terms bounds the error rate near 0.23% with 95% confidence, which for a credit support annex is around 50 agreements. Choose them across forms, laws, languages and amendment chains.
Can an AI tool be more accurate than a human abstract?
It can be, especially where the human abstract was keyed once and never updated. In the sample firm's back-test the cited pages showed 31 errors in the hand abstraction against 14 agent errors, all of which the readers' disagreement caught. The replay is how you find out for your own estate.
Should the vendor see our abstracts before the test?
No. The tool should receive the executed documents and amendments only. A tool tuned on your answers will score well on them and tell you nothing about the next agreement.