WorkflowInvestment Operations & Middle Office

Evaluate · testing an AI review tool

Credit support annex automation tool: replay it on CSAs you abstracted

No published benchmark tells you how a credit support annex automation tool will read your own ISDA annexes. The test that does is a blind replay on agreements your team has already abstracted, scored field by field and settled against the page, which is how ISDA and CSA Data Extraction is set up at every new firm.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

A credit support annex automation tool, the AI that reads each annex and turns its terms into data, should be judged on your own agreements, because accuracy on general contracts says little about ISDA credit support annexes, and no regulator or independent study publishes accuracy for collateral terms. The test that settles it is a replay: take agreements your team has already abstracted, run the tool on the executed documents without showing it your answers, compare every extracted term with your abstract and with what your collateral system holds, and have a lawyer read the clause wherever they differ. Scored field by field, the replay shows how often the tool is wrong, on which terms, and how often your existing abstracts were wrong too. Botminds' ISDA and CSA Data Extraction runs exactly this back-test at every new firm before anything reaches a system, and keeps a fixed set of hand-checked agreements to test every new version of its agents.

A clean result on a small test proves little: the most an error-free replay rules out

100 terms checked, no errors
error rate could still be 3%
300 terms, no errors
up to 1%
1,300 terms (50 agreements × 26), no errors
up to 0.23%
8,112 terms (312 agreements), no errors
up to 0.04%
Our arithmetic, using the statistical “rule of three”: if no errors are seen in n independent checks, the true error rate is below about 3/n with 95% confidence. The 26 terms per agreement and the 312-agreement batch are from the fictional sample firm in ISDA and CSA Data Extraction; they illustrate scale, not a result.

The examples come from a fictional sample firm, Quillfield Capital, with 2,400 executed ISDA, CSA, GMRA and GMSLA agreements, as it runs through the solution. Research is linked where it appears; preprints are labelled as not peer reviewed.

How accurate is a credit support annex automation tool? Nobody can tell you for your CSAs

No one can quote an accuracy figure that applies to your annexes, and a vendor that does is quoting a different test. Accuracy depends on the documents: their forms and governing laws, their scan quality, the languages of local-law annexes, and above all how many amendments and protocol letters sit on top of each base agreement. A tool that reads a clean 2016 variation margin annex perfectly can still return the wrong threshold for an agreement amended three times, because the error is not in the reading but in which document governs.

That is why a review tool's headline number is the wrong thing to compare. The questions that matter are per field (how often is the threshold wrong, the rounding direction, the eligible collateral list), per document type, and as of a date (did it return the term in force, or the term in the base annex). A broader comparison of AI tools for searching and answering questions over derivatives documents is in legal document search AI tools and their accuracy; this guide is about testing extraction of collateral terms.

What does research say about expert-labelled contract datasets?

It says reliable answers on contracts are expensive to produce and models read them unevenly. CUAD needed dozens of legal experts to create more than 13,000 annotations for contract review, and found model performance at an early stage (Hendrycks et al., NeurIPS 2021). MAUD, built on merger agreements, assembled over 39,000 examples and found models well above chance on most questions but with much room for improvement on a large subset (Wang et al., EMNLP 2023). ContractNLI found existing models struggled to tell whether a contract supports, contradicts or is silent on a statement (Findings of EMNLP 2021).

LegalBench, a benchmark of 162 legal reasoning tasks designed by legal professionals, evaluated 20 language models and set out to give lawyers and developers a shared vocabulary for what kind of legal reasoning a task requires (Guha et al., arXiv 2308.11462, not peer reviewed in this version). Two lessons carry over to a credit support annex. A good test needs an answer key that experts have checked, and the result depends on the task: finding the clause is different from extracting the number, which is different again from deciding which of two documents governs.

Why a replay on CSAs you already abstracted is the test that counts

Because you already own the answer key, the documents and the stakes. Your abstracts were made by people who knew the agreements, on the forms and languages you actually hold, and your collateral system shows which values are in use. A replay measures the tool on that, and it measures your current process at the same time: every mismatch is either the tool's error or the abstract's, and finding the second kind is often the most valuable result of the test.

The replay also tests what a demonstration cannot. It shows whether the tool applies amendments in effective-date order, whether it stops on conflicting documents or picks one, whether it reads a German or French annex in the original, and whether each value carries a citation a lawyer can check in seconds. The design that makes those checks routine rather than a one-off is described in a maker-checker design for AI on legal agreements.

How do you design the replay?

Agree the rules before the tool runs, so that nothing is adjusted to fit the result. The six steps below are the design we use and recommend.

A replay test in six steps: blind, field by field, settled against the page

1

Choose the agreements

Fifty or more already abstracted by your team, spread across forms, governing laws, languages and amendment-heavy relationships, not the cleanest fifty.

2

Freeze the answer key

Export the abstracts and the values your collateral system holds today, before the tool runs, so nothing is adjusted after the fact.

3

Run the tool blind

Give it the executed documents and every amendment, nothing else: not your abstracts, not your system values.

4

Compare field by field

Every term, every party, every currency, on the scoring rules agreed in advance.

5

Adjudicate against the page

For each mismatch a lawyer reads the clause and records who was wrong: the tool, the abstract, or both because the clause is ambiguous.

6

Report rates with bounds

Per field, with how many terms were checked, so a clean result on a small sample is not read as proof.

Our test design. Step 5 is what turns a comparison into a measurement: a mismatch is not an error until someone has read the clause.

Two choices decide whether the result means anything. The sample must include the hard agreements: long amendment chains, protocol letters filed in the wrong folder, local-law annexes, rating-linked thresholds. And the adjudication must be done by someone who did not make the original abstract, for the same reason the checker in a maker-checker design must not see the maker's answer.

Which fields should be scored, and how strictly?

Every field that a system computes with, each on a rule written in advance. A threshold is either right or wrong, including its currency and the party it belongs to; a notification time is wrong if the city is wrong; an eligible collateral schedule is scored row by row, not as one field.

Scoring rules for a CSA replay, field by field

FieldScored as correct whenCommon partial errors to count as wrong
ThresholdAmount, currency, party and any rating table all match the term in forceBase-annex value when an amendment changed it; one value for a rating-linked grid
Minimum transfer amountAmount per party, and whether it is split between initial and variation marginTreated as one combined figure when the agreement splits it
Independent amountAmount or formula, trigger and capTrigger omitted, cap omitted
RoundingMultiple and direction, for delivery and return separatelyDirection missing; one rule applied to both
Eligible collateral and haircutsEach row: asset, maturity band, currency, percentage, exclusionsParagraph captured as text; exclusions added by amendment missed
Valuation agent and timingParty or rule; valuation dates; notification time with its cityTime without city; “dealer” assumed
Interest on cashIndex, spread, day count, floor, payment datesIndex updated by amendment or protocol not applied
Mechanism and governing lawTransfer or security; law of the master and of the annexForm identified, mechanism not
Term in forceThe value as of the test date, with the document that set itRight value, wrong source document
Our scoring rules. The last row is scored on every field: a correct number cited to the wrong document will be wrong after the next amendment.

What accuracy do you need on thresholds and MTAs?

None of the errors can reach a system, which is a different requirement from a target accuracy. A threshold read wrong by one amendment can mean a call of millions not made: in the sample estate, a dealer threshold cut to zero in 2023 but still held as USD 10 million in the collateral system left a $6,820,000 delivery amount uncalled on a single Friday. No accuracy rate makes that acceptable, so the test should not ask whether the tool's error rate is low enough to trust; it should ask whether its errors are caught before a system uses them.

That changes what to measure. Count the tool's errors, and then count how many of them its own controls caught: a disagreement between readers, a confidence below the re-read level, a conflict between documents. An error caught by the controls costs a reviewer a few minutes. An error that passes them is the one that matters, and the replay should report it separately.

Score your hand abstraction too

Every replay audits the abstracts it is scored against. When the tool and the abstract disagree and a lawyer reads the clause, the abstract is sometimes the one that is wrong: a value keyed from the base annex after an amendment, a currency dropped, a rating table flattened. The sample firm's back-test found more errors in its hand abstraction than in the agents' readings, and every agent error was flagged before it could reach a system.

The sample firm's back-test: every agent error flagged, more errors found in the hand abstracts

50

agreements the sample firm had already abstracted by hand, back-tested

14

agent errors, all caught by a disagreement between the two readers

31

errors in the hand abstraction, shown by the cited pages

Fictional sample data in ISDA and CSA Data Extraction, not a customer result. Every disagreement was settled by a person reading the clause.

Count the abstract's errors in the business case, because they are already in your systems. The cost side of that case, from reading time to the cost of a wrong number, is worked through in what one CSA costs to abstract.

How many agreements are enough for a test?

Enough terms to make a clean result informative, which is more than most pilots use. The rule of three gives a quick bound: if a tool makes no errors in n independently checked terms, the true error rate is still plausibly as high as 3 in n. Ten agreements of 26 terms each is 260 checks, which cannot rule out an error rate above about 1%; fifty agreements is 1,300 checks and brings the bound to about 0.23%. The bound is per field type only if you count per field type, so a test that reads 50 thresholds rules out much less than one that reads 1,300 terms of all kinds.

Spread matters as much as size. Fifty agreements with the same dealer on the same form measure that form. Choose by the dimensions that change the reading: form and governing law, language, number of amendments, protocol adherence, scan quality. In the sample firm, the last back-catalog batch covered 312 legacy agreements and 8,112 terms, and the two readers agreed on 96.8% of them; the rest went to a person.

The evidence for each term: source, readers' agreement, and the systems

Agreement page for Brennmoor × Strategic Income Fund, a 1994 ISDA Credit Support Annex under New York law executed 23 June 2014. The Drift agent answers a legal reviewer's question with citations: the record was keyed on 30 June 2014 from the original annex and never changed, Amendment No. 3 replaced USD 10,000,000 with zero, and the risk engine has the same gap. The terms in force table lists each election, its value, the document and paragraph it came from, the readers' agreement as a percentage, whether two systems match, and a status of confirmed or drift.
ISDA and CSA Data Extraction on the fictional sample estate. Each term shows the document and paragraph that set it and how strongly the two readers agreed: the columns a replay compares against your abstracts.

What a replay on your own CSAs cannot tell you

It cannot tell you how the tool behaves on documents unlike the sample, or after a model changes. Two practices cover the gap. Keep a fixed set of hand-checked agreements and rerun it before any new version is used: in ISDA and CSA Data Extraction a new agent version runs on a golden set of 400 hand-checked agreements before the Head of Investment Operations approves it. And keep every reading on an append-only record, with the model version, so an error found later can be traced to the version that made it; FINRA lists tracking which model version was used, and logging prompts and outputs, among the effective practices it observed (FINRA, 2026 oversight report).

The audit trail: every reading, question, confirmation and load, with its version

Audit trail, append-only, 18,420 entries since go-live, viewed by an auditor with read-only access. Entries on 19 October: the Load agent version 1.2 pushed LD-2026-0327, kept the earlier record as a version and read back zero; Ruth Okafor, collateral system owner, approved the load; the Load agent prepared two loads after legal confirmed; Hannah Brooks, legal reviewer, confirmed the dealer threshold is zero under Amendment No. 3 paragraph 2(a), with the reason amendment executed and in force; she opened the sizing of a drift, asked the agents why the system is wrong, and opened the source for a German haircut clause; the Drift agent version 2.3 escalated a $2.4M load waiting over four hours. Each entry has a fingerprint.
ISDA and CSA Data Extraction on the fictional sample estate. Each agent entry carries its version, so a replay result can be tied to the version it tested.

A replay also cannot settle a clause that is genuinely ambiguous. When the Court of Appeal read a 1995 annex in 2019, it accepted that each party's reading was available on the words before deciding which was right ([2019] EWCA Civ 771). A test that scores such a clause as simply right or wrong is measuring the answer key's opinion. Record those clauses as ambiguous, and make sure the tool sends them to a lawyer rather than choosing.

Questions people ask

What is the best credit support annex automation tool?

The one that does best on a blind replay of your own agreements, scored field by field, with its errors caught by its own controls before they reach a system. General benchmarks measure other documents and other tasks; they are a starting point for a shortlist, not a decision.

How do you test an AI contract review tool?

Run it blind on documents your team has already abstracted, compare every field with the abstract and the system value, and have a lawyer settle each mismatch against the clause. Report per-field error rates with how many terms were checked, and count the abstract's errors as well as the tool's.

How many documents do you need for an AI pilot?

Enough terms that a clean result means something: about 1,300 independently checked terms bounds the error rate near 0.23% with 95% confidence, which for a credit support annex is around 50 agreements. Choose them across forms, laws, languages and amendment chains.

Can an AI tool be more accurate than a human abstract?

It can be, especially where the human abstract was keyed once and never updated. In the sample firm's back-test the cited pages showed 31 errors in the hand abstraction against 14 agent errors, all of which the readers' disagreement caught. The replay is how you find out for your own estate.

Should the vendor see our abstracts before the test?

No. The tool should receive the executed documents and amendments only. A tool tuned on your answers will score well on them and tell you nothing about the next agreement.

ISDA and CSA Data Extraction.
See it on your own files.

An agentic solution from Botminds. Reads ISDA, CSA, GMRA and GMSLA agreements into structured terms for collateral and risk teams Book a demo, or explore how it works.