SearchTrading & Execution

Evaluate · accuracy

Legal document search AI tool: how to measure it on ISDA clauses

A legal document search AI tool should be judged on your own agreements, clause type by clause type, against answers expert lawyers have labelled, with every answer cited and read a second time. Derivatives Documentation Search was built to be tested that way.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

A legal document search AI tool should be measured on your own agreements, clause type by clause type, against answers that expert lawyers have labelled, with every answer cited to its page and checked by an independent second reading, not by a single accuracy figure on someone else's documents. Research on contract AI shows why: performance varies sharply by task, negations and exceptions trip models up, and legal research tools still make up answers some of the time. For derivatives documentation the stakes are specific, because a wrong reading of a rating trigger or set-off clause becomes a wrong close-out decision. Botminds' Derivatives Documentation Search was built for this test: every value is cited, every cited passage is read again blind on a different model family, readings below a set confidence go to a lawyer, and each agent version passes release tests before it goes live.

Product figures on this page are the solution's release tests on its sample data and its sample settings for a fictional firm, Quillfield Capital; they are design checks, not a customer's measured results. Research figures are linked to their sources, with preprints labelled as not peer reviewed.

What the research says about contract AI, in four numbers

13,000+

expert annotations in CUAD, a contract-review benchmark built with dozens of legal experts

607

contracts annotated in ContractNLI, where models struggled with "negations by exceptions"

162

legal reasoning tasks in LegalBench, designed and hand-crafted by legal professionals

17–33%

share of queries on which AI legal research tools hallucinated, in one preregistered study (preprint)

Hendrycks et al., CUAD (NeurIPS 2021); Koreeda and Manning, ContractNLI (Findings of EMNLP 2021); Guha et al., LegalBench (arXiv preprint, 2023, not peer reviewed in this version); Magesh et al., Hallucination-Free? (arXiv preprint, May 2024, not peer reviewed).Source: CUAD, arXiv 2103.06268; ContractNLI, ACL Anthology; LegalBench, arXiv 2308.11462; Magesh et al., arXiv 2405.20362

Why one accuracy figure hides the clauses that matter

Because a contract search system does several different jobs, and each fails differently. Planning the reading, finding a clause, applying amendments in order, spotting documents that disagree and catching misreadings are separate tasks with separate error rates, and an average across them tells a deputy general counsel nothing about the one that failed on the row in front of her. The sample's release tests show the point: each agent is tested on its own job, against its own kind of evidence.

Seven release tests, seven different miss rates: no single figure describes the system

Checker: seeded misreadings missed (2 of 48)
4.2%
Planner: plans that differed from counsel (10 of 250)
4.0%
Reader: lawyer-checked clauses misread (of 1,200)
1.9%
Amendment agent: chains applied wrongly (of 310)
0.8%
Intake agent: filings linked to the wrong family (of 2,000)
0.4%
Notice agent: test letters wrong (0 of 60)
0%
Conflict agent: seeded conflicts missed (0 of 64)
0%
Last test before release for each agent version in the sample settings of Derivatives Documentation Search, run on seeded and lawyer-checked sample data: Reader 98.1% correct, Amendment 99.2%, Intake 99.6%, Planner 240 of 250, Checker 46 of 48, Notice 60 of 60, Conflict 64 of 64. Miss rates are our arithmetic from those results. Design checks on sample data, not a customer's measured results.

Read the top two bars as the reason the system has people in it. The Checker missed two of 48 deliberately planted misreadings, so a blind second reading is a strong control but not a guarantee; that is why a reading below the close-out threshold goes to a lawyer even when both agents agree, and why every close-out answer is verified by one lawyer and approved by a second. The Planner's plans differed from counsel's on 10 of 250 questions, which is why it asks back when a question is ambiguous instead of guessing.

How accurate is AI at reading contracts?

It depends on the task, and the research is consistent on that. CUAD, a dataset for legal contract review "created with dozens of legal experts" and holding "over 13,000 annotations", found Transformer models only beginning to perform, with results that rose and fell with model design and the size of the training set (Hendrycks et al., NeurIPS 2021). ContractNLI, built on 607 annotated contracts, asked systems whether a hypothesis is entailed by, contradicted by or not mentioned in a contract, and found that "existing models fail badly", with "linguistic characteristics of contracts, such as negations by exceptions" contributing to the difficulty (Koreeda and Manning, 2021). LegalBench assembled 162 tasks "designed and hand-crafted by legal professionals" to measure which kinds of legal reasoning models can perform (Guha et al., 2023, preprint).

"Negations by exceptions" is the derivatives problem in three words. ISDA Schedules are full of them: a set-off section that applies unless Part 5 disapplies it, a rating trigger that fires "below BBB" except where the Schedule counts only one agency, a close-out method replaced for some transactions by an amendment. A model that reads the headline clause and misses the exception is wrong in exactly the place a close-out depends on.

Does AI hallucinate in legal research?

Yes, measurably, even in tools built for lawyers. NIST calls the failure confabulation: confident output that is wrong or invented, the thing people call hallucination, which can mislead the reader (NIST AI 600-1, Generative AI Profile). A preregistered study of AI legal research tools found fewer invented answers than from a general-purpose chatbot, but each tool tested still produced them on 17% to 33% of queries (Magesh et al., arXiv 2024, not peer reviewed). Lending's guide on AI hallucination in legal citations follows the consequences for case law; for executed agreements the remedy is narrower and stronger.

Searching a firm's own executed agreements is a more constrained task than open legal research: the answer must be in a known document, so every claim can be tied to a page and checked. That constraint is the main defence against confabulation, but only if the system enforces it. A value without a citation, or a citation that does not open the passage it claims, should be treated as a confabulation until a person proves otherwise.

How do you test a legal document search AI tool on your own agreements?

Take questions your lawyers have already answered by hand, label the correct answer and its page for each agreement, run the tool on the same estate, and score it agreement by agreement and clause type by clause type. The CUAD method is the template: experts label the clauses, and the system is scored against their labels, not against its own confidence. A dealer group's estate is a natural unit: in the sample, one question across Hollins' 61 agreement families touches 214 documents in three languages.

A fair test set for ISDA clauses: what to include, and why

IncludeWhy it is in the setIn the sample estate
Both ISDA forms and local-law mastersClauses sit in different sections and use different words1992 and 2002 forms; a German Rahmenvertrag; French masters
Agreements amended after signingTests whether the tool reads the chain or the originalThree triggers moved by a 2023 amendment
Scans, including poor onesTests reading and the refusal ruleIntake rejects scans below 150 dpi
Handwritten, initialled changesThe executed copy differs from the draftA trigger struck by hand from "BBB-" to "BBB"
Agreements with no such clauseA correct "none" is an answer; a missed clause looks the same30 of 61 Hollins agreements have no rating trigger
Planted errorsMeasures whether the checks catch what they should64 seeded conflicts; 48 seeded misreadings
Ambiguous questionsTests whether the tool guesses or asks"Below investment grade" read different ways
Our recommended test set for derivatives documentation. The right-hand column shows where each case appears in the fictional sample estate of Derivatives Documentation Search.

What does an AI citation to a contract have to show?

Enough for a lawyer to verify the value without searching: the document, its date and version, the clause, the page, the passage itself, and the amendment applied if the value is not the original. In the sample, every cell, summary sentence and agent reply links to the document, clause and page it came from, amended values show the original struck through beside them, and where there is no clause the answer says so and lists where the agents looked. A citation to a document without a page, or to a page that does not contain the value, fails the test.

A citation that shows the original, the English reading, both agents' scores and the chain in force

Detail panel for Euro Rentenfonds and Hollins Brothers Bank Europe AG, Rahmenvertrag (DRV), German law, signed June 4, 2018, with the counterparty's LEI. On the left, the original German rating clause, Nr. 4 Ratingklausel on page 4, with BBB- struck through and BBB written in, initialled R.V. / M.H. On the right, the English reading of the same clause. Below, the Reader agent reads Below BBB at 0.84 and the Checker agent, reading blind, reads Below BBB at 0.86. The amendment chain in force today shows the Rahmenvertrag signed on June 4, 2018 with the typed BBB- struck by hand to BBB and initialled by both parties at signing, and the trigger in force today as below BBB. Buttons offer to open the full source, ask about this, or confirm the handwritten change.
Derivatives Documentation Search on the fictional sample estate. Both agents read the same value, but both scores are below the 0.90 a close-out answer needs, so a lawyer confirms the handwritten change with a reason code.

What error rate is acceptable for close-out questions?

None that reaches a decision unchecked. No contract reader, human or machine, is error-free, so the useful question is not "what error rate is acceptable?" but "where does each error get caught?". The sample answers it with thresholds and signatures: a reading must reach 0.90 confidence to be used in a close-out answer without a lawyer deciding the row, 0.80 for an everyday answer, and every close-out answer is verified by one lawyer and approved by a second. The design target is that 100% of close-out answers are cited to the clause and carry both signatures.

One reading or two: where errors are caught

A single readingA reading plus a blind second reading
Who reads

A single reading: One model, once.

A reading plus a blind second reading: The Reader, then the Checker on a different model family, without seeing the first answer.

What a misreading looks like

A single reading: A confident value, indistinguishable from a correct one.

A reading plus a blind second reading: A disagreement, or a low score, flagged on the row.

What happens next

A single reading: A lawyer must re-read everything to find it, or nothing.

A reading plus a blind second reading: The flagged rows, and only those, go to a lawyer.

What it cannot catch

A single reading: Anything.

A reading plus a blind second reading: Errors both readings share; hence planted-error tests and a lawyer on every close-out answer.

Our comparison. In the sample estate the Checker confirmed 213 of 214 readings on the Hollins question; the one below 0.90 went to senior counsel.

Who checks the AI in legal document search?

A second, independent reading, and then a lawyer, and the lawyer's review has to be real. The Financial Stability Board's final sound practices on AI warn that over-reliance and automation bias can hollow out human oversight, and that leaning on AI over time can wear down the expertise and judgement staff need to challenge it (FSB, 15 October 2026). The same report recommends "tracking when humans override AI recommendations" and using the patterns to refine both the system and the oversight.

The sample's design takes those warnings literally. The lawyer reviews the rows the checks flag, with the passage open beside the value, and records a reason code for every judgment; overrides of a value are corrections that need a second person. Release tests plant errors precisely so that nobody has to take the Checker's reliability on trust. The setting that decides when a reading goes to a lawyer can only be changed with the Deputy General Counsel's approval, and the sample shows a proposed change, from 0.80 to 0.85 for everyday answers, waiting for that approval.

Thresholds and release tests in one place, changed only with the Deputy General Counsel's approval

Settings screen, Agents and thresholds, listing each agent with version, model, threshold and last test before release: Planner v2.3, 240 of 250 plans match counsel; Reader v3.4, 0.90 close-out and 0.80 everyday, 98.1% on 1,200 lawyer-checked clauses; Amendment v1.9, 99.2% on 310 chains; Conflict v1.4, all 64 seeded conflicts found; Checker v2.1 on a second model family, 46 of 48 seeded misreadings caught; Exposure v1.2, Notice v1.1 and Intake v1.6 with their tests. The close-out threshold reads 0.90 and the everyday threshold 0.80, with 0.85 proposed and awaiting Daniel Okafor. A note says each answer records the agent versions that produced it and that an override is a correction needing a second person.
Derivatives Documentation Search, sample settings. Each answer records the agent versions that produced it, so a test result can be tied to the answers it covers.

A scoring sheet for any legal document search AI tool

Score the tool on what a close-out depends on, not on a demonstration. The sheet below can be run on any product in a pilot, including this one; the two-week pilot plan shows how to set it up on one dealer group.

Eight scores to record per clause type, not per tool

  1. Correct value, as amended

    Against the lawyer-labelled answer for each agreement, after every amendment and protocol.

  2. Correct "none"

    Agreements with no such clause, answered as none, with the places searched.

  3. Citation opens the passage

    Document, clause, page and the passage itself; a wrong page counts as a wrong answer.

  4. Amendment applied and shown

    The amending document cited, the original visible.

  5. Local-law and scanned documents

    Scored separately; they are where averages hide failure.

  6. Flags on what should be flagged

    Low scores, disagreements, one-sided amendments and drafts beside executed copies.

  7. Questions asked back

    Ambiguous questions answered with a question, not a guess.

  8. Time to a verified answer

    Not time to first output: time until a lawyer has verified every row.

Our scoring sheet. Record each score by clause type (rating trigger, set-off, close-out method, notices, collateral re-use) so that a weak clause type cannot hide inside a strong average.

To see the citations, scores and thresholds on the sample estate, start from Derivatives Documentation Search; for what each agent may and may not do, see the agents and their limits.

Questions people ask

Can ChatGPT read an ISDA agreement accurately?

A general chatbot can summarise a Schedule, but on its own it does not apply the amendment chain, prove execution, or tie every value to a page for checking. Research on legal tasks finds performance varies sharply by task, and a preregistered study found hallucinations even in specialist legal research tools. For a close-out answer, the controls around the model matter more than the model.

What accuracy should a contract AI vendor claim?

One per task and clause type, on documents like yours, with the test set described. A single headline percentage without the clause types, document types and labelling method behind it cannot be compared with anything. Ask for the planted-error results too: they show whether the checks work.

How many agreements does a fair test need?

Enough to cover every stratum that matters: both forms, local-law masters, amended agreements, scans, handwritten changes and agreements with no such clause. One dealer group's full estate usually does that; in the sample, the Hollins group alone has 61 agreement families and 214 documents.

Should the AI's confidence score be trusted?

As a routing signal, not as proof. In the sample, both agents read the German handwritten change correctly, at 0.84 and 0.86, and the row still went to a lawyer because the close-out threshold is 0.90. A threshold sends uncertain rows to people; it does not certify the rest, which is why every close-out answer still carries two signatures.

Who is accountable if the AI misreads a clause?

The people who verified and approved the answer, which is why the design gives them the citation, the second reading and the authority to override. Agents in the sample cannot approve anything, and every judgment and correction carries a named person and a reason on the trail.

Derivatives Documentation Search.
See it on your own files.

An agentic solution from Botminds. Ask one question of every ISDA, CSA, repo and prime brokerage agreement at once; agents read each in its own language, apply amendments and protocols in order, flag documents that disagree, read every value twice and return a grid of the whole estate, each cell cited to its clause. Book a demo, or explore how it works.