A legal document search AI tool should be measured on your own agreements, clause type by clause type, against answers that expert lawyers have labelled, with every answer cited to its page and checked by an independent second reading, not by a single accuracy figure on someone else's documents. Research on contract AI shows why: performance varies sharply by task, negations and exceptions trip models up, and legal research tools still make up answers some of the time. For derivatives documentation the stakes are specific, because a wrong reading of a rating trigger or set-off clause becomes a wrong close-out decision. Botminds' Derivatives Documentation Search was built for this test: every value is cited, every cited passage is read again blind on a different model family, readings below a set confidence go to a lawyer, and each agent version passes release tests before it goes live.
Product figures on this page are the solution's release tests on its sample data and its sample settings for a fictional firm, Quillfield Capital; they are design checks, not a customer's measured results. Research figures are linked to their sources, with preprints labelled as not peer reviewed.
What the research says about contract AI, in four numbers
13,000+
expert annotations in CUAD, a contract-review benchmark built with dozens of legal experts
607
contracts annotated in ContractNLI, where models struggled with "negations by exceptions"
162
legal reasoning tasks in LegalBench, designed and hand-crafted by legal professionals
17–33%
share of queries on which AI legal research tools hallucinated, in one preregistered study (preprint)
Why one accuracy figure hides the clauses that matter
Because a contract search system does several different jobs, and each fails differently. Planning the reading, finding a clause, applying amendments in order, spotting documents that disagree and catching misreadings are separate tasks with separate error rates, and an average across them tells a deputy general counsel nothing about the one that failed on the row in front of her. The sample's release tests show the point: each agent is tested on its own job, against its own kind of evidence.
Seven release tests, seven different miss rates: no single figure describes the system
Read the top two bars as the reason the system has people in it. The Checker missed two of 48 deliberately planted misreadings, so a blind second reading is a strong control but not a guarantee; that is why a reading below the close-out threshold goes to a lawyer even when both agents agree, and why every close-out answer is verified by one lawyer and approved by a second. The Planner's plans differed from counsel's on 10 of 250 questions, which is why it asks back when a question is ambiguous instead of guessing.
How accurate is AI at reading contracts?
It depends on the task, and the research is consistent on that. CUAD, a dataset for legal contract review "created with dozens of legal experts" and holding "over 13,000 annotations", found Transformer models only beginning to perform, with results that rose and fell with model design and the size of the training set (Hendrycks et al., NeurIPS 2021). ContractNLI, built on 607 annotated contracts, asked systems whether a hypothesis is entailed by, contradicted by or not mentioned in a contract, and found that "existing models fail badly", with "linguistic characteristics of contracts, such as negations by exceptions" contributing to the difficulty (Koreeda and Manning, 2021). LegalBench assembled 162 tasks "designed and hand-crafted by legal professionals" to measure which kinds of legal reasoning models can perform (Guha et al., 2023, preprint).
"Negations by exceptions" is the derivatives problem in three words. ISDA Schedules are full of them: a set-off section that applies unless Part 5 disapplies it, a rating trigger that fires "below BBB" except where the Schedule counts only one agency, a close-out method replaced for some transactions by an amendment. A model that reads the headline clause and misses the exception is wrong in exactly the place a close-out depends on.
Does AI hallucinate in legal research?
Yes, measurably, even in tools built for lawyers. NIST calls the failure confabulation: confident output that is wrong or invented, the thing people call hallucination, which can mislead the reader (NIST AI 600-1, Generative AI Profile). A preregistered study of AI legal research tools found fewer invented answers than from a general-purpose chatbot, but each tool tested still produced them on 17% to 33% of queries (Magesh et al., arXiv 2024, not peer reviewed). Lending's guide on AI hallucination in legal citations follows the consequences for case law; for executed agreements the remedy is narrower and stronger.
Searching a firm's own executed agreements is a more constrained task than open legal research: the answer must be in a known document, so every claim can be tied to a page and checked. That constraint is the main defence against confabulation, but only if the system enforces it. A value without a citation, or a citation that does not open the passage it claims, should be treated as a confabulation until a person proves otherwise.
How do you test a legal document search AI tool on your own agreements?
Take questions your lawyers have already answered by hand, label the correct answer and its page for each agreement, run the tool on the same estate, and score it agreement by agreement and clause type by clause type. The CUAD method is the template: experts label the clauses, and the system is scored against their labels, not against its own confidence. A dealer group's estate is a natural unit: in the sample, one question across Hollins' 61 agreement families touches 214 documents in three languages.
A fair test set for ISDA clauses: what to include, and why
| Include | Why it is in the set | In the sample estate |
|---|---|---|
| Both ISDA forms and local-law masters | Clauses sit in different sections and use different words | 1992 and 2002 forms; a German Rahmenvertrag; French masters |
| Agreements amended after signing | Tests whether the tool reads the chain or the original | Three triggers moved by a 2023 amendment |
| Scans, including poor ones | Tests reading and the refusal rule | Intake rejects scans below 150 dpi |
| Handwritten, initialled changes | The executed copy differs from the draft | A trigger struck by hand from "BBB-" to "BBB" |
| Agreements with no such clause | A correct "none" is an answer; a missed clause looks the same | 30 of 61 Hollins agreements have no rating trigger |
| Planted errors | Measures whether the checks catch what they should | 64 seeded conflicts; 48 seeded misreadings |
| Ambiguous questions | Tests whether the tool guesses or asks | "Below investment grade" read different ways |
What does an AI citation to a contract have to show?
Enough for a lawyer to verify the value without searching: the document, its date and version, the clause, the page, the passage itself, and the amendment applied if the value is not the original. In the sample, every cell, summary sentence and agent reply links to the document, clause and page it came from, amended values show the original struck through beside them, and where there is no clause the answer says so and lists where the agents looked. A citation to a document without a page, or to a page that does not contain the value, fails the test.
A citation that shows the original, the English reading, both agents' scores and the chain in force

What error rate is acceptable for close-out questions?
None that reaches a decision unchecked. No contract reader, human or machine, is error-free, so the useful question is not "what error rate is acceptable?" but "where does each error get caught?". The sample answers it with thresholds and signatures: a reading must reach 0.90 confidence to be used in a close-out answer without a lawyer deciding the row, 0.80 for an everyday answer, and every close-out answer is verified by one lawyer and approved by a second. The design target is that 100% of close-out answers are cited to the clause and carry both signatures.
One reading or two: where errors are caught
A single reading: One model, once.
A reading plus a blind second reading: The Reader, then the Checker on a different model family, without seeing the first answer.
A single reading: A confident value, indistinguishable from a correct one.
A reading plus a blind second reading: A disagreement, or a low score, flagged on the row.
A single reading: A lawyer must re-read everything to find it, or nothing.
A reading plus a blind second reading: The flagged rows, and only those, go to a lawyer.
A single reading: Anything.
A reading plus a blind second reading: Errors both readings share; hence planted-error tests and a lawyer on every close-out answer.
Who checks the AI in legal document search?
A second, independent reading, and then a lawyer, and the lawyer's review has to be real. The Financial Stability Board's final sound practices on AI warn that over-reliance and automation bias can hollow out human oversight, and that leaning on AI over time can wear down the expertise and judgement staff need to challenge it (FSB, 15 October 2026). The same report recommends "tracking when humans override AI recommendations" and using the patterns to refine both the system and the oversight.
The sample's design takes those warnings literally. The lawyer reviews the rows the checks flag, with the passage open beside the value, and records a reason code for every judgment; overrides of a value are corrections that need a second person. Release tests plant errors precisely so that nobody has to take the Checker's reliability on trust. The setting that decides when a reading goes to a lawyer can only be changed with the Deputy General Counsel's approval, and the sample shows a proposed change, from 0.80 to 0.85 for everyday answers, waiting for that approval.
Thresholds and release tests in one place, changed only with the Deputy General Counsel's approval

A scoring sheet for any legal document search AI tool
Score the tool on what a close-out depends on, not on a demonstration. The sheet below can be run on any product in a pilot, including this one; the two-week pilot plan shows how to set it up on one dealer group.
Eight scores to record per clause type, not per tool
- Correct value, as amended
Against the lawyer-labelled answer for each agreement, after every amendment and protocol.
- Correct "none"
Agreements with no such clause, answered as none, with the places searched.
- Citation opens the passage
Document, clause, page and the passage itself; a wrong page counts as a wrong answer.
- Amendment applied and shown
The amending document cited, the original visible.
- Local-law and scanned documents
Scored separately; they are where averages hide failure.
- Flags on what should be flagged
Low scores, disagreements, one-sided amendments and drafts beside executed copies.
- Questions asked back
Ambiguous questions answered with a question, not a guess.
- Time to a verified answer
Not time to first output: time until a lawyer has verified every row.
To see the citations, scores and thresholds on the sample estate, start from Derivatives Documentation Search; for what each agent may and may not do, see the agents and their limits.
Questions people ask
Can ChatGPT read an ISDA agreement accurately?
A general chatbot can summarise a Schedule, but on its own it does not apply the amendment chain, prove execution, or tie every value to a page for checking. Research on legal tasks finds performance varies sharply by task, and a preregistered study found hallucinations even in specialist legal research tools. For a close-out answer, the controls around the model matter more than the model.
What accuracy should a contract AI vendor claim?
One per task and clause type, on documents like yours, with the test set described. A single headline percentage without the clause types, document types and labelling method behind it cannot be compared with anything. Ask for the planted-error results too: they show whether the checks work.
How many agreements does a fair test need?
Enough to cover every stratum that matters: both forms, local-law masters, amended agreements, scans, handwritten changes and agreements with no such clause. One dealer group's full estate usually does that; in the sample, the Hollins group alone has 61 agreement families and 214 documents.
Should the AI's confidence score be trusted?
As a routing signal, not as proof. In the sample, both agents read the German handwritten change correctly, at 0.84 and 0.86, and the row still went to a lawyer because the close-out threshold is 0.90. A threshold sends uncertain rows to people; it does not certify the rest, which is why every close-out answer still carries two signatures.
Who is accountable if the AI misreads a clause?
The people who verified and approved the answer, which is why the design gives them the citation, the second reading and the authority to override. Agents in the sample cannot approve anything, and every judgment and correction carries a named person and a reason on the trail.