Contract clause extraction is the task of locating and labelling the passages of a contract that matter — the termination right, the assignment restriction, the prepayment premium — so that a person or a system can read them without reading everything else. On a credit agreement it works, and it is not enough. A syndicated loan is not one contract but a signed original plus every executed amendment since, so extraction returns three or four versions of the same section and no opinion about which of them a borrower is bound by on the thirtieth of October. Botminds' Loan Document Search treats extraction as the first of six steps rather than the product: it reads the executed file, builds each section as amended on the date asked, drafts a short answer in plain words, works the dates and amounts, and then reads every sentence back against the clause it cites before a person ever sees it.
The worked example throughout is a fictional sample facility — Halvorsen Cold Chain Holdings, 23 executed documents, a 2023 credit agreement and three amendments — and the research is from a public benchmark, a peer-reviewed evaluation and two standards and supervisory documents, linked where each appears.
What is contract clause extraction?
Extraction is a labelling problem. Given a contract, find the spans of text that belong to known categories and return them with their location. The best-known public benchmark for it, the Contract Understanding Atticus Dataset, states the task plainly: "The task is to highlight salient portions of a contract that are important for a human to review" (Hendrycks, Burns, Chen and Ball, 2021). The output is a passage and a label. It is not an answer, and the benchmark does not claim it is.
That matters because the three things a loan desk might mean by "search the agreement" are not the same thing at all, and a demonstration can show the first while the buyer is imagining the third.
Three ways to find a clause, and what each leaves for the reader
| What it does | What it returns | What the reader still has to do |
|---|---|---|
| Keyword or full-text search | Every page in the file where the words appear — in the original, in two amendments and in a draft that was never signed. | Open each hit, work out which document it belongs to, check whether that document was executed, and read the section around it. |
| Clause extraction | The passage itself, labelled by category, with its document and page — for each version of it that exists in the file. | Decide which version is in force on the date in question, resolve the defined terms inside it, and do the arithmetic. |
| The clause as amended, on a date | One text: the section as it stands on the date asked, with every earlier version beneath it and the amendment that replaced each one. | Check it — which is a real task, and a much smaller one. |
How accurate is contract clause extraction, and how would you know?
Published accuracy exists for the labelling task and should be read for what it is. The authors of the Atticus dataset built it because "many specialized domains remain untouched by deep learning, as large labeled datasets require expensive expert annotators", assembled it "with dozens of legal experts from The Atticus Project", and reported that it "consists of over 13,000 annotations". Their own verdict on the models they tested was measured: "We find that Transformer models have nascent performance, but that this performance is strongly influenced by model design and training dataset size. Despite these promising results, there is still substantial room for improvement."
Two cautions follow from the dataset's own description. The first is scope: it is a benchmark over commercial contracts in general, not credit agreements, and it says nothing about amendment chains, because the contracts in it are single documents. The second is the task definition: a score measures whether the right span was highlighted for a human to review, not whether a conclusion drawn from that span is correct. A tool can be excellent at the first and silent on the second.
What the benchmark measures, and what a loan question asks
The extraction benchmark: One contract, one document, read as it stands.
The question on the desk: One facility: a signed agreement plus every executed amendment, joinder, fee letter and notice since.
The extraction benchmark: Highlight the passages a human should review, by category.
The question on the desk: Say what the borrower owes, on a named date, in a sentence someone can act on.
The extraction benchmark: The right span was found and labelled.
The question on the desk: The right version was used, the defined terms inside it were resolved, and the arithmetic is right.
The extraction benchmark: A missed or mislabelled passage, found on review.
The question on the desk: A superseded clause quoted confidently to a borrower, found when the money moves.
The extraction benchmark: An annotator with the contract in front of them.
The question on the desk: Only a reader who can see which clause each sentence rests on — which means the answer has to carry them.
So "how accurate is contract clause extraction" is the wrong acceptance question for a loan book, because a good score on it is compatible with a wrong answer. The question that discriminates is whether the system can be wrong in a way you can see. Testing a cited answer sentence by sentence is the version of the question that a servicing desk can actually run, and the peer-reviewed evaluation of legal research tools makes the case for running it: its abstract closes by saying the evidence should "inform the responsibilities of legal professionals in supervising and verifying AI outputs, which remains a central open question for the responsible integration of AI into law" (Magesh and others, 2024).
A credit agreement is not one contract, it is a chain of them
This is the structural fact that extraction alone cannot accommodate. The sample facility's file holds 23 executed documents across 412 pages, which split into 640 sections. One of those sections — the prepayment premium at §2.11(f) — exists in three versions: as originally agreed in March 2023, as replaced by Amendment No. 2 in February 2025, and as replaced again by Amendment No. 3 in July 2026. Extraction returns all three, correctly labelled. Only one of them is the agreement today.
One sample facility, as the loan file reader sees it
23
executed documents in the loan file — agreement, amendments, joinders, fee and side letters, notices
412
pages, every passage keeping the page it was read from
640
sections, each with its defined terms
3
versions of the one section that decided the answer
The chain also has an ordering rule that no amount of text similarity will discover: an amendment binds from its effective date, which is not its signature date and not the date it was filed. Amendment No. 3 was signed and effective on 22 July 2026; a question about what the loan said on 1 March 2025 has to be answered from Amendment No. 2, with Amendment No. 3 marked as not yet signed. That is a date comparison, not a reading comprehension task, and it has to be done before the text is read, not after.
The same section, three versions deep, with the one in force at the top
Defined terms: the second document inside the first
Even with the right version in hand, the clause rarely means anything on its own. A prepayment premium clause that charges a fee on a "Repricing Transaction" is a pointer to a definition that may run half a page and may itself have been amended. A clause that requires notice "three U.S. Government Securities Business Days" before payment points at a definition that incorporates an external calendar by reference. Reading the clause without the definitions is how a confident wrong answer gets written.
Four defined terms behind one seven-sentence answer
| Defined term | Where it lives | Why it decides the answer |
|---|---|---|
| Repricing Transaction | Credit Agreement §1.01, page 41 | The premium applies only to a repricing. Whether a $60 million prepayment funded from cash and sale proceeds is one is settled by this definition, not by the premium clause. |
| Amendment No. 3 Effective Date | Amendment No. 3 §5, page 3 | The premium window runs for a fixed period from this date. Get the date wrong and the window closes on the wrong day. |
| Term Loans | Credit Agreement §1.01 | Fixes what is being prepaid and therefore who shares the payment — here, fourteen term lenders, pro rata. |
| U.S. Government Securities Business Day | Credit Agreement §1.01 | Decides which calendar the three-day notice period is counted on, and so which Tuesday the notice is due. |
The practical consequence is that a clause-level index is not a sufficient data structure. Each section has to carry the definitions it depends on, and each definition has to carry its own amendment history, because a definition can be amended while the clause that uses it is untouched — the quietest way for an answer to go stale. The calendar example is the one that catches the most desks, and it has a guide of its own.
From extraction to the clause as amended on a date
Put the three ideas together and the pipeline has a shape. Extraction gives passages with locations. The chain orders those passages by the document that contains them and the effective date of that document. The definitions are resolved against the same date. What comes out is not a list of hits but one text with a timestamp: this is §2.11(f) as it stood on 30 October 2026, these are the versions it replaced, and here is the executed page each came from.
A cited clause, opened at its executed page
The footer on that screen is a rule, not a caption. Loan files are full of text that reads exactly like the agreement and is not: negotiation drafts, marked-up comparison copies, term sheets, unexecuted signature pages. An index built on text similarity will happily return a draft clause as the answer. The only reliable discriminator is execution status, which has to be established when the document is read and carried with every passage taken from it.
How Loan Document Search splits the work between six agents
Why six specialists rather than one general reader? Because the six steps fail in different ways and need different limits. A reader that is also a calculator will round a number to make a sentence scan. A drafter that is also its own checker will defend what it wrote. Supervisors make a related point about model practice generally: "Practices that are appropriate and effective for one banking organization may be inappropriate and ineffective for a banking organization with a different risk profile or that uses a model for a different purpose" (OCC Bulletin 2026-13). Different purpose, different controls — and inside one answer there are six purposes.
The six steps of one answer, in order, with what each one found
Agent
Loan file readerOpened the facility’s loan file: 23 executed documents, 412 pages, 640 sections, each with its page and defined terms. 1.9 seconds.
Never: Never indexes an unsigned draft as an executed document
Agent
Amendment trackerBuilt §2.11, §2.13 and §3.05 as amended: §2.11(f) last replaced by Amendment No. 3 §4, effective 22 Jul 2026, with two earlier versions set aside. 2.6 seconds.
Never: Never treats an amendment as effective before its effective date, and never merges two versions of a clause
Agent
Servicing answer agentDrafted seven sentences from five sections, one claim per sentence, each carrying the clause it rests on. 3.4 seconds.
Never: Never answers from memory or market practice, and never sends anything to a borrower or a lender
Agent
Dates and amounts calculatorCounted the notice date on the facility’s own calendar and worked the interest and the application of the payment, line by line.
Never: Never rounds a figure, and never estimates a lender’s breakage
Agent
Citation checkerRead all seven sentences back against the clauses they named: nine citations confirmed, one sentence rewritten.
Never: Never adds a source the draft did not already use
Agent
Loan system checkerCompared the answer’s terms with what the servicing system bills from: two differ from Amendment No. 3 — the soft call end and the maturity.
Never: Never changes the loan system; every correction goes to the operations supervisor
The six steps playing while the answer is built
Six agents, and what each one is not allowed to do
The limits are the interesting half of the design. Each step is defined as much by its prohibition as by its job, and the prohibitions are the things a single general-purpose reader would quietly do.
The limits matrix: one row per step, and the thing it may never do
| Step | What it does | What it may never do | What the prohibition prevents |
|---|---|---|---|
| Loan file reader | Splits every executed document into sections with pages and defined terms. | Index an unsigned draft as an executed document. | A negotiation draft quoted back to a borrower as the agreement. |
| Amendment tracker | Builds each section as amended and keeps every earlier version. | Treat an amendment as effective before its effective date, or merge two versions of a clause. | A composite clause that no party ever signed, and dates answered from the future. |
| Servicing answer agent | Writes a short answer in plain words, one claim per sentence, each with its clause. | Answer from memory or market practice, or send anything to a borrower or lender. | A plausible market-standard answer where the agreement is actually silent or unusual. |
| Dates and amounts calculator | Counts business days on the facility’s calendar and works interest, premiums and payment application. | Round a figure, or estimate a lender’s breakage. | A tidy number that is wrong by a rounding, and a funding-loss figure only the lender can certify. |
| Citation checker | Reads each sentence back against the clause it cites. | Add a source the draft did not already use. | A citation found after the fact to justify a sentence that was written without it. |
| Loan system checker | Compares the servicing system’s terms with the version on top of each chain. | Change the loan system. | An unapproved correction to the system of record. |
Read the last line of each row and it is a list of the failure modes of document question-answering, written as a design. The one that is hardest to retrofit is the second: once two versions of a clause have been blended into a single "current" text, the information needed to notice the blend is gone, and no later check can recover it.
Reading each sentence back against the clause it cites
The last step before a person sees anything is the one that makes the rest checkable. Each sentence of the draft is taken with the clause it names and tested: does this clause support this claim? Anything that fails is rewritten or removed. In the sample run, seven sentences carried nine citations and one sentence was rewritten — the draft had put a figure on the lenders' funding loss, and the agreement provides that each lender certifies its own, so the sentence now says that instead.
This is a narrower check than it sounds, and its narrowness is the point. It does not ask whether the answer is wise or complete. It asks only whether each sentence is supported by the text it points at — which is exactly the failure that the standards literature describes. The US standards body defines the category this way:
Confabulations also include generated outputs that diverge from the prompts or other input or that contradict previously generated statements in the same context.
NIST AI 600-1, Generative Artificial Intelligence Profile, July 2024
"Diverge from the input" is precisely the error a read-back catches: the clause was retrieved correctly, and the sentence drifted from it. It is also the error that a citation makes invisible, because a reader who sees a reference beside a sentence tends to treat the sentence as checked. The checker's other rule — that it may never add a source the draft did not use — exists so that the test stays a test: a step allowed to find new support for a doubtful sentence will always find some.
What the extraction misses, and what the desk does about it
No pipeline makes the desk's judgement unnecessary. Five things sit outside it, and a buyer should know which they are before a demonstration persuades them otherwise.
Five things to settle by hand, whatever the extraction does
- Documents that are not in the file
An amendment agreed by email and never circulated as an executed copy does not exist to any system that reads the file. Completeness of the loan file is a custody control, and it is the desk’s, not the software’s.
- Silence
Where the agreement does not address the question, the only correct answer is that it does not, plus the nearest clause. That answer is unsatisfying and frequently right, and a desk has to be willing to send it.
- Genuine ambiguity between two readings
Occasionally two provisions can both be read to apply and the agreement does not rank them. The useful output is both readings with their clauses, escalated — not a confident choice between them.
- What the parties have actually been doing
A course of dealing that departs from the written terms is a legal question about the relationship, not a document question, and nothing in the file will reveal it.
- Anything outside the executed documents
Market practice, a precedent from another facility, what the agent did last time: all are reasons to ask a question, none is an answer to one.
Extraction, then, is a component and a good one. What turns it into something a servicing officer can act on is the ordering around it: the chain that dates it, the definitions that give it meaning, the arithmetic that applies it, and the read-back that tests it. If you are comparing tools on a credit agreement, the questions to ask are mostly about those four rather than about extraction accuracy, and the way to settle them is on your own amended files.
Questions people ask
What is the difference between clause extraction and contract search?
Search returns locations: every place the words occur, across every document in the file, signed or not. Extraction returns the passage itself, labelled by category and tied to its document and page. Neither decides which version of a repeatedly amended clause is in force, which is the step a loan answer cannot skip.
Can AI find a clause in a 400-page agreement?
Finding it is the part that works well. The sample facility's file runs to 412 pages across 23 executed documents and splits into 640 sections in under two seconds. The harder questions are which of the three versions of a section governs on the date asked, and whether the definitions it depends on have themselves been amended.
How do you benchmark contract clause extraction?
Public benchmarks score whether the right span was highlighted against expert annotations — the Atticus dataset was built with dozens of legal experts and carries over 13,000 annotations. For a loan book, add two tests the public benchmarks do not run: an as-of test, where the same question is asked at two dates that straddle an amendment, and a read-back test, where each sentence of an answer is checked against the clause it names.
How do you handle defined terms in a contract?
Treat them as a second document inside the first. Each section should carry the definitions it depends on, and each definition should carry its own amendment history, because a definition can be amended while the clause using it is untouched. A reading that resolves the clause but not its definitions is the most common way a confident answer goes wrong.
What does an agent do that search does not?
It takes responsibility for the steps between the passage and the answer: ordering the versions by effective date, resolving the definitions, doing the arithmetic on the right calendar, and then testing its own sentences against the clauses they cite. Search hands you the raw material for those steps; it does not perform any of them.
Does a higher extraction score mean better answers?
Not reliably. A score measures whether the right passage was located for a human to review. An answer can be built from exactly the right passage and still be wrong, if the passage was superseded, if a definition inside it has changed, or if the sentence drifted from what the passage says. Those are different failures and they need different tests.