Yes, ChatGPT can compare two contracts: paste or upload both and it will return a list of differences, often a good one. What it cannot tell you is whether the list is complete, and for a reinsurance treaty wording that is the only question that matters. The work is comparing a broker's new draft, sixty pages or more, with last year's signed wording and the agreed slip, and the costly misses are one-word changes in the middle of the document. Botminds' Treaty Wording Comparison is an agentic solution built for that comparison, with each finding cited to page and line. This page sets out a test that measures any tool, ours included: plant ten known changes in a copy of last year's wording and count what comes back.
The test is needed because we found no published study that measures it. The best-known evaluation of legal AI, a preregistered Stanford study, tested research questions, not document comparison, and its authors list "contract review" among the legal tasks whose benchmarks "remains an important open challenge for the field" (Magesh et al., 2024). A team deciding whether to rely on a chatbot, a legal-tech product or a specialist solution has to produce its own number. The figure below shows why the number has to come from planted changes rather than from reading the output: in a real draft, almost everything is unchanged.
In one 162-clause treaty draft, three clauses decide the outcome
Can ChatGPT compare two contracts? Yes, and here is what it can't tell you
A general-purpose chatbot compares two contracts the way a careful reader skims: it reads both, summarises what differs and tends to describe the differences it finds most salient. On two short agreements that is often enough. On a treaty wording three things go wrong that the answer itself does not reveal. It may leave out a change and not say so. It may describe a change that is not there. And it cannot tell an agreed change from an unagreed one, because the agreement is in a third document, the slip, that a two-document comparison never sees.
Each of those failures looks the same from the outside: a confident, tidy list. The Stanford authors found that "even presenting the same query to these models may yield different answers each time", because decoding is often not set to be deterministic (Magesh et al., 2024, footnote 26). A list you cannot reproduce is a list you cannot audit, which is why the test below runs each tool more than once.
Is AI accurate at contract review? What the published studies measure
The honest answer is that published accuracy figures exist for neighbouring tasks, not for treaty comparison, and they are sobering. The Stanford study of legal research tools found that the leading products "each hallucinate between 17% and 33% of the time", and that the general-purpose GPT-4 model they compared against gave a hallucinated answer 43% of the time (Magesh et al., 2024, Figure 4). Those are research questions, not redlines, but they measure the same failure: a fluent answer that is wrong.
Contract review research shows why the task is hard even before hallucination. In CUAD, a dataset built by legal experts from The Atticus Project, a non-profit, the clauses worth a lawyer's attention "make up about 10% of each contract on average", and each of the 41 kinds of clause covers about 0.25% of the text: "a matter of finding needles in a haystack" (Hendrycks et al., NeurIPS 2021). The annotators were law students who had first done 70 to 100 hours of contract review training, and every annotation was checked by three more. Insurance supervisors see the same risk from the other side: in EIOPA's 2025 survey of 347 insurers, hallucinations were "the top-cited risk" of generative AI (EIOPA, 2025).
What the research measures, and what it doesn't
17–33%
of answers hallucinated by leading AI legal research tools
Stanford preregistered study, 202 queries, 2024
43%
hallucinated by GPT-4 on the same legal queries
Same study, Figure 4
~10%
of a contract is clauses an expert flags for review
CUAD, 510 contracts, 13,101 labelled clauses
Open
benchmarks for contract review, in the Stanford authors’ words
“an important open challenge for the field”
Why a 64-page treaty is a hard case for a chat window
Length is the first problem. Researchers at Stanford and Berkeley found that language models' performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., 2023). A treaty's hours clause sits on page 30 of 64. Two full wordings pasted into one prompt put most of the clauses that matter exactly where models attend least.
Renumbering is the second. Articles move between years, so clause 12.2(a) this year may be 11.3 last year. In the sample, the wording reader paired 162 clauses between Draft 2 and the 61-page signed wording and found 9 renumbered. A comparison that pairs by number reports nine spurious changes and can miss the real one hidden inside a moved clause. The third problem is references: a slip line that says "as expiring" or cites a market clause by number only makes sense with last year's wording or the clause library open beside it, which is three or four documents, not two.
AI contract comparison tools compared with treaty comparison takes the product question further. The point here is narrower: none of these failures is visible in the output, so none can be judged by reading the output. They can only be counted.
How do you test AI contract review? Plant ten changes and count
Take last year's signed wording for a programme you know well. Make a copy and plant ten changes of the kinds that cost money, at different depths in the document. Give each tool the original and the copy, and, if it can use one, a slip that agrees some of the changes and not others. Then count. Because you planted every change, you know the right answer, which no reading of the output can tell you. The ten below are the ones we would plant in a property catastrophe wording; adjust them to your book.
Ten planted changes for a property catastrophe treaty, and what each one tests
| Planted change | Example | What it tests |
|---|---|---|
| Hours in a loss occurrence | Windstorm 72 → 168 consecutive hours | A one-number change inside a defined term, mid-document |
| A peril moved between periods | Hail moved from the 72-hour list to the 168-hour list | Words moved, not added: a diff can show it as noise |
| Reinstatement basis | “100% as to time” → “pro rata as to the unexpired term” | A phrase that reads like drafting but changes price |
| ECO and XPL share | 90% → 100%, in both places it appears | A repeated change: is each instance found? |
| Notice of loss trigger | 30 days and 50% of retention → “as soon as practicable” | A deletion, which models report less reliably than additions |
| Follow the settlements | Adds “including ex gratia settlements” | A three-word insertion in a counsel clause |
| Business covered | Adds business written by managing general agents | A widening of scope in a definition |
| Offset | Offset extended to “all contracts” between the parties | A one-word change of scope |
| Arbitration | Days to appoint an arbitrator, 30 → 15 | A small number in a long boilerplate article |
| Renumbered and changed | Article 11 moved to 12, with “shall” → “may” inside | Pairing by meaning, not number |
Score three numbers. Recall: how many of the ten each tool found, with the right location. False alarms: how many of the controls it flagged. And classification, if you gave it the slip: whether it marked agreed changes as agreed and the others as not. Run each tool three times on the same files; a tool whose list changes between runs has given you a range, not a result.
What does AI miss in contract comparison?
The misses fall into four kinds, and the seeded test is designed so each kind shows up. Deletions are the first: a trigger or a cap that disappears leaves nothing to point at, and a summary of "what changed" tends to describe what was added. Small numbers in long clauses are the second: 72 becoming 168 is two characters on page 30. Moved and renumbered text is the third, producing false changes and hiding true ones. The fourth is the miss no two-document comparison can avoid: a change that is textually obvious but whose significance depends on the slip.
The fourth is the one that costs most. In the sample, Draft 2's reinstatement clause changed "without regard to the date of loss" into "pro rata as to the unexpired term of this Contract at the date of loss". Any comparison shows the words changed. Only the slip shows that line 10 agreed "100% as to time", and only the pricing file shows what the change is worth: for a full-limit loss on 1 September, reinstatement premium of $9.28 million instead of $27.75 million, and $2.31 million to the sample reinsurer's 12.5% share. A list of differences is the start of a review, not the review.
The reinstatement change, against last year and slip line 10
Developing benchmarks for the full range of legal tasks—e.g., deposition summaries, legal memoranda, contract review—remains an important open challenge for the field.
Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024)
Is a PDF compare enough for treaty wordings?
A PDF or word-processor compare is reliable at what it does: it shows every character that changed between two files, deterministically, every time. Most word processors will do it, which answers the common question of how to compare two contracts in Word. It is not enough for a treaty wording, for two reasons. It cannot pair renumbered clauses by meaning, so a reordered draft becomes a wall of red. And it is two-way: it shows what changed since last year and has no idea what the slip agreed, so a change the slip contradicts looks exactly like one it approved.
A chatbot adds summarising to the compare and takes away the determinism. A specialist tool should keep the determinism of the compare for the text, add pairing by meaning for renumbered clauses, and hold the slip and the market clause as references, which is the design of the redline software built for treaties discussed elsewhere in this series.
Running the test fairly: same files, three runs, a stopwatch
A fair test treats every tool alike, including ours. Give each the same three files, the same instructions and the same time, and record what you would record of a person: what it found, where, how sure it was, and how long the reviewer then needed to confirm it. Speed matters only after recall: a tool that returns in a minute and misses three of ten has not saved time.
Seven rules for a seeded test of any contract comparison tool
- Use your own wording
A vendor’s sample is chosen to pass. Last year’s signed wording for a programme you know is the right base.
- Plant at different depths
Put at least three changes in the middle third of the document, where long-context models are weakest.
- Include deletions and moves
Not only insertions. A deleted trigger is the most common thing a summary leaves out.
- Add controls
A reformatted article and a pure renumbering should produce no finding. Count the ones that do.
- Give it the slip
If a tool cannot use the slip, record that it cannot classify changes as agreed or not; that is a result.
- Run three times
Same files, same prompt. Any difference between runs is part of the score.
- Check the citations
A finding without a page and line costs the reviewer the search the tool was meant to save.
Where Botminds' Treaty Wording Comparison fits, and where it does not
Treaty Wording Comparison is built for the treaty case, not for contracts in general. Its wording reader splits each draft and last year's wording into clauses and pairs renumbered clauses by meaning; its slip mapper links every slip line to the clause that must carry it; its three-way comparer redlines each clause word by word against last year and the market or house clause and tests it against its slip line. Every finding cites the draft's page and line, the slip line, last year's words and the clause in the library. In the sample, Draft 2 was compared three ways in 6 minutes 12 seconds, and Draft 3 in 3 minutes 13 seconds.
Each step of the comparison, with what it found and how long it took
It is not a drafting tool and does not edit the broker's wording. It does not decide anything: the impact rater describes what a change does and never says it is acceptable, and the query drafter never sends. It needs the slip and a clause library to do the three-way part; without them it is a good two-way compare. And its figures here are from a fictional sample programme, not a customer's result. The offer on the Treaty Wording Comparison page is to run it on one of your own past renewals, which is the seeded test with real history instead of planted changes.
Should treaty wordings go into a public chatbot at all?
That is a policy question for each company, and the answer is often no. Wordings carry negotiated prices, retentions and reinsurer shares. Florida's insurance code, for example, treats as proprietary business information "information relating to bids or other contractual data, the disclosure of which would impair the efforts of the insurer or its affiliates to contract for goods or services on favorable terms" (Florida Statutes s. 624.4212). Insurers in EIOPA's survey named data privacy and security concerns among the most significant challenges to using generative AI at all (EIOPA, 2025).
Whatever the tool, the questions are the same: where the documents go, who can see them, whether they train anything, and what record is kept of each comparison. A seeded test on a real wording should be run under the same terms the tool would have in production, so the test answers the security question as well as the accuracy one.
Questions people ask
Can Copilot compare two contracts?
General-purpose assistants built into office software can summarise the differences between two documents, with the same strengths and limits as ChatGPT. The seeded test applies unchanged: plant ten known changes, run it three times, and count what it finds and where, before relying on it for a treaty wording.
How do you compare two contracts in Word?
Word processors have a compare function that produces a redline of every change between two versions. It is deterministic and complete at the character level, which makes it a good control in a seeded test. It does not pair renumbered clauses by meaning and does not know what the slip agreed, so it answers "what changed" but not "is this what we agreed".
How many changes should a seeded test plant?
Ten is enough to separate tools that find most changes from those that find some, and small enough to score by hand in an afternoon. Add two controls that should not be flagged. If two tools tie, plant ten more in a different wording rather than adding to the first.
What are legal AI hallucination rates?
The most cited figure is from a preregistered Stanford study of 202 legal research queries: leading AI research tools hallucinated between 17% and 33% of the time, and GPT-4 on its own 43%. The study measured research answers, not contract comparison; for comparison work, the rate that matters is the one your own seeded test produces.
Does an AI comparison need the slip?
It needs the slip to say whether a change was agreed. Without it, every tool, however good, can only report that a clause changed since last year. In the sample draft, 14 clauses changed since last year and only 3 of them contradicted the slip; the slip is what separates the three from the eleven.