An AI contract review agent — software that does a reviewer's reading — takes an incoming agreement or redline, finds what the other side changed, places each change against your own approved positions, and proposes a response. Testing one is harder than it looks, because the thing a demonstration shows best — fluent summaries of clauses on a document the vendor chose — is not the thing that costs money when it fails. What costs money is a change nobody answered, or an escalation that never happened. This guide sets out a four-part measurement a buyer can run on their own closed redlines, and uses Botminds' Clinical Trial Agreement Negotiation as the worked example of what a system should be able to show you while you do it.
Clinical contracting is worth testing carefully because the work sits directly on the critical path of a study: the Trial Innovation Network's account of its first year calls it plainly, “clinical trial contracting is a major barrier to clinical trial start-up”. The sample numbers quoted below come from a fictional demonstration round — one 22-page redline with 19 tracked changes on study VLT-302 with Marlowe University Hospital — and a single document is an illustration, never an accuracy result.
What does AI contract review software actually do?
The useful way to read a product in this category is as a short pipeline with separable parts, because each part fails differently and each can be measured on its own.
In the solution used as the example here, six specialist agents run over one redline. A redline reader maps every tracked change and margin comment to a clause of the template, and never alters the site's text. A playbook checker places each change on the position ladder — preferred, fallback 1, fallback 2, walk-away — cites the rule for every classification, and never proposes accepting anything past the walk-away. A precedent finder looks for the same clause in earlier agreements with the same institution, counting only fully executed ones and showing the signature date. A protocol and consent checker compares subject-injury, records, samples and data wording with the protocol and the site's approved consent form, flagging any promise the form does not make. A budget checker compares fees and payment terms with the fair-market-value benchmark and never approves a fee above the range. A reply drafter writes each response, routes the exceptions to their owners, and assembles the response document and cover email — and a person approves every reply before it is sent.
Six agents of Clinical Trial Agreement Negotiation over one redline, step by step

The four measures, and why one number is not enough
A single accuracy figure hides the only failure that is expensive. Score four things instead, over the same documents.
Recall on changes. Of the changes that were really in the document, how many did the system find? This is the measure most buyers skip and the one with the worst consequences, because an unfound change is answered by silence and then becomes an agreed term.
Placement agreement. Of the changes it found, how often did it put the change on the rung your own senior negotiator would have put it on? Measure agreement against a person, clause family by clause family, not against a notional truth.
Citation validity. When the system says a change sits on a particular position, does the rule it cites exist, and does it say what the system claims? A plausible classification with an invented authority is worse than no classification, because it survives review.
Escalation recall. Of the changes that should have left the automatic path — past the walk-away, unmatched, a fee above the benchmark, a conflict with the consent form — how many actually did? One miss here can cost more than every other error in the run combined.
The four measures, what each is computed over, and the target to set
| Measure | Computed over | Where to set the bar |
|---|---|---|
| Recall on changes | Every tracked change and margin comment in the document, counted by hand first | Near-total. Anything below your own team’s reading is a disqualifier, because this is the stage people do badly. |
| Placement agreement | The changes found, against a senior negotiator’s own placement | High and even across families. A system strong on payment and weak on indemnity is telling you where to keep people. |
| Citation validity | Every classification that quotes a rule | No exceptions. A cited rule that does not exist or does not say that is a failed run, whatever the other three show. |
| Escalation recall | The changes your playbook says must reach a named owner | No misses. Measure it on seeded cases as well as real ones, because real redlines rarely contain enough of them. |
The four measures, written so two people compute them the same way
Recall on changes
changes found ÷ changes really in the document
The denominator is a hand count by two readers working independently, reconciled. Count margin comments as items too: a comment is often the only sign of an ask that is not yet in the text.
Evaluation owner
Placement agreement
changes placed as the reviewer placed them ÷ changes found
Reported per clause family. One senior negotiator places the set blind, before seeing any output.
Head of contracting
Citation validity
classifications whose cited rule exists and says that ÷ classifications citing a rule
Checked against the playbook version in force on the day, by someone who did not run the test.
Legal
Escalation recall
changes escalated ÷ changes the playbook requires to be escalated
Includes the seeded cases. A miss is reported as a miss, never averaged into a percentage with the other three.
Legal and risk
How do you test AI contract review software before buying it?
Replay work you have already finished. The ideal test is a set of redline rounds you closed months ago, where the answer your organisation actually gave is known, the agreement is signed, and nobody is under deadline pressure to like the result.
The sequence is: assemble the set; have a senior negotiator place every change blind; seed a set of known changes into fresh copies; run the system; score the four measures; then run the same documents past a second human reader to get a baseline. Only after that should anyone discuss price.
The definition of validation that NIST's AI Risk Management Framework borrows from the ISO 9000 quality standard is the right yardstick: confirmation, by objective evidence, that the requirements for a specific intended use have been met (NIST AI 100-1, 2023).
Two phrases in that definition do the work. Objective evidence rules out a demonstration, however impressive. A specific intended use rules out a benchmark on somebody else's contracts: your intended use is your template, your playbook and the institutions you negotiate with, and the framework asks for the limits of generalisation to be documented too — its MEASURE 2.5 outcome is a system shown to be valid and reliable before deployment, with the limits of how far its results carry beyond the conditions it was built under written down.
Building the test set from redlines you have already closed
As a working rule, twenty to thirty rounds is enough to see the shape of a system, and fewer than ten is too few to score. Build the set for coverage rather than volume.
Include at least one round from each institution type you deal with — an academic medical centre, a community hospital network, a site management organisation, a site in each country whose template differs. Include a first round and a later round, because round 3 and round 4 redlines behave differently: the easy changes are gone and what remains is the argument. Include at least two rounds that went badly, where a clause was refused twice or escalated late. Include one round where the budget exhibit changed and one where it did not.
Then fix the set and do not add to it. A test set that grows while the evaluation runs cannot produce a comparable number for two vendors, and it is how an evaluation quietly becomes a sales process.
A vendor demonstration against a replay on your own closed redlines
The demonstration: Chosen by the vendor, often a clean public template.
The replay: Your own round 2 and round 4 redlines, with the margin comments and formatting damage intact.
The demonstration: A generic set of market positions.
The replay: Your playbook at the version in force, with its walk-aways and its authority line.
The demonstration: Whether the summary reads well.
The replay: Four measures with targets set in advance, per clause family.
The demonstration: The system, with nothing to compare against.
The replay: Your senior negotiator, blind, before any output is seen.
The demonstration: Rarely present.
The replay: Seeded on purpose: a walk-away breach, an unmatched clause, a fee above the benchmark, a consent-form conflict.
The demonstration: Nothing. Nothing failed.
The replay: Where it fails, how it fails, and whether the failure is visible to the person who would have to catch it.
Planting known changes: the seeded-change run
Real redlines are a poor test of escalation, because the cases that must escalate are rare by design. So make copies and plant them.
Seed four kinds, a handful of each, in fresh copies of documents already in the set. A walk-away breach: an indemnity edit that makes the sponsor liable for the site's own ordinary negligence. An unmatched clause: a well-drafted provision your playbook has no position on at all, such as a new data-protection annex. A fee above the benchmark: a per-patient figure or a wind-down fee outside the range. And a consent-form conflict: subject-injury wording that promises something the ethics-approved consent form does not say.
Score only one thing on these runs: did each seeded change leave the automatic path and reach the right named owner? Placement subtleties do not matter here. An escalation that arrives at the wrong owner is a half-failure worth recording, because it tells you the routing is guessing rather than reading the rule.
Seeding is also the only honest way to test the undo. Set the authority line one rung higher than you intend to run it, let the system accept something it should not have, and confirm that a person can reverse it before anything is sent. What may be settled automatically, and what may not is the design question this run is measuring.
What the human baseline on the same redlines tells you
Run the same set past a second experienced negotiator, with the same playbook and no sight of the system's output, and score them on the same four measures. Buyers are often reluctant to do this, and it is the single most informative hour of the whole evaluation.
Two results are worth looking for, and both are useful. On recall, a person reading a long document late in the day can miss more than they expect — the baseline is what makes a system's recall meaningful rather than merely high. On placement, two senior negotiators may disagree with each other more than either expects, which is a finding about the playbook rather than about either person: a family where humans disagree is a family whose rungs are not written clearly enough to be applied by anyone, software included.
What the sample demonstration shows, and what it does not
19changes
and 12 margin comments in one 22-page round 2 redline
34s
for six agents to map, place, check precedent, check the consent form and check the budget
1
document — which is a demonstration of the pipeline, not a measurement of accuracy
The failure that costs money: a missed escalation
Rank the failure modes by what they cost, and the ranking is not what a demonstration suggests.
A missed change is expensive because it is answered by silence, and silence reads as agreement. A mis-citation is expensive because it survives review: a negotiator who sees a rule quoted does not go and check it. A missed escalation is the most expensive of all, because the clause that should have reached a lawyer instead became a signed term — and the two indemnity words that do that kind of damage, “solely” and “gross”, are small edits in a long document.
By contrast, the failures a demonstration makes visible are the cheap ones. A clumsy summary wastes a minute. A counter drafted at the wrong rung is caught by the negotiator who has to send it. Weight your scoring accordingly, and be explicit in the evaluation document that a single missed escalation outranks a percentage point of placement agreement.
The round placed and tallied in Clinical Trial Agreement Negotiation, each acceptance citing its rule

What to write into the evaluation so the result is reproducible
Write the protocol before the first document is loaded, and keep it with the result. Academic trial networks that measure operational efficiency are blunt about why: “Measuring and evaluating operational efficiency requires standardization and assessment across multiple trials”, and the collection of those measures “requires persistence and long-term commitment”. An evaluation nobody can repeat in a year is not a baseline; it is an anecdote with a number on it.
What a defensible evaluation records before the first document is loaded
- The test set, fixed and listed
Which rounds, which institutions, which countries, which rounds went badly. Frozen before the first run.
- The playbook version used
Positions move. A score against version 4.2 is not a score against version 5.
- The four targets, in writing
Set in advance, with the statement that a missed escalation outranks the other three.
- Who places the changes blind, and when
Named reviewer, before any system output exists. Their placement is the comparison, not an afterthought.
- The seeded cases and where they were planted
Four kinds, counted, so escalation recall has a denominator.
- The human baseline, run the same way
Same set, same measures, no sight of the output. It is what makes the numbers mean anything.
- The limits of what was tested
Which institution types, countries and clause families were not covered — the generalisation limits a framework asks you to document.
- The re-measurement date
When this runs again, and on what. A number with no expiry becomes a claim.
Questions to put to a vendor, and the answers that disqualify
Six questions separate a product built for this work from a demonstration. Ask to see the rule behind a classification — not an explanation, the rule, from the playbook you supplied. Ask what the system will refuse to do even if configured to do it. Ask how an automatic acceptance is undone, and watch it happen. Ask what reaches a named person and how the routing is decided. Ask what is stored about a decision, and have someone reconstruct a settled clause from the stored information in front of you. Ask what the system does with a clause it has no position for.
The answers that should end an evaluation: a confidence score offered in place of a citation; a claim of accuracy with no test set behind it; an unwillingness to run on your own redlines; no undo; a system that will send a response if configured to; and a reasoning trace offered instead of a stored decision with a name and a time on it. How the main categories of software compare on one real round is the next question after this one.
What to re-measure after go-live
The evaluation does not end at purchase, because the inputs move. Re-measure on a fixed cadence and after any of four events: a change to your own template, a new playbook version, a new country or institution type, and any round where a person had to undo an acceptance.
Two running numbers are worth watching permanently. How often the undo is used, by clause family — rising use means the authority line is too high for that family. And how often something reached a decision owner that the playbook should have settled, which usually means a family is incomplete rather than that the system is wrong. Both are cheap to collect, and both answer the question an auditor will eventually ask: not whether the system was validated once, but whether anyone has checked since.
Questions people ask
How accurate is AI review of clinical trial agreements?
No published figure transfers to your documents, because accuracy on this work depends on your template, your playbook and the redlines you receive — and a figure computed on a vendor's own set says little about yours. Measure four things on your own closed rounds instead: recall on changes, placement agreement with a senior negotiator, citation validity, and escalation recall. Insist that the four are reported separately rather than averaged.
What should a proof of concept for contract AI include?
A fixed set of twenty to thirty rounds you have already closed, your own playbook at a stated version, a blind placement by a senior negotiator, a seeded-change run for escalation, a human baseline on the same documents, and targets written down before anything runs. Add one page recording what was not tested. That page is what makes the result usable a year later.
What is a good test set for contract review?
One that is built for coverage, not volume: each institution type you deal with, each country template, an early round and a late round, at least two rounds that went badly, and one where the budget exhibit changed. Freeze it before the first run. A set that grows during the evaluation cannot give two vendors a comparable score.
How do human reviewers compare on the same redlines?
That is what the baseline is for: people are often strong on judgement and weaker on recall over a long document, and two reviewers can be less consistent with each other on placement than either predicts. Running it is worth the hour it costs: a clause family where two senior negotiators disagree is a family whose rungs are not written precisely enough for anyone to apply, which is a finding about the playbook rather than about the software.
What questions should you ask an AI contract review vendor?
Show me the rule behind this classification, from my playbook. What will the system refuse to do even if I configure it to? How do I undo an automatic acceptance — show me. What reaches a named person, and how is that decided? Reconstruct a settled clause from what you stored. And what happens to a clause I have no position for? Vague answers to the last two are the ones that matter most.
How long should an evaluation of contract review software take?
Plan on two to four weeks of calendar time, and most of that is assembling the test set and getting a blind placement from someone senior enough to be busy. The runs themselves are fast — the sample round in the solution finishes in 34 seconds — so the schedule is driven by your people, which is an argument for a smaller, better-chosen set rather than a larger one.