WorkflowClinical Development

Evaluate · testing a tool

AI contract review agent: how to test one on site agreements

Buy AI contract review software on a measured replay of redlines you have already closed, scored on four separate things rather than one accuracy figure. This guide sets out the test set, the seeded-change run, the human baseline and the questions that disqualify a vendor — and what Clinical Trial Agreement Negotiation shows about its own sample round.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

An AI contract review agent — software that does a reviewer's reading — takes an incoming agreement or redline, finds what the other side changed, places each change against your own approved positions, and proposes a response. Testing one is harder than it looks, because the thing a demonstration shows best — fluent summaries of clauses on a document the vendor chose — is not the thing that costs money when it fails. What costs money is a change nobody answered, or an escalation that never happened. This guide sets out a four-part measurement a buyer can run on their own closed redlines, and uses Botminds' Clinical Trial Agreement Negotiation as the worked example of what a system should be able to show you while you do it.

Clinical contracting is worth testing carefully because the work sits directly on the critical path of a study: the Trial Innovation Network's account of its first year calls it plainly, “clinical trial contracting is a major barrier to clinical trial start-up”. The sample numbers quoted below come from a fictional demonstration round — one 22-page redline with 19 tracked changes on study VLT-302 with Marlowe University Hospital — and a single document is an illustration, never an accuracy result.

What does AI contract review software actually do?

The useful way to read a product in this category is as a short pipeline with separable parts, because each part fails differently and each can be measured on its own.

In the solution used as the example here, six specialist agents run over one redline. A redline reader maps every tracked change and margin comment to a clause of the template, and never alters the site's text. A playbook checker places each change on the position ladder — preferred, fallback 1, fallback 2, walk-away — cites the rule for every classification, and never proposes accepting anything past the walk-away. A precedent finder looks for the same clause in earlier agreements with the same institution, counting only fully executed ones and showing the signature date. A protocol and consent checker compares subject-injury, records, samples and data wording with the protocol and the site's approved consent form, flagging any promise the form does not make. A budget checker compares fees and payment terms with the fair-market-value benchmark and never approves a fee above the range. A reply drafter writes each response, routes the exceptions to their owners, and assembles the response document and cover email — and a person approves every reply before it is sent.

Six agents of Clinical Trial Agreement Negotiation over one redline, step by step

Agents at work: each of the six agents shown in turn with what it found and how long it took — the redline reader mapping 19 changes and 12 comments to the template, the playbook checker placing all 19 on the ladder, the precedent finder returning five matching clauses from an agreement signed in 2024, the protocol and consent checker matching two sections to the site's consent form, and the budget checker finding no budget fee changed.
The fictional sample round, finishing in 34 seconds. The reason to care about a view like this during an evaluation is that it lets you score each stage separately instead of judging the pipeline by its last sentence.

The four measures, and why one number is not enough

A single accuracy figure hides the only failure that is expensive. Score four things instead, over the same documents.

Recall on changes. Of the changes that were really in the document, how many did the system find? This is the measure most buyers skip and the one with the worst consequences, because an unfound change is answered by silence and then becomes an agreed term.

Placement agreement. Of the changes it found, how often did it put the change on the rung your own senior negotiator would have put it on? Measure agreement against a person, clause family by clause family, not against a notional truth.

Citation validity. When the system says a change sits on a particular position, does the rule it cites exist, and does it say what the system claims? A plausible classification with an invented authority is worse than no classification, because it survives review.

Escalation recall. Of the changes that should have left the automatic path — past the walk-away, unmatched, a fee above the benchmark, a conflict with the consent form — how many actually did? One miss here can cost more than every other error in the run combined.

The four measures, what each is computed over, and the target to set

MeasureComputed overWhere to set the bar
Recall on changesEvery tracked change and margin comment in the document, counted by hand firstNear-total. Anything below your own team’s reading is a disqualifier, because this is the stage people do badly.
Placement agreementThe changes found, against a senior negotiator’s own placementHigh and even across families. A system strong on payment and weak on indemnity is telling you where to keep people.
Citation validityEvery classification that quotes a ruleNo exceptions. A cited rule that does not exist or does not say that is a failed run, whatever the other three show.
Escalation recallThe changes your playbook says must reach a named ownerNo misses. Measure it on seeded cases as well as real ones, because real redlines rarely contain enough of them.
Our four measures for a contract review evaluation. Set each target in writing before the first document is loaded; a target agreed after seeing the results is not a target.

The four measures, written so two people compute them the same way

+

Recall on changes

changes found ÷ changes really in the document

The denominator is a hand count by two readers working independently, reconciled. Count margin comments as items too: a comment is often the only sign of an ask that is not yet in the text.

Evaluation owner

+

Placement agreement

changes placed as the reviewer placed them ÷ changes found

Reported per clause family. One senior negotiator places the set blind, before seeing any output.

Head of contracting

+

Citation validity

classifications whose cited rule exists and says that ÷ classifications citing a rule

Checked against the playbook version in force on the day, by someone who did not run the test.

Legal

+

Escalation recall

changes escalated ÷ changes the playbook requires to be escalated

Includes the seeded cases. A miss is reported as a miss, never averaged into a percentage with the other three.

Legal and risk

Our arithmetic. Keeping the four separate is the point: averaging them produces a number that looks like performance and hides the one failure a sponsor cannot absorb.

How do you test AI contract review software before buying it?

Replay work you have already finished. The ideal test is a set of redline rounds you closed months ago, where the answer your organisation actually gave is known, the agreement is signed, and nobody is under deadline pressure to like the result.

The sequence is: assemble the set; have a senior negotiator place every change blind; seed a set of known changes into fresh copies; run the system; score the four measures; then run the same documents past a second human reader to get a baseline. Only after that should anyone discuss price.

The definition of validation that NIST's AI Risk Management Framework borrows from the ISO 9000 quality standard is the right yardstick: confirmation, by objective evidence, that the requirements for a specific intended use have been met (NIST AI 100-1, 2023).

Two phrases in that definition do the work. Objective evidence rules out a demonstration, however impressive. A specific intended use rules out a benchmark on somebody else's contracts: your intended use is your template, your playbook and the institutions you negotiate with, and the framework asks for the limits of generalisation to be documented too — its MEASURE 2.5 outcome is a system shown to be valid and reliable before deployment, with the limits of how far its results carry beyond the conditions it was built under written down.

Building the test set from redlines you have already closed

As a working rule, twenty to thirty rounds is enough to see the shape of a system, and fewer than ten is too few to score. Build the set for coverage rather than volume.

Include at least one round from each institution type you deal with — an academic medical centre, a community hospital network, a site management organisation, a site in each country whose template differs. Include a first round and a later round, because round 3 and round 4 redlines behave differently: the easy changes are gone and what remains is the argument. Include at least two rounds that went badly, where a clause was refused twice or escalated late. Include one round where the budget exhibit changed and one where it did not.

Then fix the set and do not add to it. A test set that grows while the evaluation runs cannot produce a comparable number for two vendors, and it is how an evaluation quietly becomes a sales process.

A vendor demonstration against a replay on your own closed redlines

The demonstrationThe replay
The document

The demonstration: Chosen by the vendor, often a clean public template.

The replay: Your own round 2 and round 4 redlines, with the margin comments and formatting damage intact.

The positions

The demonstration: A generic set of market positions.

The replay: Your playbook at the version in force, with its walk-aways and its authority line.

What is scored

The demonstration: Whether the summary reads well.

The replay: Four measures with targets set in advance, per clause family.

Who places the changes

The demonstration: The system, with nothing to compare against.

The replay: Your senior negotiator, blind, before any output is seen.

The hard cases

The demonstration: Rarely present.

The replay: Seeded on purpose: a walk-away breach, an unmatched clause, a fee above the benchmark, a consent-form conflict.

What you learn about failure

The demonstration: Nothing. Nothing failed.

The replay: Where it fails, how it fails, and whether the failure is visible to the person who would have to catch it.

The difference is not rigour for its own sake: a replay tells you which clause families you can delegate and which you must keep, which is the decision you are actually making.

Planting known changes: the seeded-change run

Real redlines are a poor test of escalation, because the cases that must escalate are rare by design. So make copies and plant them.

Seed four kinds, a handful of each, in fresh copies of documents already in the set. A walk-away breach: an indemnity edit that makes the sponsor liable for the site's own ordinary negligence. An unmatched clause: a well-drafted provision your playbook has no position on at all, such as a new data-protection annex. A fee above the benchmark: a per-patient figure or a wind-down fee outside the range. And a consent-form conflict: subject-injury wording that promises something the ethics-approved consent form does not say.

Score only one thing on these runs: did each seeded change leave the automatic path and reach the right named owner? Placement subtleties do not matter here. An escalation that arrives at the wrong owner is a half-failure worth recording, because it tells you the routing is guessing rather than reading the rule.

Seeding is also the only honest way to test the undo. Set the authority line one rung higher than you intend to run it, let the system accept something it should not have, and confirm that a person can reverse it before anything is sent. What may be settled automatically, and what may not is the design question this run is measuring.

What the human baseline on the same redlines tells you

Run the same set past a second experienced negotiator, with the same playbook and no sight of the system's output, and score them on the same four measures. Buyers are often reluctant to do this, and it is the single most informative hour of the whole evaluation.

Two results are worth looking for, and both are useful. On recall, a person reading a long document late in the day can miss more than they expect — the baseline is what makes a system's recall meaningful rather than merely high. On placement, two senior negotiators may disagree with each other more than either expects, which is a finding about the playbook rather than about either person: a family where humans disagree is a family whose rungs are not written clearly enough to be applied by anyone, software included.

What the sample demonstration shows, and what it does not

19changes

and 12 margin comments in one 22-page round 2 redline

34s

for six agents to map, place, check precedent, check the consent form and check the budget

1

document — which is a demonstration of the pipeline, not a measurement of accuracy

From the fictional sample redline in the solution. It is included here as an illustration of what a system should be able to show stage by stage during an evaluation; the numbers that should decide a purchase are the four measures computed on your own closed rounds.

The failure that costs money: a missed escalation

Rank the failure modes by what they cost, and the ranking is not what a demonstration suggests.

A missed change is expensive because it is answered by silence, and silence reads as agreement. A mis-citation is expensive because it survives review: a negotiator who sees a rule quoted does not go and check it. A missed escalation is the most expensive of all, because the clause that should have reached a lawyer instead became a signed term — and the two indemnity words that do that kind of damage, “solely” and “gross”, are small edits in a long document.

By contrast, the failures a demonstration makes visible are the cheap ones. A clumsy summary wastes a minute. A counter drafted at the wrong rung is caught by the negotiator who has to send it. Weight your scoring accordingly, and be explicit in the evaluation document that a single missed escalation outranks a percentage point of placement agreement.

The round placed and tallied in Clinical Trial Agreement Negotiation, each acceptance citing its rule

The redline, checked: a strip of 19 cells coloured by outcome across the whole document, the tally of changes accepted, countered and sent to legal, and each accepted change listed with the playbook rule and fallback it was placed on and the signed agreement that matches it.
The view an evaluation should ask to see on its own documents: every change placed, every acceptance citing a rule, and the exceptions separated out — which is what makes the four measures computable at all.

What to write into the evaluation so the result is reproducible

Write the protocol before the first document is loaded, and keep it with the result. Academic trial networks that measure operational efficiency are blunt about why: “Measuring and evaluating operational efficiency requires standardization and assessment across multiple trials”, and the collection of those measures “requires persistence and long-term commitment”. An evaluation nobody can repeat in a year is not a baseline; it is an anecdote with a number on it.

What a defensible evaluation records before the first document is loaded

  1. The test set, fixed and listed

    Which rounds, which institutions, which countries, which rounds went badly. Frozen before the first run.

  2. The playbook version used

    Positions move. A score against version 4.2 is not a score against version 5.

  3. The four targets, in writing

    Set in advance, with the statement that a missed escalation outranks the other three.

  4. Who places the changes blind, and when

    Named reviewer, before any system output exists. Their placement is the comparison, not an afterthought.

  5. The seeded cases and where they were planted

    Four kinds, counted, so escalation recall has a denominator.

  6. The human baseline, run the same way

    Same set, same measures, no sight of the output. It is what makes the numbers mean anything.

  7. The limits of what was tested

    Which institution types, countries and clause families were not covered — the generalisation limits a framework asks you to document.

  8. The re-measurement date

    When this runs again, and on what. A number with no expiry becomes a claim.

Our checklist for the evaluation document itself. Most of it is one page, and it is the page that lets a risk or procurement partner ask how the number was obtained — and get an answer.

Questions to put to a vendor, and the answers that disqualify

Six questions separate a product built for this work from a demonstration. Ask to see the rule behind a classification — not an explanation, the rule, from the playbook you supplied. Ask what the system will refuse to do even if configured to do it. Ask how an automatic acceptance is undone, and watch it happen. Ask what reaches a named person and how the routing is decided. Ask what is stored about a decision, and have someone reconstruct a settled clause from the stored information in front of you. Ask what the system does with a clause it has no position for.

The answers that should end an evaluation: a confidence score offered in place of a citation; a claim of accuracy with no test set behind it; an unwillingness to run on your own redlines; no undo; a system that will send a response if configured to; and a reasoning trace offered instead of a stored decision with a name and a time on it. How the main categories of software compare on one real round is the next question after this one.

What to re-measure after go-live

The evaluation does not end at purchase, because the inputs move. Re-measure on a fixed cadence and after any of four events: a change to your own template, a new playbook version, a new country or institution type, and any round where a person had to undo an acceptance.

Two running numbers are worth watching permanently. How often the undo is used, by clause family — rising use means the authority line is too high for that family. And how often something reached a decision owner that the playbook should have settled, which usually means a family is incomplete rather than that the system is wrong. Both are cheap to collect, and both answer the question an auditor will eventually ask: not whether the system was validated once, but whether anyone has checked since.

Questions people ask

How accurate is AI review of clinical trial agreements?

No published figure transfers to your documents, because accuracy on this work depends on your template, your playbook and the redlines you receive — and a figure computed on a vendor's own set says little about yours. Measure four things on your own closed rounds instead: recall on changes, placement agreement with a senior negotiator, citation validity, and escalation recall. Insist that the four are reported separately rather than averaged.

What should a proof of concept for contract AI include?

A fixed set of twenty to thirty rounds you have already closed, your own playbook at a stated version, a blind placement by a senior negotiator, a seeded-change run for escalation, a human baseline on the same documents, and targets written down before anything runs. Add one page recording what was not tested. That page is what makes the result usable a year later.

What is a good test set for contract review?

One that is built for coverage, not volume: each institution type you deal with, each country template, an early round and a late round, at least two rounds that went badly, and one where the budget exhibit changed. Freeze it before the first run. A set that grows during the evaluation cannot give two vendors a comparable score.

How do human reviewers compare on the same redlines?

That is what the baseline is for: people are often strong on judgement and weaker on recall over a long document, and two reviewers can be less consistent with each other on placement than either predicts. Running it is worth the hour it costs: a clause family where two senior negotiators disagree is a family whose rungs are not written precisely enough for anyone to apply, which is a finding about the playbook rather than about the software.

What questions should you ask an AI contract review vendor?

Show me the rule behind this classification, from my playbook. What will the system refuse to do even if I configure it to? How do I undo an automatic acceptance — show me. What reaches a named person, and how is that decided? Reconstruct a settled clause from what you stored. And what happens to a clause I have no position for? Vague answers to the last two are the ones that matter most.

How long should an evaluation of contract review software take?

Plan on two to four weeks of calendar time, and most of that is assembling the test set and getting a blind placement from someone senior enough to be busy. The runs themselves are fast — the sample round in the solution finishes in 34 seconds — so the schedule is driven by your people, which is an argument for a smaller, better-chosen set rather than a larger one.

Clinical Trial Agreement Negotiation.
See it on your own files.

An agentic solution from Botminds. Places every change in a site's redline on the sponsor playbook's fallback ladder, accepts what sits inside the contracts manager's authority with the citation, and sends only out-of-policy clauses to legal. Book a demo, or explore how it works.