WorkflowResearch & Discovery

Decide · the pilot

Due diligence automation: pilot it on a room you have already closed

The only honest pilot of due diligence automation is a replay: run it on a deal you have already done, where you know the answer, and score it on coverage, citation accuracy, flags found and time. This guide sets out the four measures, the comparison that makes them meaningful, and what BD Search & Evaluation needs from you on day one.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

Due diligence automation is worth buying only if you can measure it, and the only measurement that means anything is a replay. Take a licensing deal you closed, where the memos were written, the red flags were argued and the outcome is known; run the automated pass over the same data room; and compare what it produced with what your own people produced, on four measures — how much of the room it covered, whether its citations point where it says they point, which of your flags it found and which it missed, and how long each took. Everything else is a demonstration. Botminds' BD Search & Evaluation is built to be piloted this way: you bring your search criteria, your scoring weights, your function reviewers and one past room, and the comparison is the deliverable.

This guide is written for the Search & Evaluation head with a budget and one quarter to prove something, and for the function leads whose signatures the output will carry. The figures taken from the solution are fictional sample data, and the design target and estimate are labelled as such.

What due diligence automation actually automates

The phrase covers a wide range, and the range matters because the honest version of this is narrow. What can be automated is the reading at volume: classifying every file in a seller's room by function, document type, version and data cut-off; extracting the facts with the page each came from; drafting function memos where every sentence cites a page; putting two documents side by side where they disagree; and assembling the same findings into a scored recommendation using weights your committee agreed in advance.

What is not automated is every decision about what those findings mean. Whether a route change after the toxicology lots is a condition to signing or a reason to walk; whether an impurity the animals never saw costs six months of bridging work or kills the deal; whether a question should be asked of the seller at all. Those stay with the function leads, who sign, and the committee, which decides.

The reason to be precise about the line is that the pilot has to measure the automated half against something. If the claim is "it does diligence", there is no comparison. If the claim is "it drafts a cited first pass in hours that your function leads then check and sign", the comparison is obvious: the first pass you already did, on a room you already have.

Where a first pass's expert hours go, by hand and with a drafted pass

Clinical efficacy and safety memos
14 h → 1.5 h
CMC: routes, lots, specifications, impurities
12 h → 1.5 h
Patents, licences and agency correspondence
10 h → 1.2 h
Sorting the room by function, version and cut-off
8 h → 0.2 h
Toxicology, NOAELs and margins
8 h → 0.8 h
Red flags, seller questions, recommendation
8 h → 1.4 h
Bar length is the hours by hand; the second figure on each bar is the same work with a drafted pass, as BD Search & Evaluation's own dashboard reports it on fictional sample deals. The 6.6-hour median first pass is a design target measured in the live solution; the roughly 60 hours by hand is our estimate. Treat the shape of the saving as the thing your pilot tests, not as a result you will reproduce.

The pilot: one room you have already closed

A closed deal is the only test material with a known answer. You have the memos your own people wrote, the flags they raised, the questions they sent, the price you offered and — most valuable of all — what you learned afterwards about which concerns turned out to matter. Running the automated pass over that room produces a comparison rather than an impression.

Three choices make or break the exercise. Pick a deal that was genuinely hard, not the cleanest one: a room with contradictions, a late amendment and at least one finding that changed the price. Freeze your own answer before the pass runs, in writing, so the comparison is not reconstructed afterwards from memory. And use the function leads who read it the first time, because their disagreement with a drafted memo is the finding, not a complication.

A six-week pilot on one closed room

Pilotsix weeks
Choose the room, freeze your own answerweek 1
Load criteria, weights, reviewers and the roomweek 2
Run the pass; function leads read the draftsweeks 3-4
Score the four measures and the missesweek 5
Write the comparison; decideweek 6
  • Choose the room, freeze your own answerweek 1
  • Load criteria, weights, reviewers and the roomweek 2
  • Run the pass; function leads read the draftsweeks 3-4
  • Score the four measures and the missesweek 5
  • Write the comparison; decideweek 6
Our suggested shape for a replay pilot, not a vendor commitment. The two weeks in the middle are the only part that scales with the size of the room; the rest is your own team's time, and the last week is where the decision actually gets made.

How do you measure whether the automated pass is right?

By defining what right means before it runs, in terms somebody can count. The temptation is to judge the drafts by reading a few and forming an impression, which is how every demonstration ends and why so few of them survive contact with a real room. An impression cannot be compared with last quarter's, cannot be shown to a sceptical function lead, and cannot tell you whether a change to the settings helped.

So the measurement has to be agreed in advance, on your own closed room, by the people who read it the first time — and it has to produce counts rather than adjectives. Four measures do that, and each of them answers a different way the work can fail.

The four measures: coverage, citation accuracy, flags found, time

Coverage answers "did it read everything?" Citation accuracy answers "when it says page 44, is it page 44?" Flags found answers "did it raise what we raised?" Time answers "was it worth it?" A pilot that reports only the fourth is a demonstration with a stopwatch.

The four measures, how each is counted, and what a pass looks like

MeasureHow it is countedWhat a pass looks likeWhat a failure tells you
CoverageFiles classified and read, as a share of files in the room — with the unreadable ones named rather than silently dropped.Every file accounted for: read, set aside as superseded, or listed as unreadable with a reason.Silent omissions. A system that drops a scanned appendix without saying so will drop the one that matters.
Citation accuracyA sample of memo sentences, checked by the function lead against the page the citation points to.The passage supports the sentence, on the page named, in the final version of the document.The failure that destroys trust fastest. A citation that nearly points at the right place is worse than none.
Flags foundThe flags your own team raised, against those the pass raised: matches, misses and additions.Every flag you raised is found, and the additions are judged on their merits rather than counted as noise.A miss shows where the reading is thin. An addition that turns out to be real is the clearest evidence of value.
TimeElapsed hours from the room opening to drafted memos, and expert hours spent reviewing them.Both fall, and the expert hours that remain are spent checking rather than transcribing.If elapsed time falls but expert hours do not, the drafts are not trusted — look at citation accuracy first.
The four measures to agree before a replay pilot starts. They are our framing of what a diligence pass has to prove, not a standard; the point is that each is counted on your own closed deal, by your own reviewers.

Two practical notes. Score citation accuracy on a random sample drawn by you, not on examples chosen by anyone selling you something. And record the additions separately from the misses: a pass that raises a flag your team did not is often the most interesting result the pilot can produce, and it is the one most likely to be dismissed as a false positive on the first reading.

Why a second reader is the right comparison, and what the evidence shows

The instinct is to compare an automated pass against perfection. The honest comparison is against a human doing the same job once, because that is what a first pass is — and the evidence on single versus double reading of documents is not flattering to the single reader.

In a randomised comparison of data extraction for systematic reviews, "single data extraction resulted in more errors than double data extraction (relative difference: 21.7%, P = .019)," while "the average time spent for single data extraction was less than the average time for double data extraction (relative difference: 36.1%, P = .003)" (Buscemi et al., Journal of Clinical Epidemiology, 2006). It is a pilot study on a different task, and should be read as one. But the shape of the finding — one reader is faster and more wrong, two readers are slower and more right — is exactly the trade a diligence team makes under a bid deadline, and it is the trade an automated pass is trying to change.

Three numbers to hold in mind before a pilot

6.6h

median first pass from the room opening to drafted memos — a design target, measured in the live solution

~60h

the same first pass by hand across six functions — our estimate, and the number your pilot should test

21.7%

more errors from single than from double data extraction in a randomised comparison

First two: BD Search & Evaluation's own dashboard on fictional sample deals — the 6.6 hours is a design target and the 60 hours an estimate. Third: Buscemi, Hartling, Vandermeer, Tjosvold and Klassen, 'Single data extraction generated more errors than double data extraction in systematic reviews', Journal of Clinical Epidemiology 59(7):697-703, July 2006 — a pilot study on systematic-review data extraction, not on diligence.Source: Buscemi et al., J Clin Epidemiol, 2006

Framed that way, the automated pass is not competing with your best reviewer. It is offering to be the second reader your deadline never allowed — one that reads everything, cites everything, and hands the expert the disagreements. That is why the flags-found measure should count additions as well as misses, and why the review time that remains is the number to watch: it should be spent adjudicating, not re-reading.

What do you need to bring to automate due diligence?

Less than most teams expect, and all of it is material you should have anyway. The uncomfortable part is that assembling it exposes decisions your organisation has been making implicitly.

What you bring on day one

  1. Your search criteria and thesis

    Therapy areas, modalities, the fit and differentiation thresholds that make an asset worth pursuing, and the royalty stack limit above which an asset is flagged.

  2. Your scoring weights, agreed by the committee

    On the sample settings: clinical 30%, safety 20%, CMC 15%, IP 15%, regulatory 10%, commercial 10%, totalling 100. If your committee has never written these down, writing them down is the first benefit.

  3. Your score bands

    What score proceeds, what proceeds with conditions, what pauses and what passes. Bands turn an argument about a number into a decision rule.

  4. Named function reviewers

    One named person per function memo, because the signature is the point. A function with no named reviewer is a function whose memo nobody owns.

  5. Your sign-off rules

    Who signs a memo, who signs the recommendation, and what blocks a signature — an open deal-breaker should, and on the sample deals it does.

  6. One past data room, with your own answer frozen

    A folder or a .zip. The memos, flags and price you produced at the time, written down before the pass runs.

  7. The information barrier you want enforced

    Who is wall-crossed on which deal, and the rule that agents serving a deal read only that deal. This is a setting, and each change to it is recorded.

What BD Search & Evaluation is configured from. Every item is a setting rather than a build, and each change is recorded — which matters because a pilot's result is only meaningful if you know which settings produced it.

The settings that produced the result: criteria, weights, reviewers, barrier

The settings screen in BD Search and Evaluation: therapy areas, modalities, the thesis zone thresholds of fit 82 and differentiation 65, a royalty stack limit of 10 per cent and a weekly Monday 06:00 sweep schedule at the top; below them the recommendation weights by function totalling 100 per cent, the score bands, the named function reviewers, the sign-off rules and the information barrier.
BD Search & Evaluation on fictional sample settings. Criteria, scoring, reviewers and the barrier are configuration rather than code, and each change is recorded so a pilot result can be traced to the settings that produced it.

Governance before go-live: governance, data, performance, monitoring

A replay pilot answers "does it work on one room?" Going live asks a wider question, and there is a published framework for it that is neither a vendor's nor your own. The US Government Accountability Office's accountability framework for artificial intelligence "is organized around four complementary principles, which address governance, data, performance, and monitoring," and for each principle it "describes key practices for federal agencies and other entities that are considering, selecting, and implementing AI systems" (GAO-21-519SP, 30 June 2021).

Those four headings map onto a licensing function's questions without much translation.

The four accountability principles, as questions a Search & Evaluation function should answer

PrincipleThe question to answer before go-liveWhat answering it looks like here
GovernanceWho owns this, who may change it, and who is accountable when a memo is wrong?Named function reviewers, recorded sign-off rules, and settings whose every change is recorded against a person.
DataWhat goes in, where does it come from, and what must never mix?One deal's room per deal, public sources kept separate from deal data, superseded drafts set aside, and an information barrier per deal.
PerformanceHow well does it do the job, measured on our own material?The four measures from the replay pilot, scored on a closed room by the people who read it the first time.
MonitoringHow will we know if it degrades, and what happens then?Recurring checks on citation accuracy and flags found on live deals, with the function leads' corrections counted rather than silently absorbed.
Left column: the four principles named in GAO-21-519SP. The second and third columns are our application of them to licensing diligence; the framework is written for federal agencies and other entities considering and implementing AI systems, and is not specific to this work.Source: GAO-21-519SP

Two other texts sharpen the Performance and Monitoring rows. FDA's draft guidance on AI supporting regulatory decisions insists on writing down what the model is for before assessing it: the context of use fixes the model's role and scope for one question, and has to say in detail what is modelled and how its outputs will be used (FDA, January 2025). And the NIST framework is explicit that correctness is a property over time, not at launch: "reliability is a goal for overall correctness of AI system operation under the conditions of expected use and over a given period of time, including the entire lifetime of the system" (NIST AI 100-1). Where the FDA guidance does and does not apply to this work is set out in what agents may and may not do in pharma business development.

What a life-cycle check looks like after go-live

The pilot's measures do not stop being useful when the pilot ends; they become the standing checks. Citation accuracy is the one to sample continuously, because it is the measure a reviewer's trust rests on and the one that fails quietly. Flags found becomes a running comparison between what the pass raised and what the function leads added during review — every addition is a miss, and a drift in that count is the earliest signal that something has changed.

Three operational habits make that work. Count the corrections: when a function lead edits a drafted memo sentence, that edit is data about the system, and absorbing it silently throws the signal away. Watch the dashboard measures as a series rather than a number — assets screened, first-pass hours, red flags found before committee, time per function — because the shape of the series is where degradation shows. And keep one closed room in reserve as a re-test, so a change to settings or models can be scored against a known answer rather than argued about.

The measures as a running series, not a one-off result

The dashboard in BD Search and Evaluation: tiles for assets screened, median first-pass diligence hours, red flags found before committee and the deal funnel from watchlist through screening, diligence and committee, with a breakdown of time per function beneath. The red-flag tile shows 14 raised before committee over the last 30 days.
BD Search & Evaluation on fictional sample data: 14 red flags raised before committee in 30 days, with first-pass hours and time per function tracked as series. The series, not any single figure, is what tells you whether the pass is still working.

What not to automate, and how long does going live take?

The short answer to the second question, honestly given, is that the software is the fast part. We build and deploy BD Search & Evaluation from your requirements and documents — your search criteria, your scoring weights and a past data room — and it goes live once every quality check has passed. The slow part is the organisational work the pilot exposes: agreeing weights, naming reviewers, writing down what blocks a signature. Teams that have those settled move quickly; teams that discover they have never written them down should budget for that conversation, which is worth having whether or not you automate anything.

What to automate, and what to keep with a person

Automate, then checkKeep with a named person
Sorting the room

Automate, then check: Classifying 214 files by function, document type, version and data cut-off, and setting superseded drafts aside.

Keep with a named person: Deciding that an unreadable or missing document is itself a finding worth raising with the seller.

Extracting facts

Automate, then check: Pulling the numbers out with the page reference kept for every one of them.

Keep with a named person: Deciding which of two conflicting numbers the valuation will use.

Drafting memos

Automate, then check: A cited first draft per function, every sentence pointing at the page it came from.

Keep with a named person: Reading the cited pages and signing, with the meaning of the signature recorded.

Raising findings

Automate, then check: Proposing a classification — deal-breaker, price item or covenant — with the sources attached.

Keep with a named person: Owning the classification, and deciding whether it becomes a condition, a price adjustment or a reason to pass.

Questions to the seller

Automate, then check: Drafting them from the gaps each memo found, with the page that prompted them.

Keep with a named person: Approving, rewriting or killing each one before it is sent.

The recommendation

Automate, then check: Scoring with the agreed weights and assembling the committee materials.

Keep with a named person: Two signatures before the committee sees it — and the committee's decision itself.

How the line is drawn in BD Search & Evaluation. The right-hand column is the part a pilot should confirm your own people are willing and able to do at the pace the left-hand column makes possible.

What not to automate is everything in the right-hand column, and one thing beyond it: the decision about whether a seller process is worth entering at all. That is a judgement about your portfolio and your appetite, and it belongs with the people who will live with it. What a first pass should cover once you are in is set out seat by seat in the pharmaceutical due diligence checklist, and what a buyer should expect from the room itself is separated out in the guide to virtual data room due diligence.

Questions people ask

How do you pilot AI on due diligence without risking a live deal?

Replay a deal you have already closed. The room exists, the memos were written, the flags were argued and you know what mattered in the end — so the automated pass can be scored against a known answer with no live transaction exposed. Freeze your own answer in writing before the pass runs, use the function leads who read it the first time, and pick a hard room rather than a clean one.

What should a due diligence pilot measure?

Four things: coverage, with unreadable files named rather than dropped; citation accuracy, checked on a sample you draw yourself; flags found, counting misses and additions separately; and time, split into elapsed hours and expert hours. If elapsed time falls but expert review hours do not, the drafts are not being trusted — look at citation accuracy before anything else.

Is an automated pass better than a human reviewer?

That is the wrong comparison. A first pass under a bid deadline is one reader doing the job once, and the evidence on single reading is unflattering: in a randomised comparison, single data extraction produced 21.7% more errors than double extraction while taking 36.1% less time. The useful framing is that an automated pass is the second reader the deadline never allowed — which is why a flag it raises that your team missed is the most interesting result a pilot can produce.

How do you monitor an automated review after go-live?

Keep the pilot's measures running. Sample citation accuracy continuously, count every correction a function lead makes to a drafted memo rather than absorbing it silently, and watch the dashboard measures as a series — assets screened, first-pass hours, red flags found before committee, time per function. Hold one closed room in reserve so any change to settings can be re-scored against a known answer.

What should never be automated in licensing diligence?

Anything whose failure mode is "we judged it wrong" rather than "we did not see it": choosing between two conflicting numbers, deciding whether a finding is a deal-breaker, a price item or a covenant, approving a question to the seller, signing a memo, and the committee's decision. An open deal-breaker should block the recommendation's signature until it becomes a condition or the deal is passed.

How long does it take to go live with automated diligence?

The build is configuration, not a project: criteria, weights, score bands, reviewers, sign-off rules and the information barrier are settings, and we deploy from your requirements, your documents and a past data room, going live once every quality check has passed. The timetable is usually set by your own decisions rather than ours — a function whose weights and reviewers are already agreed moves quickly, and one that has never written them down should budget for that conversation first.

BD Search & Evaluation.
See it on your own files.

An agentic solution from Botminds. Every external asset on one landscape of fit against differentiation, with red flags from each data room drawn on the asset. Book a demo, or explore how it works.