Evaluating AI drug discovery software means running an experiment with a control, and the control is sitting in your own files: the target reviews your scientists already wrote and signed. Take targets you have already decided on — including one you held and one you stopped — hide the conclusions, run them through the software, and measure three things: whether every claim it writes resolves to a real passage that says what the claim says, whether its grades agree with your team's after both have been challenged, and how long each assessment took from request to signature. Everything else a vendor shows you is a demonstration. This guide sets out that protocol in full, with the pass mark written down in advance, and says where Botminds' Drug Target Assessment is designed to meet it.
Two public frameworks give the design its spine — a medicines regulator's approach to model credibility and a standards body's risk framework — and both are quoted and linked where they are used. The run numbers quoted from the solution are from its sample portfolio, whose targets, people and studies are fictional; where a number is a design goal rather than a measurement, it says so.
How do you evaluate AI drug discovery software?
By fixing what the software is for before you look at it, and then collecting evidence proportionate to what goes wrong if it is wrong. That is the FDA's framing for AI models in its draft guidance, and although the guidance explicitly leaves drug discovery outside its scope, the method is the most useful thing published for a buyer in this position. It starts with the context of use: the specific role and scope of the model, for one stated question (FDA draft guidance, January 2025).
Write yours in one sentence before the first meeting with a vendor. For example: this model assembles and grades the published and internal evidence for a named target in a named indication, for a therapeutic-area lead who signs a recommendation to a portfolio committee. Everything in the evaluation follows from that sentence — which targets to use, who must be in the room, what counts as a failure.
The second half of the framing is proportionality. The same guidance says the recommendations and assessment activities used to establish credibility "will generally be tailored to the specific COU and will depend on model risk". A tool that routes a screening budget needs less evidence about itself than one whose conclusions will later be quoted in a submission. Decide which you are buying, because the two deserve different evaluations and vendors will happily sell you the cheaper one.
Start from the decision, not the demonstration
A demonstration is designed to work. It uses a target the vendor has run before, in an indication with a rich literature, with no internal data and no awkward licence. An evaluation is designed to find out, which means it has to be built on decisions you have already taken and can therefore score.
The structural move is simple and uncomfortable: your own past reviews become the answer key. Take the signed assessment, hide the conclusions from whoever runs the pilot, and compare grade by grade at the end. It is uncomfortable because it scores your own team as much as the software, and the disagreements are the most valuable output of the whole exercise. Expect several of them to be cases where the original review was defensible but under-evidenced rather than wrong; those are worth more than the ones where the software is simply mistaken, because they are about your rubric.
The US standards body says the same about any AI system: it should be tested before it is deployed, and tested regularly while it is in operation (NIST AI Risk Management Framework 1.0, section 5.3).
The second half of that rule is the one organisations skip. An evaluation produces a decision to buy; it does not produce a system that stays correct. The same framework adds that independent review makes testing more effective and guards against internal bias and conflicts of interest (NIST) — which in practice means the person who championed the purchase should not be the person who reports on whether it is still working a year later.
Which targets to put in the evaluation set
Six to eight targets, chosen for what they will reveal rather than for being representative. A set of eight that are all strong, recent and well-published will tell you nothing except that the software can read.
How to pick the eight targets, and what each one is there to test
- Two you advanced, with strong human genetics
The easy case. If the grades disagree with yours here, stop the evaluation and find out why before going further.
- One you stopped
The hardest and most informative. A system that cannot reach a D on a target you killed is grading confidence, not evidence.
- One you held for weak evidence
Tests whether the software distinguishes "no evidence" from "evidence against" — the distinction a hold depends on.
- One where the key data is internal
A target whose case rests on your own single-cell or screening data. This is where public-data tools and specialist ones separate.
- One with a safety liability you know about
Check that the human loss-of-function phenotype surfaces at the top of the safety section rather than in paragraph four.
- One in a crowded competitive field
Competitor and patent evidence is the fastest-ageing part of an assessment; check the dates on what comes back.
- One with almost no literature
The honesty test. A confident grade on a thin evidence base is the single worst failure a tool of this kind can have.
Hide the conclusions properly. Whoever runs the targets through the software should not know how each one was decided, and the comparison should happen in one sitting at the end with both documents open. An evaluator who knows the answer will read the output as confirming or contradicting it rather than as evidence.
What to measure: citations that resolve, grades that agree, days that fall
Three measurements carry the decision. Everything else is a nice-to-have that will not change what you do.
The three measurements, how to take them, and what to compare them against
| What to measure | How to take it | What a pass looks like | The design goal the solution states |
|---|---|---|---|
| Claims that resolve to a real passage | Sample 20 claims at random per assessment. Open the passage behind each and check that it says what the claim says, including the number. | Set this yourself before you start; anything below the high nineties means a reviewer must check every claim, which removes the saving. | 100% of claims carry their source passage, one click from the grade. |
| Grade agreement with your own review | Compare grade by grade against the hidden conclusions, after both sides have been challenged. Record each disagreement and its reason. | Agreement matters less than whether every disagreement is explainable from evidence either side missed. | Not a stated goal: the rubric is yours, and the grades are graded against it. |
| Days from request to signature | Timestamp the request and the signature on every target in the set. Count working days, not hours of effort. | A fall large enough to change behaviour. A week is a different process from a month; three weeks is the same process slightly faster. | A median of about five days from request to a signed assessment. |
| Challenges accepted before signing | Count the challenges raised against each assessment, and how many the owner accepted. | Neither zero nor everything: a system that raises nothing is not reading sceptically, and one whose challenges are always accepted is grading badly in the first place. | No stated goal. The sample dashboard shows 65% of challenges accepted by reviewers — sample data, not a target. |
| Where the evidence came from | Count the sources behind each assessment by type, and check that the ones you switched off were not read. | Every source type you enabled is represented, and none you disabled appears. | Not a goal but a setting: only the sources switched on are read, with licences respected. |
Write the pass mark down before the first target is run, and have the person who will sign the purchase sign the pass mark too. An evaluation whose standard is set after the results are in has measured nothing; it has produced an opinion with numbers attached.
How do you know the output is accurate?
By resolving claims, not by reading them. The accuracy question for this category of software is overwhelmingly a citation question, and it has been studied directly: the authors of one published count of fabricated references describe exactly the labour involved, having "searched multiple databases and websites to determine the prevalence of fabricated citations, to identify errors in the citations to non-fabricated papers, and to evaluate adherence to APA citation format". Nobody is going to do that for a target assessment every month. The design has to make it unnecessary.
So the accuracy test is mechanical. Pick twenty claims at random from a finished assessment — not the ones the vendor highlights — and for each one open the source, find the passage, and ask three questions: does the passage exist, does it say what the claim says, and is the number the claim quotes the number in the table rather than a rounded version from an abstract. Twenty claims across eight assessments is 160 checks, perhaps two days of one person's time, and it is the most valuable two days in the evaluation.
Watch for a specific and easy failure: a correct citation attached to an overstated claim. "Loss of function is protective" is a different statement from "a loss-of-function variant was associated with reduced risk in one cohort of 4,000", and only one of them is what the table supports. What agents can and cannot be trusted with goes through the failure modes in more detail.
One finished run: what there is to count in it

The questions to ask before anyone signs anything
Grouped so that each group has an owner in your organisation — which is also the fastest way to find out whether the right people have been involved at all.
The evaluation questions, grouped by who should be asking them
- Evidence and citation — for the scientist
Can I open the passage behind any claim, in the source document? Is the evidence type stated — human, animal, cell, transcript, protein? What happens to a claim the system cannot cite?
- Grading and challenge — for the therapeutic-area lead
Whose rubric is it, and can we change the bar for each grade? What argues against a grade before I see it, and can I see what it argued? Can I change a grade, and does that force me to write a reason?
- Control and signature — for governance
Who signs, what does the signature mean, and what routes a weak safety grade to a second reader? Can a grade move without a person confirming it? Does a signed version stay as signed?
- Sources and licences — for information management
Which sources are read, and who decides? Does licensed full text leave the system? Can we switch a source off and prove it was not read?
- Monitoring — for whoever owns it in a year
How often are signed assessments re-checked, what happens when new evidence moves a grade, and who is asked to confirm it? What does the dashboard show a leader who was not involved in the purchase?
What you have to bring: sources, rubric, past reviews, licences
Four things, and the first two take longer to assemble than teams expect.
A written rubric comes first. If your six questions are graded differently by different scientists today, the evaluation will measure that variation rather than the software, and you will not be able to tell the two apart. Writing the bar for each grade — what earns an A, what makes something a C — is a day's work that pays for itself whatever you decide to buy. In the sample portfolio the default is that an A needs two independent human sources, with one human source plus a functional study, or any three sources, as the alternatives.
Then the sources: licensed literature with full text where you hold the licence, the public genetics and target-disease portals, your internal bulk and single-cell omics, your screening and chemistry reports, your competitor pipeline extract, and the patent landscape. Check the licences before the pilot rather than during it; the commonest delay in an evaluation of this kind is a publisher agreement that nobody has read since 2019.
Then the past reviews — six to eight signed ones, with their conclusions removable. And then the people: a scientist who will check claims, a lead who will grade, a safety reader, and someone who owns the result. An evaluation run by one person in the gaps of their week produces a finding nobody believes.
What an evaluation is actually configuring

Four weeks, week by week
Four weeks is enough if the preparation is real and too short if it is not. The shape below assumes the rubric and the licences are settled before week one starts.
A four-week evaluation, drawn to scale
- Week 1 · scope, rubric, pass mark5 d
- Week 2 · the eight targets, conclusions hidden5 d
- Week 3 · the runs and the challenge pass5 d
- Week 4 · claim checks and the readout5 d
Two scheduling traps are worth naming. The claim checks in week four take longer than anyone plans — budget two full days of one scientist, not an afternoon. And the disagreement review needs the original reviewers in the room; booking them in week four is too late, because they will be in a laboratory.
What a pass looks like, written down in advance
A pass is three numbers and one judgement, all agreed in week one. The numbers are your citation-resolution threshold, the fall in days you need to change how the work is done, and the share of grade disagreements you are willing to leave unexplained. The judgement is whether the scientists who checked the claims would be willing to sign an assessment built this way — which is the only question that predicts adoption.
Resist importing an industry benchmark here. Published clinical success rates, for instance, are routinely quoted as a single number and should not be: one large trial-level study of them notes that "Previous estimates of drug development success rates rely on relatively small samples from databases curated by the pharmaceutical industry and are subject to potential selection biases." If that is true of one of the most-studied numbers in drug development, it is certainly true of any benchmark a vendor offers for its own category.
The useful external anchor is not a benchmark but the killer-experiment habit from the published target-assessment recommendations, which ask teams to design the experiment that could invalidate a target rather than the one that would confirm it (GOT-IT, Nature Reviews Drug Discovery, 2021). Applied to an evaluation: decide in week one which single result would make you walk away, and make sure the protocol can produce it. For most teams it is the target they stopped. If the software grades it B, nothing else in the readout matters. The same logic sits behind how a target decision is designed.
Where the days actually go
The reason to run this evaluation at all is a claim about time, so it is worth being precise about which time. Reading and research collapse; reviewing, challenging and signing do not, and should not.
One target assessment, by hand and with agents, in working days
One sample run: 142 claims, 61 sources, 4.6 days
142claims
cited from 61 sources in the sample assessment
4.6days
from request to the finished assessment, before scientific review
38min
of agent time across the nine agents for that assessment
After go-live: the monitoring nobody plans for
The standards body's rule has two halves and organisations buy only the first. Testing before deployment is the evaluation; testing regularly while in operation is a job somebody has to own, with a cadence and a report. Set it up in week four, while the evaluation team still exists.
What to watch is a smaller list than what you measured during the pilot. The share of claims carrying a source should not drift. Days from request to signature should be stable or falling. The challenge-acceptance rate should sit somewhere sensible, and a sudden move in either direction is a signal — all challenges accepted suggests grading has got worse, none accepted suggests the challenge pass has got lazy. And the mix of sources should match what you switched on, which is also a licence check.
A month in: what a leader who was not in the pilot can see

One more thing to plan for, because it is the commonest disappointment at month six: evidence moves, and a signed assessment that is never re-read is a document, not a position. Re-checking signed work on a schedule — weekly, fortnightly or monthly — and asking the owner to confirm any grade that moves, is what keeps a portfolio current between committee meetings. Where the public resources fit alongside all of this is set out in what the free platforms answer and what they leave you. To run this protocol against a real system, start from Drug Target Assessment, which is walked through on a few of your own targets first.
Questions people ask
How do you run a pilot on your own data?
Pick six to eight targets you have already decided on, hide the conclusions from whoever runs them, connect only the sources you are licensed to use, and compare grade by grade at the end with the original reviewers present. The two things that make a pilot worthless are using targets the vendor chose and letting the evaluator see the answers first.
What data do you have to bring?
A written grading rubric, the sources you want read — licensed literature, the public genetics and target-disease portals, your internal omics, your screening and chemistry reports, your competitor extract and patent landscape — and six to eight past signed reviews to act as the answer key. Check publisher licences before the pilot starts; that is the usual cause of a two-week delay.
How long should a software evaluation take?
Four weeks of elapsed time once the rubric and licences are settled, of which about two days are the claim checks that actually decide it. Shorter than that and you are watching a demonstration; much longer and the original reviewers lose interest and the comparison never happens.
Who needs to be involved in an evaluation?
Five roles: a scientist who checks claims against passages, a therapeutic-area lead who grades against the rubric, a safety reader for the weak safety grades, someone from information management for sources and licences, and an owner who will still be accountable in a year. The standards framework's point about independent review applies: the person who championed the purchase should not be the only one reporting on it.
How do you compare software against your current process?
On the same targets, with the same rubric, and on three axes rather than one: claims that resolve to a real passage, grades that agree after challenge, and working days from request to signature. Measure your current process honestly first — most teams discover that their own claim-to-source rate is not 100% either, which changes the conversation.
What does it take to go live?
Beyond the technical work: the rubric agreed across therapeutic areas rather than within one, the source list and its licences settled, the signature rules decided — who signs, what a signature means, what routes a weak safety grade to a second reader — and a named owner for the monthly monitoring. Plan the monitoring while the evaluation team still exists, because it will not be rebuilt later.