An AI medical information platform is software that takes a healthcare professional's question from the inbox to a reply a specialist can sign — reading and splitting the question, finding the evidence, composing the letter with a source on every paragraph, guarding any off-label part and checking every sentence. Botminds' Medical Information Response is one. How you pilot it decides more than which one you choose, because a demonstration on a vendor's curated library tells you about the vendor's library and nothing about yours.
The pilot design that settles it is a replay. Take letters your own department has already written, checked and signed; give the system the same sources that were available at the time; let it draft; then compare its draft against what was sent, sentence by sentence. Your signed letters are the answer key, and they are the only answer key that reflects your products, your markets, your templates and your reviewers' judgement. Everything below is how to run that, and what has to be true before the result is allowed to touch a live inbox.
The shape of a desk the pilot has to be measured against
1,184
questions answered in 30 days, of which 236 — one in five — needed a custom letter
2,124
sentences checked in the same 30 days, each carrying a source
41
questions submitted by field teams, every one confirmed with the requester before drafting
20%
of questions that no single standard response covered
What an AI medical information platform has to prove
Four things, in this order, and they are not equally hard. That it can draft a letter at all is the easy one and the one demonstrations are built around. That it draws on your current approved sources, at the right version, for the right country, is harder. That it handles the off-label part the way your procedures require — split per part of the question, disclosure attached, advice struck, second signature — is harder still. And that a specialist reading the output spends their time deciding rather than reconstructing is the one that decides whether the department adopts it or quietly goes back to the word processor.
Notice what is missing from that list: speed. Time to draft is easy to measure and easy to improve, and it is rarely the constraint. The constraint is how long it takes a named person to be willing to sign, which is a function of how much of the letter they have to re-derive. Measure that instead.
What does a replay pilot actually do?
It re-runs history with the clock held still. Pick a sample of signed letters — fifty is enough to see the shape and small enough for two reviewers to work through — and for each one feed the system the question as it arrived and the sources as they stood on the day. It drafts. You then compare, sentence by sentence, against the letter that actually went out.
Three design rules make the difference between a replay and a demonstration with extra steps. Sample across the work rather than across the easy parts: stratify by product, by country template, by channel and by whether the letter contained off-label content. Freeze the sample before anyone looks at the output, because a sample edited during the pilot measures the editing. And use the sources as they stood: a draft built from today's library against a letter signed last year will disagree for reasons that have nothing to do with the system.
What the comparison yields is not a pass mark but four verdicts per sentence, and the interesting work is in the middle two.
Four verdicts on one sentence, and what each one tells you
Matched
The drafted sentence rests on the same passage as the signed one and says the same thing.
The ordinary case. A high share here means the library and the retrieval agree with your specialists.
Corrected
The draft used a different passage, or a newer version, and the difference is defensible.
Read every one. This is where a pilot earns its keep — and where a retired version in the signed letter shows up.
Struck
The signed letter contained a sentence the draft did not produce, or the guard removed one the draft had.
Both directions matter. A struck advisory sentence is a win; a struck safety limit is a stop.
Added
The draft contains a factual sentence the signed letter did not.
Judge it on the question asked. An addition that answers something nobody asked fails the tailoring test.
The corrected column is where a replay most often pays for itself before go-live. A draft that reaches for a newer version of a standard response than the signed letter used has not made an error; it has found one in the archive. A department running this for the first time may well discover sent letters resting on wording that was retired between drafting and signature — the failure mode checking an answer sentence by sentence is built to catch.
What a medical information AI pilot should measure, and what it should not
Two numbers and one list. The numbers are a replay score and a reviewer-effort count; the list is every disagreement, kept for argument.
The replay score, and the number that actually decides
Factual sentences
every sentence in the signed letter that asserts something checkable
The denominator, taken from the signed letter rather than from the draft, so a short draft cannot score well by saying less.
Fixed before the pilot starts
Matched sentences
same passage, same claim, same numbers as what was sent
The numerator. Version differences are not matches; they go in the corrected column and are read individually.
Two reviewers, marking together
Replay score
matched ÷ factual
Useful for comparing two systems on the same frozen sample. Useless as an absolute: it is as much a measure of your library’s consistency as of the system’s.
The programme owner
Sentences to decide
corrected + struck + added, per letter
The number a specialist meets on the morning of go-live. If it is above a handful per letter, the department will not adopt it, whatever the score says.
The specialist signing
Keep the disagreements as a list, not as a total. Each one is a sentence, a passage, what was sent and what was drafted, and your two reviewers' marks. That list is what makes the go-live conversation an argument about evidence instead of an argument about confidence — and it is also where you find out how often your own reviewers disagree with each other, which is the ceiling on any number you report afterwards.
What you bring: procedures, templates, the library, past inquiries
A replay needs real material, and assembling it is usually the longest part of the preparation. None of it is exotic; all of it has an owner who has to agree to release it.
What you bring on day one
- Your procedures, as written
Who may sign a custom letter, who adds the second signature on off-label content, how questions from patients are handled, and what the agents may cite. These become settings; if they are not written down, the pilot will encode somebody’s memory of them.
- The letter templates for every market in scope
A US template is not a UK template, and a Canadian pilot needs both English and French. Running a replay in one market and going live in three is how a go-live gets rolled back.
- The standard response library, with its version history
Not the current documents — all of them, with approval dates, review dates, owners and retirements. Retired versions are what make the version check meaningful, and a library flattened to “current only” cannot support a replay against a letter signed two years ago.
- The label set for each country, at the revisions in force
USPI, SmPC and Product Monograph, with revision dates, so a replay can use the label that applied on the day rather than the one in force now.
- Fifty past inquiries and their signed letters
Stratified by product, country, channel and off-label content, and including the awkward ones: three-part questions, badly worded questions, and questions a field colleague relayed.
- Literature and data on file, with the licensing settled
Peer-reviewed articles from your subscription and reports approved for external use. Confirm that your subscription permits the retrieval and internal reuse a drafting layer performs — before, not during.
- The requester directory and the consent register
Verification that the requester is a healthcare professional, their consent to reply, and how they want the answer. Without it the request check is decoration.
- Two reviewers with a week between them
Not one. The point of two is the subset they both mark, which tells you how much of any difference is your own disagreement rather than the system’s.
Six checkpoints between a pilot and a live desk
A pilot that succeeds does not become a live desk by decision. It becomes one by passing a sequence, each step owned by a named person who can refuse it. The sequence below is ours, offered as a starting point; the ownership column is the part that matters most.
A replay pilot, week by week, with the six checkpoints marked
- Assemble the material and settle the licensing2 w
- Freeze the sample1 w · CP1
- Procedures as settings1 w · CP2
- Draft the fifty and mark them2 w · CP3
- Walk every disagreement2 w · CP4
- Validation evidence and sign-off2 w · CP5
- Shadow running on live questions, nothing sent3 w
- Live on one product1 w · CP6
The six checkpoints, who holds each one, and the evidence it needs
| Checkpoint | Who can refuse it | The evidence it needs |
|---|---|---|
| CP1 — The sample is representative and frozen | The MI manager | The stratification, the count per stratum, and a note that nothing has been added or removed since the freeze. |
| CP2 — Procedures are settings, and they are right | The MI manager | A readback of who signs, the second signer, both clocks, the patient-question route and what may be cited — compared against the written procedure line by line. |
| CP3 — Every drafted sentence carries a passage, version and section | The specialist reviewers | One letter opened at random: for any sentence a reviewer picks, the passage, its version and its section, immediately. |
| CP4 — Every disagreement has been read by two reviewers | The specialist reviewers | The disagreement list with both marks, and a written disposition for each: acceptable, a library problem, or a system problem. |
| CP5 — The validation and audit evidence stands up | Quality or validation | The stated role of the system, the audit trail on one complete letter, a human-readable export, the signature linkage demonstrated, and the access boundaries enforced. |
| CP6 — Shadow running produced nothing a reviewer would not have signed | The medical reviewer | Every shadow letter with off-label content, read in full, with the struck sentences and their reasons. |
How do you validate an AI drafting tool?
Against a stated role, with evidence gathered for that role in particular. Write one page naming what the system does — drafts, cites, guards, checks — and what it does not: it does not sign, does not decide a finding, does not answer a patient, and does not hold the record of truth. Everything else in the validation pack is evidence for that statement.
Where the letters are electronic records signed electronically, the controls are specified. They include a system validated to be accurate and reliable and able to spot records that are invalid or altered, and secure, time-stamped audit trails the computer writes on its own whenever someone creates, changes or deletes a record — with the rule that a change must not hide what was recorded before (21 CFR 11.10). The last phrase is the one to test rather than read: open one letter's history and check that the drafted sentence, the finding, the decision and the final wording all survive, each with a name and a time.
One borrowed discipline is worth the trouble, with its caveat stated plainly. FDA's January 2025 draft guidance on AI used to support regulatory decision-making is built on a context of use — the particular job and boundaries the model has for the question at hand — and on credibility evidence gathered for that role, with planned life-cycle maintenance afterwards because of “the potential for the model’s performance to change over time or across deployment environments”.
The guidance also draws its own boundary. It leaves out AI used in drug discovery, and AI used for operational efficiency — internal workflows, resource allocation, drafting a submission — where that use does not affect patient safety, drug quality or the reliability of study results (FDA draft guidance, January 2025).
So on its face it does not govern medical information letters, and nobody should cite it as a requirement for one. Borrow the shape — name the role, gather evidence for that role, plan who looks at performance afterwards and how often — and leave the authority where it belongs. What a validation colleague asks for before go-live sets out the evidence itself.
How long does going live actually take?
The honest answer is that the software is rarely the longest part. Medical Information Response is built and deployed from your requirements and documents — your procedures, templates, standard response library and a sample of past inquiries — and goes live once every quality checkpoint has passed, walked through on your own inquiries first. What governs the calendar is how long the material takes to assemble, how long your reviewers can give, and how long validation takes in your organisation.
The fourteen weeks in the timeline above is indicative for one market and one product family, and three parts of it stretch most often: licensing for the literature join, which is a contract rather than a configuration; a standard response library flattened to current versions, which has to be reconstructed before a replay is possible; and validation sign-off, which has its own queue. None of these is a reason to shorten the shadow-running phase, which is the one that earns the go-live.
Nobody should promise you a date before seeing the library. A vendor who does has not asked how many of your standard responses are overdue for review.
What changes for the specialist on day one
Less than they fear and more than they expect. The question still arrives on the case system. The clock is still theirs. What changes is what they open: not an empty template beside a search box, but a composed letter in the right country template, one section per part of the question in the order asked, every factual sentence carrying the passage it came from, with the off-label section already checked in order — disclosure, evidence, limits, safety — and any advisory sentence struck with its reason visible.
Their work becomes decision work. Read the letter. Open the one or two sentences with findings against them and accept, reject or rewrite each. Read the off-label section properly, because it is the part with rules attached. Sign, with the meaning of the signature recorded. On the sample desk that letter went out the same morning it arrived, with three references and the Prescribing Information enclosed, and the second signature from the medical reviewer in between.
Two things do not change, and saying so on day one prevents most of the resistance. Agents draft and never sign. And a finding is never accepted by the thing that raised it — a person takes every one. What a medical information specialist signs for sets out the signature itself.
The signing route, and the meaning of the signature

Who signs the system off, and against what evidence
Three people, and they sign for different things. The MI manager signs that the procedures are correctly expressed as settings and that the sample was representative — their evidence is the readback at CP2 and the stratification at CP1. The specialist reviewers sign that the drafts are traceable and that the disagreements have been read — their evidence is the disagreement list with two marks and a disposition on each. Quality or validation signs that the system behaves as its stated role claims and that the record will survive inspection — their evidence is the audit trail, the export, the signature linkage and the access boundaries.
The medical reviewer is the fourth signature and the one that should be last, because it is specific to the content rather than to the system: every shadow letter containing off-label material, read in full, with the struck sentences and their reasons. If that reading produces nothing they would have signed differently, go live. If it produces one, read the next twenty.
What to measure after go-live, and what to stop measuring
Stop measuring the replay score. It was a decision instrument for a frozen sample, and once letters are live it measures nothing — there is no answer key any more, because the drafts become the letters. Keep the measures that describe the work: volume and turnaround, the share of questions needing a custom letter, the share of sentences carrying a source, and the field-submitted requests confirmed with the requester before drafting. That last one is a compliance measure in operational clothing, and a number an auditor is likely to ask for.
Add two the pilot cannot give you. The first is findings per letter over time: a rising number usually means the library is drifting out of date rather than that the system is degrading, and it points at which standard responses are overdue. The second is the recurring-question count — topics with four or more custom letters in 90 days and no current standard response. On the sample desk three topics crossed that line in a quarter, and each one is a standard response the library should have had, which is a cheaper fix than answering the same question six times.
And keep looking at the model's performance on purpose rather than by exception. Name who looks, how often, and what they would act on. The reason is not regulatory: it is that a quiet system is indistinguishable from a system nobody is checking.
The desk at a glance, 30 days in

What would make you stop
Write these down before the pilot starts, when nobody is invested. Five are worth stopping for.
A sentence that cannot show its passage. If, at CP3, a reviewer picks a sentence and the system cannot immediately show the passage, the version and the section, the mapping is being reconstructed rather than kept, and nothing later in the sequence fixes that.
An off-label section that advises. One sentence recommending an unapproved dose that no control removed is not a tuning problem; it is the failure the whole arrangement exists to prevent. The right response is to stop and find out which control was absent, not to add a reviewer.
Answers that move on their own. Ask the same question twice, a fortnight apart, as part of the pilot. A system whose answer changes without a change you authorised cannot be validated, because there is nothing stable to validate.
A retired version cited as current. One is a finding; a pattern means the library join is reading by document name rather than by approved revision, and the system will launder old wording at scale.
Reviewers spending longer per letter, not less. If sentences-to-decide stays high after the disagreements have been dispositioned, the department is reconstructing rather than deciding, and adoption will fail quietly a month after go-live. Better to find that in week six.
One more, which is a reason to pause rather than stop: if your own two reviewers disagree with each other more than they disagree with the system, the pilot has found something real about your procedures, and the right next step is to fix those before judging anything else. Published comparisons of AI answers to drug information questions have found expert scoring itself to be variable — in one, agreement between evaluators was a Fleiss' kappa of 0.015 (J Am Coll Clin Pharm 2026) — and another, which found that hardly any chatbot answer reached its top content band, drew its conclusion about the continuing necessity of pharmacist expertise rather than about a score (J Am Coll Clin Pharm 2026).
The approach the published medical information work recommends is the same one a pilot should encode: a process with points along the way at which human writers step in and strengthen what the AI has drafted (JMIR AI 2025). The checkpoints in the table above are that idea applied to a project rather than to a document.
If you want the replay run rather than described, that is the offer: your own past inquiries and the standard responses behind them, drafted and compared against what you actually sent. Start from Medical Information Response, or read how the four kinds of medical information software compare first if you are still deciding what to pilot.
Questions people ask
How do you pilot an AI medical information platform?
By replay. Freeze a stratified sample of fifty inquiries you have already answered and signed, give the system the sources as they stood on the day, let it draft, and compare sentence by sentence against what was sent. Mark each factual sentence as matched, corrected, struck or added, and have two reviewers mark a shared subset so you know how much of the disagreement is your own. A demonstration on a vendor's content cannot tell you any of this.
What data does a medical information pilot need?
Your written procedures, the letter templates for every market in scope, the standard response library with its full version history including retirements, the label revisions in force for each country, fifty past inquiries with their signed letters, literature and data on file with the licensing settled, and the requester directory and consent register. The version history is an item often missing, and a library flattened to current versions makes a replay against older letters impossible.
How long does a medical information platform take to go live?
Indicatively fourteen weeks for one market and one product family: three weeks of preparation, three on the replay, two reading the result, two on validation evidence, three of shadow running and one to go live on a single product. The software is rarely the longest part — literature licensing, reconstructing the library's version history and the validation queue stretch more often. Nobody should give you a date before seeing the library.
What changes for the specialist on day one?
They open a composed letter in the right country template rather than an empty one, with each section answering one part of the question in the order asked and every factual sentence carrying its passage. The off-label section has already been checked in order and any advisory sentence struck with its reason. Their work becomes deciding findings, reading the off-label section and signing — rather than searching, pasting and re-deriving.
What should you measure after go-live?
Volume and turnaround, the share of questions needing a custom letter, the share of sentences carrying a source, and the field-submitted requests confirmed with the requester before drafting. Add findings per letter over time, which usually points at a library going out of date rather than at the system, and the count of topics with four or more custom letters in 90 days and no standard response. Stop measuring the replay score: once the drafts become the letters, there is no answer key left.
What would make you stop the project?
A sentence whose passage, version and section cannot be shown on demand; an off-label section that advises and no control removed it; answers that change when the same question is asked a fortnight later; a pattern of retired versions cited as current; or reviewers spending longer per letter after the disagreements have been resolved. Write these down before the pilot starts, while nobody is invested in the answer.