WorkflowMedical Affairs

Buying · pilot and go-live

AI medical information platform: how to pilot it and go live

The only pilot that settles the question is a replay: draft letters you have already signed, from the same sources, and compare them sentence by sentence against what was sent. This guide sets out the replay, the six checkpoints between it and a live desk, what a validation colleague signs against, what changes for the specialist on day one — and what should make you stop.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

An AI medical information platform is software that takes a healthcare professional's question from the inbox to a reply a specialist can sign — reading and splitting the question, finding the evidence, composing the letter with a source on every paragraph, guarding any off-label part and checking every sentence. Botminds' Medical Information Response is one. How you pilot it decides more than which one you choose, because a demonstration on a vendor's curated library tells you about the vendor's library and nothing about yours.

The pilot design that settles it is a replay. Take letters your own department has already written, checked and signed; give the system the same sources that were available at the time; let it draft; then compare its draft against what was sent, sentence by sentence. Your signed letters are the answer key, and they are the only answer key that reflects your products, your markets, your templates and your reviewers' judgement. Everything below is how to run that, and what has to be true before the result is allowed to touch a live inbox.

The shape of a desk the pilot has to be measured against

1,184

questions answered in 30 days, of which 236 — one in five — needed a custom letter

2,124

sentences checked in the same 30 days, each carrying a source

41

questions submitted by field teams, every one confirmed with the requester before drafting

20%

of questions that no single standard response covered

Medical Information Response on its fictional sample data over 30 days — the shape of what a pilot measures, not a result on anyone's department. The people, products and letters in these examples are sample data in the solution. Your own baseline is the equivalent count from your case system for the same period, and you should have it before the pilot starts.

What an AI medical information platform has to prove

Four things, in this order, and they are not equally hard. That it can draft a letter at all is the easy one and the one demonstrations are built around. That it draws on your current approved sources, at the right version, for the right country, is harder. That it handles the off-label part the way your procedures require — split per part of the question, disclosure attached, advice struck, second signature — is harder still. And that a specialist reading the output spends their time deciding rather than reconstructing is the one that decides whether the department adopts it or quietly goes back to the word processor.

Notice what is missing from that list: speed. Time to draft is easy to measure and easy to improve, and it is rarely the constraint. The constraint is how long it takes a named person to be willing to sign, which is a function of how much of the letter they have to re-derive. Measure that instead.

What does a replay pilot actually do?

It re-runs history with the clock held still. Pick a sample of signed letters — fifty is enough to see the shape and small enough for two reviewers to work through — and for each one feed the system the question as it arrived and the sources as they stood on the day. It drafts. You then compare, sentence by sentence, against the letter that actually went out.

Three design rules make the difference between a replay and a demonstration with extra steps. Sample across the work rather than across the easy parts: stratify by product, by country template, by channel and by whether the letter contained off-label content. Freeze the sample before anyone looks at the output, because a sample edited during the pilot measures the editing. And use the sources as they stood: a draft built from today's library against a letter signed last year will disagree for reasons that have nothing to do with the system.

What the comparison yields is not a pass mark but four verdicts per sentence, and the interesting work is in the middle two.

Four verdicts on one sentence, and what each one tells you

Matched

The drafted sentence rests on the same passage as the signed one and says the same thing.

The ordinary case. A high share here means the library and the retrieval agree with your specialists.

Corrected

The draft used a different passage, or a newer version, and the difference is defensible.

Read every one. This is where a pilot earns its keep — and where a retired version in the signed letter shows up.

Struck

The signed letter contained a sentence the draft did not produce, or the guard removed one the draft had.

Both directions matter. A struck advisory sentence is a win; a struck safety limit is a stop.

Added

The draft contains a factual sentence the signed letter did not.

Judge it on the question asked. An addition that answers something nobody asked fails the tailoring test.

The replay produces one of these per factual sentence. A pilot report that gives only the first figure has thrown away the information you ran the pilot to get.

The corrected column is where a replay most often pays for itself before go-live. A draft that reaches for a newer version of a standard response than the signed letter used has not made an error; it has found one in the archive. A department running this for the first time may well discover sent letters resting on wording that was retired between drafting and signature — the failure mode checking an answer sentence by sentence is built to catch.

What a medical information AI pilot should measure, and what it should not

Two numbers and one list. The numbers are a replay score and a reviewer-effort count; the list is every disagreement, kept for argument.

The replay score, and the number that actually decides

+

Factual sentences

every sentence in the signed letter that asserts something checkable

The denominator, taken from the signed letter rather than from the draft, so a short draft cannot score well by saying less.

Fixed before the pilot starts

+

Matched sentences

same passage, same claim, same numbers as what was sent

The numerator. Version differences are not matches; they go in the corrected column and are read individually.

Two reviewers, marking together

+

Replay score

matched ÷ factual

Useful for comparing two systems on the same frozen sample. Useless as an absolute: it is as much a measure of your library’s consistency as of the system’s.

The programme owner

−

Sentences to decide

corrected + struck + added, per letter

The number a specialist meets on the morning of go-live. If it is above a handful per letter, the department will not adopt it, whatever the score says.

The specialist signing

Report the last term per letter, not averaged across the sample. A mean of two hides the letter with eleven, and the letter with eleven is the one that tells you which part of the library is not ready.

Keep the disagreements as a list, not as a total. Each one is a sentence, a passage, what was sent and what was drafted, and your two reviewers' marks. That list is what makes the go-live conversation an argument about evidence instead of an argument about confidence — and it is also where you find out how often your own reviewers disagree with each other, which is the ceiling on any number you report afterwards.

What you bring: procedures, templates, the library, past inquiries

A replay needs real material, and assembling it is usually the longest part of the preparation. None of it is exotic; all of it has an owner who has to agree to release it.

What you bring on day one

  1. Your procedures, as written

    Who may sign a custom letter, who adds the second signature on off-label content, how questions from patients are handled, and what the agents may cite. These become settings; if they are not written down, the pilot will encode somebody’s memory of them.

  2. The letter templates for every market in scope

    A US template is not a UK template, and a Canadian pilot needs both English and French. Running a replay in one market and going live in three is how a go-live gets rolled back.

  3. The standard response library, with its version history

    Not the current documents — all of them, with approval dates, review dates, owners and retirements. Retired versions are what make the version check meaningful, and a library flattened to “current only” cannot support a replay against a letter signed two years ago.

  4. The label set for each country, at the revisions in force

    USPI, SmPC and Product Monograph, with revision dates, so a replay can use the label that applied on the day rather than the one in force now.

  5. Fifty past inquiries and their signed letters

    Stratified by product, country, channel and off-label content, and including the awkward ones: three-part questions, badly worded questions, and questions a field colleague relayed.

  6. Literature and data on file, with the licensing settled

    Peer-reviewed articles from your subscription and reports approved for external use. Confirm that your subscription permits the retrieval and internal reuse a drafting layer performs — before, not during.

  7. The requester directory and the consent register

    Verification that the requester is a healthcare professional, their consent to reply, and how they want the answer. Without it the request check is decoration.

  8. Two reviewers with a week between them

    Not one. The point of two is the subset they both mark, which tells you how much of any difference is your own disagreement rather than the system’s.

Medical Information Response is built and deployed from exactly this material — your procedures, templates, standard response library and a sample of past inquiries. The list is also a useful audit in its own right: a department that cannot assemble it has a library problem before it has an AI problem.

Six checkpoints between a pilot and a live desk

A pilot that succeeds does not become a live desk by decision. It becomes one by passing a sequence, each step owned by a named person who can refuse it. The sequence below is ours, offered as a starting point; the ownership column is the part that matters most.

A replay pilot, week by week, with the six checkpoints marked

Preparation3 weeks
Assemble the material and settle the licensing2 w
Freeze the sample1 w · CP1
  • Assemble the material and settle the licensing2 w
  • Freeze the sample1 w · CP1
The replay3 weeks
Procedures as settings1 w · CP2
Draft the fifty and mark them2 w · CP3
  • Procedures as settings1 w · CP2
  • Draft the fifty and mark them2 w · CP3
Read the result2 weeks
Walk every disagreement2 w · CP4
  • Walk every disagreement2 w · CP4
To the desk6 weeks
Validation evidence and sign-off2 w · CP5
Shadow running on live questions, nothing sent3 w
Live on one product1 w · CP6
  • Validation evidence and sign-off2 w · CP5
  • Shadow running on live questions, nothing sent3 w
  • Live on one product1 w · CP6
An indicative fourteen weeks for one market and one product family, drawn to scale. The shadow-running phase is the one most tempting to cut: it is the only phase where the system meets questions nobody chose for it. Lanes run in sequence, not in parallel.

The six checkpoints, who holds each one, and the evidence it needs

CheckpointWho can refuse itThe evidence it needs
CP1 — The sample is representative and frozenThe MI managerThe stratification, the count per stratum, and a note that nothing has been added or removed since the freeze.
CP2 — Procedures are settings, and they are rightThe MI managerA readback of who signs, the second signer, both clocks, the patient-question route and what may be cited — compared against the written procedure line by line.
CP3 — Every drafted sentence carries a passage, version and sectionThe specialist reviewersOne letter opened at random: for any sentence a reviewer picks, the passage, its version and its section, immediately.
CP4 — Every disagreement has been read by two reviewersThe specialist reviewersThe disagreement list with both marks, and a written disposition for each: acceptable, a library problem, or a system problem.
CP5 — The validation and audit evidence stands upQuality or validationThe stated role of the system, the audit trail on one complete letter, a human-readable export, the signature linkage demonstrated, and the access boundaries enforced.
CP6 — Shadow running produced nothing a reviewer would not have signedThe medical reviewerEvery shadow letter with off-label content, read in full, with the struck sentences and their reasons.
Each checkpoint is a person with a veto rather than a status in a plan. CP5 is where a project that skipped the audit trail discovers it, which is why it sits before shadow running rather than after it.

How do you validate an AI drafting tool?

Against a stated role, with evidence gathered for that role in particular. Write one page naming what the system does — drafts, cites, guards, checks — and what it does not: it does not sign, does not decide a finding, does not answer a patient, and does not hold the record of truth. Everything else in the validation pack is evidence for that statement.

Where the letters are electronic records signed electronically, the controls are specified. They include a system validated to be accurate and reliable and able to spot records that are invalid or altered, and secure, time-stamped audit trails the computer writes on its own whenever someone creates, changes or deletes a record — with the rule that a change must not hide what was recorded before (21 CFR 11.10). The last phrase is the one to test rather than read: open one letter's history and check that the drafted sentence, the finding, the decision and the final wording all survive, each with a name and a time.

One borrowed discipline is worth the trouble, with its caveat stated plainly. FDA's January 2025 draft guidance on AI used to support regulatory decision-making is built on a context of use — the particular job and boundaries the model has for the question at hand — and on credibility evidence gathered for that role, with planned life-cycle maintenance afterwards because of “the potential for the model’s performance to change over time or across deployment environments”.

The guidance also draws its own boundary. It leaves out AI used in drug discovery, and AI used for operational efficiency — internal workflows, resource allocation, drafting a submission — where that use does not affect patient safety, drug quality or the reliability of study results (FDA draft guidance, January 2025).

So on its face it does not govern medical information letters, and nobody should cite it as a requirement for one. Borrow the shape — name the role, gather evidence for that role, plan who looks at performance afterwards and how often — and leave the authority where it belongs. What a validation colleague asks for before go-live sets out the evidence itself.

How long does going live actually take?

The honest answer is that the software is rarely the longest part. Medical Information Response is built and deployed from your requirements and documents — your procedures, templates, standard response library and a sample of past inquiries — and goes live once every quality checkpoint has passed, walked through on your own inquiries first. What governs the calendar is how long the material takes to assemble, how long your reviewers can give, and how long validation takes in your organisation.

The fourteen weeks in the timeline above is indicative for one market and one product family, and three parts of it stretch most often: licensing for the literature join, which is a contract rather than a configuration; a standard response library flattened to current versions, which has to be reconstructed before a replay is possible; and validation sign-off, which has its own queue. None of these is a reason to shorten the shadow-running phase, which is the one that earns the go-live.

Nobody should promise you a date before seeing the library. A vendor who does has not asked how many of your standard responses are overdue for review.

What changes for the specialist on day one

Less than they fear and more than they expect. The question still arrives on the case system. The clock is still theirs. What changes is what they open: not an empty template beside a search box, but a composed letter in the right country template, one section per part of the question in the order asked, every factual sentence carrying the passage it came from, with the off-label section already checked in order — disclosure, evidence, limits, safety — and any advisory sentence struck with its reason visible.

Their work becomes decision work. Read the letter. Open the one or two sentences with findings against them and accept, reject or rewrite each. Read the off-label section properly, because it is the part with rules attached. Sign, with the meaning of the signature recorded. On the sample desk that letter went out the same morning it arrived, with three references and the Prescribing Information enclosed, and the second signature from the medical reviewer in between.

Two things do not change, and saying so on day one prevents most of the resistance. Agents draft and never sign. And a finding is never accepted by the thing that raised it — a person takes every one. What a medical information specialist signs for sets out the signature itself.

The signing route, and the meaning of the signature

A route-to-sign screen: the checks on one line — eight of eight sentences cited, the off-label section built from scientific evidence, one sentence removed by the guard, three references attached — then the signing route showing the specialist first, then the medical reviewer for the off-label section, then secure email to the requester with the case closed and archived. The meaning of the signature is shown above the password field.
Medical Information Response on its sample data. The signature records the meaning, the name and the time; the second signature on the off-label section is a step in the route rather than a line in a procedure.

Who signs the system off, and against what evidence

Three people, and they sign for different things. The MI manager signs that the procedures are correctly expressed as settings and that the sample was representative — their evidence is the readback at CP2 and the stratification at CP1. The specialist reviewers sign that the drafts are traceable and that the disagreements have been read — their evidence is the disagreement list with two marks and a disposition on each. Quality or validation signs that the system behaves as its stated role claims and that the record will survive inspection — their evidence is the audit trail, the export, the signature linkage and the access boundaries.

The medical reviewer is the fourth signature and the one that should be last, because it is specific to the content rather than to the system: every shadow letter containing off-label material, read in full, with the struck sentences and their reasons. If that reading produces nothing they would have signed differently, go live. If it produces one, read the next twenty.

What to measure after go-live, and what to stop measuring

Stop measuring the replay score. It was a decision instrument for a frozen sample, and once letters are live it measures nothing — there is no answer key any more, because the drafts become the letters. Keep the measures that describe the work: volume and turnaround, the share of questions needing a custom letter, the share of sentences carrying a source, and the field-submitted requests confirmed with the requester before drafting. That last one is a compliance measure in operational clothing, and a number an auditor is likely to ask for.

Add two the pilot cannot give you. The first is findings per letter over time: a rising number usually means the library is drifting out of date rather than that the system is degrading, and it points at which standard responses are overdue. The second is the recurring-question count — topics with four or more custom letters in 90 days and no current standard response. On the sample desk three topics crossed that line in a quarter, and each one is a standard response the library should have had, which is a cheaper fix than answering the same question six times.

And keep looking at the model's performance on purpose rather than by exception. Name who looks, how often, and what they would act on. The reason is not regulatory: it is that a quiet system is indistinguishable from a system nobody is checking.

The desk at a glance, 30 days in

A dashboard showing volume, custom letters, turnaround, sentences carrying a source and field-submitted requests confirmed, with a flow of today's questions from how they arrived — phone, web portal, email, field medical, chat — through how they were answered to where they are now.
Medical Information Response on its sample data. Volume, turnaround, citation coverage and the field-submitted confirmations are the four that stay useful after go-live; the replay score is not among them.

What would make you stop

Write these down before the pilot starts, when nobody is invested. Five are worth stopping for.

A sentence that cannot show its passage. If, at CP3, a reviewer picks a sentence and the system cannot immediately show the passage, the version and the section, the mapping is being reconstructed rather than kept, and nothing later in the sequence fixes that.

An off-label section that advises. One sentence recommending an unapproved dose that no control removed is not a tuning problem; it is the failure the whole arrangement exists to prevent. The right response is to stop and find out which control was absent, not to add a reviewer.

Answers that move on their own. Ask the same question twice, a fortnight apart, as part of the pilot. A system whose answer changes without a change you authorised cannot be validated, because there is nothing stable to validate.

A retired version cited as current. One is a finding; a pattern means the library join is reading by document name rather than by approved revision, and the system will launder old wording at scale.

Reviewers spending longer per letter, not less. If sentences-to-decide stays high after the disagreements have been dispositioned, the department is reconstructing rather than deciding, and adoption will fail quietly a month after go-live. Better to find that in week six.

One more, which is a reason to pause rather than stop: if your own two reviewers disagree with each other more than they disagree with the system, the pilot has found something real about your procedures, and the right next step is to fix those before judging anything else. Published comparisons of AI answers to drug information questions have found expert scoring itself to be variable — in one, agreement between evaluators was a Fleiss' kappa of 0.015 (J Am Coll Clin Pharm 2026) — and another, which found that hardly any chatbot answer reached its top content band, drew its conclusion about the continuing necessity of pharmacist expertise rather than about a score (J Am Coll Clin Pharm 2026).

The approach the published medical information work recommends is the same one a pilot should encode: a process with points along the way at which human writers step in and strengthen what the AI has drafted (JMIR AI 2025). The checkpoints in the table above are that idea applied to a project rather than to a document.

If you want the replay run rather than described, that is the offer: your own past inquiries and the standard responses behind them, drafted and compared against what you actually sent. Start from Medical Information Response, or read how the four kinds of medical information software compare first if you are still deciding what to pilot.

Questions people ask

How do you pilot an AI medical information platform?

By replay. Freeze a stratified sample of fifty inquiries you have already answered and signed, give the system the sources as they stood on the day, let it draft, and compare sentence by sentence against what was sent. Mark each factual sentence as matched, corrected, struck or added, and have two reviewers mark a shared subset so you know how much of the disagreement is your own. A demonstration on a vendor's content cannot tell you any of this.

What data does a medical information pilot need?

Your written procedures, the letter templates for every market in scope, the standard response library with its full version history including retirements, the label revisions in force for each country, fifty past inquiries with their signed letters, literature and data on file with the licensing settled, and the requester directory and consent register. The version history is an item often missing, and a library flattened to current versions makes a replay against older letters impossible.

How long does a medical information platform take to go live?

Indicatively fourteen weeks for one market and one product family: three weeks of preparation, three on the replay, two reading the result, two on validation evidence, three of shadow running and one to go live on a single product. The software is rarely the longest part — literature licensing, reconstructing the library's version history and the validation queue stretch more often. Nobody should give you a date before seeing the library.

What changes for the specialist on day one?

They open a composed letter in the right country template rather than an empty one, with each section answering one part of the question in the order asked and every factual sentence carrying its passage. The off-label section has already been checked in order and any advisory sentence struck with its reason. Their work becomes deciding findings, reading the off-label section and signing — rather than searching, pasting and re-deriving.

What should you measure after go-live?

Volume and turnaround, the share of questions needing a custom letter, the share of sentences carrying a source, and the field-submitted requests confirmed with the requester before drafting. Add findings per letter over time, which usually points at a library going out of date rather than at the system, and the count of topics with four or more custom letters in 90 days and no standard response. Stop measuring the replay score: once the drafts become the letters, there is no answer key left.

What would make you stop the project?

A sentence whose passage, version and section cannot be shown on demand; an off-label section that advises and no control removed it; answers that change when the same question is asked a fortnight later; a pattern of retired versions cited as current; or reviewers spending longer per letter after the disagreements have been resolved. Write these down before the pilot starts, while nobody is invested in the answer.

Medical Information Response.
See it on your own files.

An agentic solution from Botminds. Composes the reply to a healthcare professional’s question paragraph by paragraph, each paragraph tagged with its approved source, with off-label parts built from scientific evidence only, for a medical information specialist to sign. Book a demo, or explore how it works.