WorkflowMedical Affairs

Governance · the line, and how to test it

AI for MLR review: what it can decide, and what it must not

Nobody is going to approve your reviewer: FDA's draft AI guidance says it does not address AI used for operational efficiencies such as internal workflows, which is where promotional review checking most plausibly sits. What governs is unchanged and is enough — and it draws a clean line between the comparing an agent may do and the judging that stays with named people. This guide draws that line, and sets out how to test the thing before it touches a live piece, as MLR Review Automation is built to be tested.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

AI for MLR review is best understood as a division of labour rather than a product category. The comparing in medical, legal and regulatory review is mechanical: open the cited reference at its page, check the value, the population, the comparator and the timepoint, compare the safety text with the current label word for word, measure how prominent risk is beside benefit. The judging is not: whether a comparative claim is fair, whether a finding can be dismissed, whether a piece may go out. Botminds' MLR Review Automation puts six agents on the first list and nothing on the second — they never approve and never sign. This guide sets out why that line sits where it does, what rules actually govern, and how to test an agent against your own past pieces before it sees a live one.

Start with the question most teams ask first, because the answer removes an obstacle rather than creating one.

Can AI do MLR review?

It can do the checking and it cannot do the reviewing, and the two are not the same work. A reviewer's signature is a statement that a named qualified person examined the piece and concluded it was accurate, supported and consistent with the label. Nothing can make that statement on their behalf. What can be done before they look is every comparison the conclusion rests on, so that the reviewer is reading findings with the evidence open beside each one rather than opening fourteen references by hand for a four-page detail aid.

That is also the honest answer to the question behind the question. The constraint is not technical capability; it is accountability. Somebody has to own the conclusion, and ownership is only meaningful if the owner can see what it was based on.

What an agent can check at the page

The checks that can be done mechanically share one property: each has a right answer that a person can verify in seconds by looking at a page. That is the test for whether a check belongs on this list.

One claim through the checks, and the line each agent must not cross

  1. Agent

    Find the claim

    Every claim, statistic and reference anchor on the piece, pinned where it sits, with value, comparator, endpoint and timepoint recorded.

    Never: paraphrases — the claim is quoted as written

  2. Agent

    Open the cited page

    The reference opened at its anchored page and the supporting passage found; an anchor pointing at the wrong page is corrected.

    Never: reports a verdict without naming the page and passage

  3. Agent

    Compare the four fields

    Value, population, comparator and timepoint, each compared between the piece and the source and marked matched or not.

    Never: treats a near match as a match

  4. Agent

    Name the analysis

    Primary endpoint, pre-specified key secondary, exploratory, or descriptive subgroup — stated for every result a claim rests on.

    Never: records an exploratory or subgroup result as supported efficacy

  5. Agent

    Compare the label

    Claims and safety information checked word for word against the label version marked current, naming the section and any missing warning.

    Never: uses a superseded label version

  6. Agent

    Measure prominence

    Share of words, type size, contrast, placement and, for audio, words a second, each as a number against the threshold the team set.

    Never: passes or fails a piece on a reading

  7. A person decides

    Decide each finding

    A named reviewer accepts the change, sends it back to the agency, or dismisses it — with a written reason where the finding is high.

  8. A person decides

    Sign, with the meaning

    Medical, regulatory and legal reviewers sign; each signature records the signer, the time and what the signature means.

    Never: a filing package built before every required signature

How MLR Review Automation divides the work; the stated limits under each agent step are the product's own. The green steps are the two the regulation and the guidance put in human hands.

Where it has to stop, and why

Three decisions sit on the far side of the line, and they share a property too: each requires weighing something against a standard that has no single right answer.

The first is whether a comparative or superiority claim is fair. An agent can establish that a market-share report covers one prescriber setting, one product class and one quarter while the claim implies all three without limit. Whether the qualified wording is adequate, and whether the claim should be made at all, is a legal judgement about how a reader will take a sentence.

The second is whether a finding may be dismissed. A dismissal is a reviewer saying the check was right about the facts and wrong about the significance, which is precisely the judgement that cannot be delegated. It should cost something: in MLR Review Automation a dismissal of a high finding requires a written reason before it takes effect.

The third is approval. Nothing in a piece's record should be able to say approved except a person's signature, and nothing should be able to assemble the filing package before every required signature is in.

Twelve checks: what an agent may conclude, what it must hand over, and who owns the decision

The checkWhat an agent may concludeWhat it must escalateWho owns it
Claim extractionThese are the claims, quoted as written, at these positions.A sentence it cannot classify as a claim or not.Agent, verifiable at a glance
Reference anchoringThe cited page does or does not contain the passage; here is the corrected page.A reference it could not open or index.Agent
Value comparisonThe number in the piece matches, or does not match, the number in the source.A number derived by a calculation it cannot reproduce.Agent
Population comparisonThe piece's population is wider, narrower or the same as the study's.A population described too loosely in either place to compare.Agent, reviewer confirms
Comparator comparisonThe comparator differs: placebo, active, another class, or none.An implied comparison with no comparator named on the piece.Reviewer
Timepoint comparisonWeek 12 is not week 52; the piece cites a different timepoint from the source.A claim with no timepoint where the source has several.Agent
Analysis typeThe result was a primary, key secondary, exploratory or descriptive subgroup analysis.Always, where anything but primary or key secondary carries an efficacy claim.Reviewer decides; the agent never calls it supported
Label consistencyThe safety text matches, or does not match, the current label word for word; this warning is missing.A difference in wording that may or may not change meaning.Agent finds, regulatory reviewer decides
ProminenceShare of words, type size, contrast, placement, words a second — six readings against the threshold.Always: the reading is evidence, not a verdict.Reviewer
Comparative and superiority wordingThe data covers this population, this comparator and this period.Always: whether the claim is fair.Legal reviewer
Dismissing a findingNothing. It may record the dismissal and its reason.Always.Reviewer, with a written reason where the finding is high
Approval, and the filing packageNothing. It may build the package once the last signature is in.Always.Medical, regulatory and legal signatories
The division MLR Review Automation is built on. Read the right-hand column as the answer to the only question that matters in a quality review: when this is wrong, whose name is on it?

Does the FDA AI guidance apply to promotional review?

Not on its face, and the draft guidance says why in a sentence worth reading before any procurement conversation. FDA's January 2025 draft guidance on artificial intelligence to support regulatory decision-making is about AI that produces information or data intended to support decisions on the safety, effectiveness or quality of a drug. Its scope paragraph then excludes uses that look like an internal review workflow — with a condition attached.

In the draft's own scope, it does not cover AI used in drug discovery, nor AI used for operational efficiency — internal workflows, resource allocation, drafting a submission — where that use does not affect patient safety, drug quality or the reliability of study results (FDA, draft guidance, January 2025).

Note the condition: the exclusion covers uses that do not impact patient safety, drug quality or the reliability of study results, and a firm should be able to say why its checker fits — which is easiest when the agent checks and a named person decides. Two more things follow, and they point in opposite directions. There is no credential to wait for, no submission to make and no approval to claim: a vendor saying its checker is FDA-approved, or compliant with the AI guidance, is describing something that does not exist. And there is no external standard to hide behind either. The credibility of the thing is yours to establish and yours to document.

The draft guidance is still useful for how it frames that work. It treats credibility as trust in how a model performs for one particular context of use, earned by collecting evidence — and the context of use as the specific role and scope the model plays in answering a question (FDA, draft guidance, January 2025). Narrow the context of use and the evidence becomes gatherable: not "is this model trustworthy" but "does it find the cited page for claims of this kind in pieces of this kind, and how often".

What does govern: the review, the record and the signature

Three unchanged rules carry all the weight, and together they are enough. The first is the standard itself: promotional material must be truthful and not misleading, and FDA's guidance on communications consistent with the labeling recommends careful review by qualified medical or scientific, legal and regulatory people, before dissemination, to make sure a communication is not false or misleading (FDA, June 2018). The recommendation names people and qualifications. Nothing in it is satisfied by a system.

The second is what a signature on an electronic record has to carry. 21 CFR 11.50(a) requires a signed electronic record to show who signed, when, and what the signature meant — the regulation's own examples are review, approval, responsibility and authorship. A signature that records only that a button was pressed does not meet it — and a meaning is exactly what an agent cannot supply, because meaning is a claim about what a person did.

The third is the record itself: every check, every accepted change, every dismissal and every signature kept with who and when, and every change to the routing, the signers or the checks versioned. What a reviewer's signature commits them to follows that chain through the committee.

How do you test an AI claims checker?

Against pieces where you already know the answer. The NIST AI Risk Management Framework states the requirement plainly, and it settles most arguments about demonstrations.

In NIST's terms, an accuracy figure means little unless it comes with a clearly defined test set that is realistic and representative of how the system will actually be used, and with the test method written down alongside it (NIST AI 100-1, section 3.1).

Representative of conditions of expected use means your pieces, in your formats, citing your references — not a vendor's curated set. And because your conditions move, the measurement has to be repeated. The FDA draft guidance names the reason: a model's performance can drift over time, or between settings, when the inputs it sees stop resembling the data it was trained on — which is why it calls for maintenance across the model's life. A new brand, a new reference format, a new channel: each is a new input distribution.

Designing the test: eight decisions to make before anything is run

  1. Choose twenty pieces you already approved

    Across the formats you actually review — detail aid, HCP email, deck, broadcast script — and across more than one brand, so the result is not a single product's vocabulary.

  2. Pull their references exactly as they were cited

    Including the data on file. A test on pieces whose references were tidied up afterwards measures a world you do not work in.

  3. Pull the findings you recorded at the time

    These are the answers. Where the record is thin, ask the reviewer who signed to re-read the claim, and note that you did.

  4. Run it blind, and keep the output

    No tuning between pieces, no hand-fed page numbers. If someone helps it find a reference, that piece is out of the test.

  5. Compare claim by claim on the four fields

    Value, population, comparator, timepoint — plus whether the analysis type was named correctly. Count agreements and both kinds of disagreement.

  6. Report the misses as well as the hits

    Claims it missed entirely, pages it could not open, findings it raised that a reviewer rejected. The last number is the one that predicts whether reviewers will keep using it.

  7. Disaggregate the result

    By format, by brand and by reference type. NIST notes that accuracy measurements may include disaggregation for different data segments; a single average hides that broadcast scripts or data-on-file citations perform differently.

  8. Write down the method, and schedule the re-run

    The test set, how it was chosen, what was measured and when. Then repeat it when a new brand, format or reference source arrives — the drift the FDA draft guidance warns about is not hypothetical.

The eight steps are ours; the requirements for a defined, realistic test set and for documented methodology, and the note on disaggregated results, are from NIST AI 100-1 section 3.1, and the life-cycle maintenance point is from FDA's January 2025 draft guidance.Source: NIST AI RMF 1.0; FDA draft guidance, January 2025

The baseline it has to beat

A checker is not being compared with perfection. It is being compared with what reaches print after human review, and the peer-reviewed audits of journal advertising have measured that repeatedly.

What has reached print, as the audits measured it

44.1%

of promotional claims not supported by the reference cited, in advertisements in six Spanish medical journals in 1997

18%

of the references in that study could not be retrieved at all, from monographic works and unpublished data on file

49.4%

of US physician-directed journal advertisements were non-adherent to at least one FDA guideline item, November 2008

28.9%

failed to present adequate quantification of efficacy

First two: Villanueva P, Peiró S, Librero J, Pereiró I, The Lancet 2003;361:27-32, on 125 promotional claims carrying references. Second two: Korenstein D, Keyhani S, Mendelson A, Ross JS, PLoS ONE 2011;6(8):e23336, on 83 unique full advertisements; a further 18.1% were adherent to all FDA guideline items. Both measure journal advertising in the year stated, and both are old enough that they describe a baseline rather than a current rate.Source: Villanueva et al., The Lancet, 2003; Korenstein et al., PLoS ONE, 2011

The failure these audits kept finding is instructive about what to test for. It was rarely an invented number; it was a real number carried into a population, a comparator or a timepoint the study did not cover. That is a comparison failure, and comparison is what a checker is for. What a claim can rest on, and how often cited references fail, works through the published audits.

Can an agent judge fair balance?

It can measure fair balance and it must not judge it, and the regulation is the reason. 21 CFR 202.1(e)(7)(viii) asks that risk be as prominent and as readable as the benefit material, judged on factors such as typography, layout, contrast, headlines, paragraphing and white space. Those factors are measurable; "reasonably comparable" is a judgement about a whole piece.

So the useful output is a set of readings, each of which a reviewer can dispute, rather than a verdict they can only accept or override. A threshold is a team's own setting, not a regulatory number, and a reading below it is a reason to look rather than a finding of fact.

Prominence as six readings a reviewer can argue with, not a verdict

A detail aid on the left with its safety block shaded, and a fair balance panel on the right headed 'Risk is less prominent than benefit'. A dial shows a balance reading of 41 against a line at 70 described as the team's threshold, with a note that a reviewer makes the call and the reading is evidence, not a verdict. Below, a words-on-the-page bar reads benefit 86 per cent, risk 14 per cent, then a list of readings: share of words on risk 14 per cent; risk type size 8 pt against 22 pt; contrast of risk text 45 per cent; placement, last page after references; risk beside the safety claim, no; current label safety text, rev 03/2025; direction to the full Prescribing Information, none. A footnote cites 21 CFR 202.1 and points to the threshold setting.
MLR Review Automation on sample data; every brand, person and piece named here is fictional. Each row is a measurement the reviewer can check against the piece. The reading itself decides nothing — which is why the panel says so on the screen.

The same logic applies in the other direction, to a claim an agent says is supported. FDA's own reasoning about a single-arm study makes the point: with no comparator arm, it called the reported disease-free survival results — a time-to-event endpoint — uninterpretable (warning letter to ImmunityBio, Inc., March 2026). That conclusion is reachable from the study design and the endpoint type, which is why it belongs to the checker. Whether to run the claim anyway, qualified, belongs to the reviewers. What FDA keeps citing in its letters collects the rest of the arguments the agency actually makes.

Who is accountable when an agent is wrong

The signatory, which is why the signature has to record its meaning and the finding has to show its page. NIST puts the dependency in two short sentences: AI can only be trusted if someone is accountable for it, and no one can be accountable for what they cannot see. In a review process that cashes out as three concrete properties. Every verdict names the page and the passage it rests on, so the reviewer can check it rather than trust it. Every decision records who made it and when, including a dismissal and its written reason. And every change to the checks themselves — a new rule, a changed threshold, a different signer — is versioned, so a piece approved last quarter can be read against the rules that applied then.

The practical test of all three is a question from a quality review a year later: show me why this claim was approved. A process that answers with the page, the comparison, the reviewer and the meaning of their signature has nothing to reconstruct.

The signature that carries its meaning, with the findings all decided first

A sign-off dialogue over a reviewed piece. It names the piece and version, states 'Findings: 10 accepted, 0 with agency, 0 dismissed, 0 open', and shows the meaning of the signature: 'I reviewed this piece for medical accuracy and confirm each claim is supported by the cited evidence and consistent with the current label.' A password field sits below, and the button reads 'Sign as Dr. Maya Chen'. Behind the dialogue, a panel lists the three required signatures — medical, regulatory, legal — with two waiting, and a 2253 package whose items are all marked pending until approval. A record below shows timestamped entries naming who accepted which finding.
MLR Review Automation on sample data; the reviewers and the piece are fictional. Two details are the whole argument of this page: the signature states what it means, and the filing package stays pending until every required signature is in.

What changes in a medical reviewer's week

Less than the marketing suggests, and more than reviewers expect. The volume of judgement does not fall; the volume of fetching does.

A reviewer's week, before the checking arrives done and after

Opening references by handChecking arrives done
The first hour on a piece

Opening references by hand: Finding the references, matching each claim to one, opening PDFs and hunting for the passage.

Checking arrives done: Reading findings, each with its page open and the four fields compared.

What gets checked

Opening references by hand: What time allows. The twelfth reference gets less attention than the first.

Checking arrives done: Every claim, in the same way, in the order the piece presents them.

What the reviewer argues about

Opening references by hand: Whether the number is right.

Checking arrives done: Whether the claim is fair, which is the thing they are qualified for.

The committee agenda

Opening references by hand: Every piece, because nobody can tell in advance which ones are routine.

Checking arrives done: The pieces with open judgement calls; the routine ones took one signature.

What a signature means

Opening references by hand: The same words, resting on what one person had time to check.

Checking arrives done: The same words, resting on a record of what was checked and against which page.

What does not change

Opening references by hand: The reviewer decides each finding and signs.

Checking arrives done: The reviewer decides each finding and signs.

The right-hand column describes how MLR Review Automation is designed to be used; it is not a measured result. The last row is the point of the comparison.

Reviewers are the people best placed to tell whether a checker is any good, and the number that says so is the share of its findings they keep. A tool whose findings are routinely dismissed is not saving anyone time; it is adding a queue. How to pilot MLR review automation, and what to measure, makes that one of the four numbers, and MLR Review Automation shows the checking on a piece.

Questions people ask

Is AI allowed in regulatory review of promotional material?

Nothing prohibits it, and nothing authorises it either. FDA's January 2025 draft guidance on AI for regulatory decision-making explicitly does not address AI used for operational efficiencies such as internal workflows that do not impact patient safety, drug quality or the reliability of study results — which is where a checking tool that leaves every decision to people most plausibly sits. What applies is unchanged: the piece must be truthful and not misleading, qualified medical, scientific, legal and regulatory people should review it before it goes out, and an electronic signature must record its own meaning.

What can AI check in a promotional piece?

Anything with a verifiable right answer: the claims on the piece and where they sit, whether the cited page contains the passage, whether the value, population, comparator and timepoint match the source, what kind of analysis the result came from, whether the safety text matches the current label word for word, and how prominent risk is beside benefit as a set of measurements. Each of those can be confirmed by a person in seconds, which is the test for whether it belongs on the list.

Who is accountable when AI reviews a claim?

The person who signs. That is why an electronic signature has to carry the printed name, the date and time and the meaning of the signature under 21 CFR 11.50(a), and why every finding should name the page and passage it rests on. As NIST puts it, accountability presupposes transparency: a reviewer can only own a conclusion they can check.

Will AI replace medical reviewers?

No. The work that can be automated is the fetching and comparing; the work that cannot is deciding whether a claim is fair and signing to say so. What changes is the proportion: a reviewer who spent most of a piece opening references spends it on the findings that need a medical judgement, and the routine pieces stop taking up committee time.

What audit trail does an AI reviewer need?

Every agent check with its result and the page it cites; every accepted change, every dismissal with its written reason where the finding was high, and every signature, each with who and when; and a version history for the checks themselves, the routing thresholds and the signers, so a piece approved last quarter can be read against the rules in force then. Without the last of those, a change to a threshold quietly rewrites the history of every earlier decision.

How do you know an AI claims checker is accurate?

Measure it on a defined, realistic test set drawn from your own approved pieces, with the findings you recorded at the time as the answers, and document the method — which is what NIST's framework asks of any accuracy claim. Report results disaggregated by format, brand and reference type, and repeat the measurement when new kinds of input arrive, because performance drifts when the inputs do.

MLR Review Automation.
See it on your own files.

An agentic solution from Botminds. Every claim in a promotional piece pinned on the piece itself and checked against the cited reference page and the current label, so medical, legal and regulatory reviewers start from the evidence. Book a demo, or explore how it works.