WorkflowMedical Affairs

Investigator-Initiated Study Review

How accurate is AI review of investigator-initiated study proposals?

Accuracy in investigator-initiated study (IIS) review is five numbers, not one: score agreement, overlap recall, citation fidelity, enrolment forecast error and compliance sensitivity. Each can be measured on your own past concepts before go-live, and Botminds' Investigator-Initiated Study Review is built so that every one of them can be counted.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

Investigator-initiated study review is the medical affairs work of judging the study concepts that independent investigators send a drug or device company for funding or product: does the concept fit published research priorities, is the design sound, can the site recruit, is the budget at fair market value, is it free of commercial influence, and does an existing study already answer the question? Botminds' Investigator-Initiated Study Review gives the preparation to AI agents that score each concept, search for overlapping studies and build a cited packet, while people decide and sign. Whoever builds the tool, the question buyers type, how accurate is AI grant review, has a short published answer and a longer local one, and "is the AI accurate?" is the wrong way to ask it.

The right way is to name the five errors that would hurt a committee and count each one. Does the agent's criterion score agree with a careful reviewer about as well as two reviewers agree with each other? Of the concepts that duplicate an ongoing study, how many does the overlap search find? Does each cited passage actually say what the finding claims? How far were the enrolment forecasts from what funded sites delivered? And of the concepts with a real compliance red flag, how many were held? All five can be measured on concepts your committee has already decided.

Medical affairs has barely begun to use AI on evidence work

~10%

of medical affairs leaders surveyed use generative AI for medical evidence and education

Half

of medical affairs organisations do not use natural language processing for literature review

>80%

say a lack of internal capabilities and capacity holds back digital adoption

2024 MAPS Benchmark Report on digital, advanced analytics and AI in medical affairs, a small survey of 34 completed responses. With so little in use, and no peer-reviewed evaluation of AI applied to industry IIS review that we could find, the measures below have to be run on your own programme.Source: Medical Affairs Professional Society, 2024 Benchmark Report

How accurate is AI grant review? What the first studies found

As a judge of whole proposals, not very yet; as a checker of specific flaws, useful but uneven. The two most direct tests, both published in March 2026, point the same way: a language model's overall score tracks an expert panel only weakly, while a model asked to examine one section at a time finds planted problems far more reliably than one asked for a single verdict.

What the first tests of AI grant review measured, and what each means for IIS review

StudyWhat was testedWhat it foundWhat it means for IIS review
Sandström and Thelwall, 2026Language-model scores against expert panel scores for Swedish Medical Council postdoctoral fellowship applications from 1994A weak positive correlation with the panel’s average, mean Spearman 0.22Do not ask a model for the overall verdict; a weak correlation cannot rank a cycle
Thorne et al., 2026Six UK research council proposals, each deliberately degraded on six axes: funding, timeline, competency, alignment, clarity and impactSection-by-section review beat single-pass on detection and scoring reliability; a multi-persona “council” did no better than baseline; alignment problems were caught, clarity flaws largely missed; variability was highNarrow checks per criterion, each tested on planted flaws, and repeated runs to measure variability
Liang et al., 2023Points raised by GPT-4 against points raised by human reviewers of research papersOverlap with a human reviewer close to the overlap between two humansUseful as a source of review points; it measured feedback, not funding decisions
None of these studies tested industry investigator-initiated study review, and we found none that did. The method each used, not its number, is what transfers.Source: arXiv:2603.14565; arXiv:2603.08281; arXiv:2310.01783

The details matter for anyone buying. Sandström and Thelwall found the models' scores "correlated weakly but positively" with the experts, a mean Spearman correlation of 0.22. Thorne and colleagues found feedback that human assessors judged largely valid but that leaned toward compliance checking rather than holistic assessment, which is a fair description of where software helps a committee and where it does not. Both results argue for the design this page measures: separate agents for separate criteria, each finding cited, and the verdict left to people.

Why "is the AI accurate?" is the wrong question for IIS review

Because a single accuracy figure rewards the wrong behaviour. Most concepts are declined, so a tool that agrees with past decisions can look excellent while recognising nothing. In a peer-reviewed series of thoracic oncology concepts sent to pharmaceutical sponsors, 15 of 60 were approved and 43 were declined (Gerber et al., Clin Cancer Res 2025). A rule that declined every concept would have matched 43 of the 58 decisions made: 74% "accuracy" from a tool that never read a page.

The decision is also the wrong unit. A committee's vote folds in strategy, budget, timing and the mood of the room, and two committees given the same concept can reasonably differ. What an agent produces is not a decision but the inputs to one: four criterion scores, an overlap finding, a feasibility projection, a budget comparison and a compliance screen, each with the passage it rests on. Each of those inputs fails in its own way and needs its own measure.

The bar is yours to set. FDA's 2025 draft guidance on AI explicitly does not address AI "used for operational efficiencies (e.g., internal workflows, resource allocation…)" that do not affect patient safety, drug quality or the reliability of study results (FDA, January 2025). No regulator will tell a medical affairs team how good its proposal screening must be. That makes a written evaluation protocol, set before the test and kept with the validation file, the only defensible answer when compliance or audit asks.

Five measures of IIS review accuracy, and the error each one catches

Score agreement, overlap recall and precision, citation fidelity, forecast error and compliance sensitivity cover what an IIS committee relies on. Each is a ratio with a numerator someone can audit, and each maps to one part of the review packet.

Five numbers that replace one accuracy claim

  1. 1 · Score agreement

    Agent criterion scores vs blind reviewer scores on the same concepts, per criterion, set beside reviewer-vs-reviewer agreement. Catches a scorer that drifts from the rubric.

  2. 2 · Overlap recall and precision

    Known duplicates found ÷ known duplicates; true overlaps ÷ everything flagged. Catches the concept that repeats your own phase 3 and reaches the committee anyway.

  3. 3 · Citation fidelity

    Findings whose cited passage supports the claim ÷ all findings. Catches the confident summary built on a passage that says something else.

  4. 4 · Enrolment forecast error

    Projected vs actual months to full enrolment for studies you funded. Catches the optimistic site that recruits a third of its promise.

  5. 5 · Compliance sensitivity

    Red-flag concepts held ÷ red-flag concepts. Catches research initiated by sales or tied to prescribing. The one measure where the target is a miss rate of zero.

Measure 1 needs reviewers' time; measures 2, 3 and 5 can be tested with planted cases; measure 4 needs studies that have finished enrolling.

The five are not equally weighted. A score that is half a point off moves a concept a few places in the ranking, and the reviewer confirms every score anyway. A missed duplicate sends the committee a study the company is already paying for. A missed red flag is the kind of error regulators have prosecuted. Set the bar for each measure by the cost of its error, not by what the vendor's demonstration happened to achieve.

How much do human grant reviewers agree with each other? The baseline for score agreement

Less than committees assume. When 43 reviewers rated the same 25 NIH applications in a replica of an NIH study section, their ratings showed no agreement on quality, and the outcome depended more on the reviewer assigned than on the research proposed (Pier et al., PNAS 2018). So an agent's criterion score is good enough when it agrees with a reviewer about as well as two qualified reviewers agree with each other. Perfect agreement is the wrong target, because reviewers themselves do not reach it. The largest public comparison of language-model and human review comments found the model's points overlapped a human reviewer's about as much as two humans' overlapped each other (Liang et al., 2023, later NEJM AI).

An AI reviewer overlapped a human about as much as two humans overlapped each other

Nature journals · GPT-4 vs a human reviewer
30.85%
Nature journals · human vs human
28.58%
ICLR · GPT-4 vs a human reviewer
39.23%
ICLR · human vs human
35.25%
Share of review points raised by one reviewer that the other also raised, over 3,096 Nature-family papers and 1,709 ICLR submissions. These are research papers, not study concepts; the lesson for IIS review is the method, not the number.Source: Liang et al., arXiv:2310.01783

Translate that to IIS review in three steps. First, measure per criterion, not on the total: priority fit, scientific quality and feasibility fail differently, and a total can hide two errors that cancel. Second, use a statistic that respects order, such as a weighted kappa or the mean absolute difference on the 1–5 scale, and report the reviewer-vs-reviewer figure on the same concepts beside it. Third, weight the tolerance by the committee's own weights. With the committee default of 30 · 30 · 20 · 20, a one-point disagreement on scientific quality moves the 100-point score by six points; the same disagreement on feasibility moves it by four.

The trap is anchoring. Once a reviewer has seen the agent's 3.9, their own score is no longer independent, and "the reviewer confirmed it" says little about agreement. Agreement must be measured blind: reviewers score a set of concepts before opening the packet, or on past concepts they scored before any agent existed. After go-live, the honest signal is the opposite one, the override rate, covered below.

Rank agreement matters as much as score agreement. A committee funds a cut, not a score, so check how many of the concepts in the reviewers' top group also land in the agent's top group, and whether any concept moves across the funding line. In a sample cycle run through the solution, one concept scores 74 and ranks 7th before the overlap check, then drops to 62 and 9th of 9 once a near-identical company study is found. That is a correct move, but it shows how much of the ranking one measure can carry. How reviewers score a concept sets out the rubric being measured.

Overlap recall and precision: plant duplicates you already know about

Overlap recall is the share of genuinely duplicative concepts the search flags; precision is the share of flags that are genuine. The fastest way to measure both is to plant cases. Take concepts your committee declined for duplicating an ongoing study, add variants that change one element (a different comparator, a later timepoint, another region), and run them through the search with the answer hidden.

The stakes are legal as well as scientific. HHS-OIG lists research "that is unnecessarily duplicative or is not needed by the manufacturer for any purpose other than the generation of business" among its indicia of questionable research (OIG Compliance Program Guidance, 2003), and practitioner guidance has long said a "good tracking system [is] necessary to identify overlap" (MAPS, 2018). A missed duplicate is a scientific waste and a weak spot in the compliance file at once.

A seeded overlap test, scored: recall protects the committee, precision protects the reviewer

Confusion matrix for a seeded overlap test300 past concepts, 40 of them planted duplicates. The search flags 36 duplicates and misses 4. Of the 260 concepts that are not duplicates, it flags 12 and passes 248. Recall is 36 divided by 40, 90 percent. Precision is 36 divided by 48, 75 percent.THE SEARCH SAYSOverlapNo overlapPlantedduplicate40 concepts36found4missedNot aduplicate260 concepts12false alarms248passedRECALL36 ÷ (36 + 4) = 90%4 reach the committeePRECISION36 ÷ (36 + 12) = 75%12 checked for nothing
Illustrative numbers to show the arithmetic, not a measured result. Run the same table on your own planted duplicates, and run it again at each alert threshold the programme might choose.

Three refinements make the test honest. Score at the element level too: the overlap finder in Investigator-Initiated Study Review compares population, intervention, comparator, outcome, timepoint and region as same, partly or different, and each call can be checked against a reviewer's reading. Measure recall at more than one alert threshold, because the threshold is a dial: the product's default is 0.6, with 0.5 and 0.7 as settings, and lowering it buys recall with false alarms. And measure coverage separately. Recall can never exceed what was searched; in the sample cycle the search covers 412 studies (96 company, 61 funded investigator, 255 public registry), and a duplicate of a study that is not in that set will be missed however good the matching is.

Both sides of an overlap call are cited, so a reviewer can check the match

The overlap section of a review packet. A grid compares the concept with three studies across population, treatment, control, outcome, timing and region, marked Same, Partly or a dash, with overlap scores 0.86, 0.54 and 0.38. A tooltip on the company study's outcome cell reads: Outcome same, endoscopic response, SES-CD decrease of at least 50% from baseline. On the right, the company registry entry for that study is open with the matching endpoint highlighted and the note: Internal record, never shared with investigators.

This is what makes element-level scoring cheap: an evaluator checks each Same or Partly call against two highlighted passages instead of re-reading two protocols.

Sample data (fictional company and studies). Clicking an Outcome · Same cell opens the matching study's registry entry with the endpoint highlighted, beside 'What is the same' and 'What this proposal adds'.

How to check a study concept for overlap explains the element-by-element method itself; this page is about proving that it works.

Citation fidelity: does the cited passage support the finding?

Citation fidelity is the share of findings whose cited passage actually says what the finding claims. It is the measure that makes the other four auditable, and it has four levels. Counting only the first, "has a citation", is the commonest mistake in AI evaluation.

The fourth level matters most for concept forms, which are short and often incomplete. A model under pressure to fill a structured field will invent a plausible power calculation. In the solution, the concept reader reports a missing field as missing rather than filling it; on the sample concept it extracted 44 fields, each cited, and named the power calculation and ethics plan as absent. Plant concepts with deleted sections and count how often the gap is reported.

A citation in the reviewer summary opens the passage it rests on

A review packet showing the reviewer summary and decision. The drafted summary carries seven numbered citations, followed by four conditions for the investigator and four recommendation choices. On the right, the concept form is open with the primary objective highlighted: to estimate the rate of endoscopic response, a decrease in SES-CD of at least 50% from baseline, at week 12.

An evaluator samples citations straight from this view: click, read the highlight, mark it supported or not. That is the support level of the audit, at a few seconds a citation.

Sample data. The summary drafted for the reviewer carries seven citations; citation 1 opens the concept form at the primary objective. Tabs above the document switch to the cover email, budget sheet, site history, priorities and the overlapping studies.

Grade support with people, not with a second model. An LLM asked whether a passage supports a claim is another model with its own error rate, which you would then have to evaluate. Two reviewers marking a few hundred sampled citations, with disagreements settled by a third, gives a figure an auditor can re-check.

Enrolment forecast error: score the projections against what sites delivered

A feasibility projection is accurate when the months it predicted to full enrolment are close to the months the funded study actually took. This is the slowest measure, because the answer arrives years later, and the most neglected. It is also where investigator studies most often fail: among randomised trials approved by six ethics committees, 26.5% of investigator-sponsored trials were discontinued for poor recruitment against 8.4% of industry-sponsored ones (Kasenda et al., JAMA 2014).

Back-test it on studies you have already funded. For each, run the projection as it would have been made at submission, using only the site's enrolment history up to that date, and compare it with the actual rate. Report the median error in months and, more usefully, calibration: of the studies projected to take 24 months or less, how many did? Compare the result with the baseline that matters, which is the site's own request. In the sample concept, the investigator asks for 60 patients in 18 months, 3.3 a month; the site enrolled 1.1 a month in its last two studies with 35% screen failure, and the projection is about 55 months.

Count the scenarios too. A forecast that offers "add a second site" or "reduce to 40 patients" can be checked when a funded study later took one of those routes. Forecasting recruitment from the site's own history covers the projection method.

Compliance sensitivity: why a missed red flag costs more than a false alarm

For the compliance screen, the measure that matters is sensitivity: of the concepts that carry a red flag, how many were held for a person to review. Specificity matters less, because a false alarm costs a compliance officer a short review, while a miss can cost the company a prosecution.

The red flags are well documented. In 2020 Pacira Pharmaceuticals paid $3.5 million to resolve allegations that it used "bogus research grants" as kickbacks; the grants "were typically initiated by Pacira sales representatives or marketing executives", and the company "required that EXPAREL be placed on formulary at the physician's institution before awarding any research grant" (U.S. Attorney, District of New Jersey, 2020; allegations only, no admission of liability). Each of those is a pattern a screen can be tested on.

The two errors of a compliance screen are not symmetric

False alarm: a clean concept is heldMiss: a tainted concept passes
Who notices

False alarm: a clean concept is held: Compliance, the same day, when they open the hold.

Miss: a tainted concept passes: Possibly nobody until an audit, a whistleblower or an investigation.

What it costs

False alarm: a clean concept is held: A review and a short delay to scientific review of one concept.

Miss: a tainted concept passes: Under the Anti-Kickback Statute, up to $100,000 and 10 years per violation, plus civil settlements.

How to test

False alarm: a clean concept is held: Count holds on concepts compliance later cleared.

Miss: a tainted concept passes: Plant concepts with a named sales colleague, a formulary condition, promotional wording or a company sponsor.

Target

False alarm: a clean concept is held: Low enough that compliance reads every hold.

Miss: a tainted concept passes: Zero misses on every planted pattern.

Plant each red-flag pattern several times in different wording, in the concept, the cover email and the budget notes. A screen that only catches the phrase it was shown has not been tested.Source: 42 U.S.C. 1320a-7b(b); USAO-NJ on Pacira, 2020

Design matters as much as the measure. In the solution the compliance screener can hold a concept but never clear one; a held concept is not ranked until a compliance officer decides, and switching the commercial-involvement screen off requires compliance approval. That design makes a false alarm cheap by construction, so the screen can be tuned for sensitivity. Research, not reward covers the compliance controls in full.

A held concept goes to a person, with the reason and the time

A compliance review dialog over a review packet. A red banner reads: A sales colleague is named in the cover email and the study is linked to prescribing experience. Two timed lines follow: Oct 5, 11:40, compliance screener held the concept; Oct 5, 11:41, the compliance officer opened a review and asked the field team for the contact log. Buttons: Close and Message the compliance officer.
Sample data. A concept held because the cover email names a sales colleague; the screener's hold and the compliance officer's review are both timed. These timed holds are the numerator of the sensitivity measure in live use.

Back-test on last year's decisions, then run one shadow cycle

The evaluation runs in two stages: a back-test on concepts you have already decided, then one live cycle in which the agents run alongside reviewers without influencing them. The back-test gives numbers quickly; the shadow cycle shows how the agents behave on concepts no one has pre-judged.

An evaluation plan that measures the agents without letting them mark their own work

  1. A person decides

    Build the golden set

    Two to four past cycles: concepts, reviewer scores, decisions, known duplicates, compliance holds and, for funded studies, actual enrolment.

  2. A person decides

    Blind second scoring

    A second reviewer scores a subset without the original scores, giving the reviewer-vs-reviewer band.

  3. Agent

    Back-test

    Agents score, search, project and screen every concept in the set.

    Never: sees the decision or the reviewers’ scores

  4. A person decides

    Plant and adjudicate

    Add planted duplicates, red flags and deleted sections; settle every disagreement against the documents.

  5. Agent

    Shadow cycle

    Agents run on one live cycle; reviewers score first, then open the packet.

The programme lead, compliance and QA then accept the five figures against thresholds they wrote down before the back-test, so the result cannot be argued into a pass afterwards.

Past decisions are a reference, not an answer key. Where the agent and the original reviewer disagree, settle it against the documents, and expect cases where the reviewer was the one who missed something: a duplicate nobody remembered, a budget line above benchmark. Size the set for the rare errors. If a planted test finds zero misses in n cases, the upper bound of the 95% confidence interval on the true miss rate is roughly 3/n, so 30 planted red flags with no miss still allow a miss rate of about 10%. For compliance, plant more. The go-live plan places the shadow cycle in the rollout.

Keep the test confidential by design. NIH prohibits generative AI in its own peer review because "using generative AI technologies requires the sharing of material from applications" (NIH, 2023). Investigator concepts are unpublished research ideas, and a back-test sends years of them through the system at once. Run it in the environment that will run production, never in a public chatbot.

After go-live: watch the override rate and re-test when anything changes

Once agents are live, the best accuracy signal is how often reviewers change what they prepared. Track overrides per criterion and per reviewer: a criterion that reviewers change on a third of concepts is a rubric problem, while one reviewer who never changes anything may be anchoring. Track which overlaps the committee dismissed, which held concepts compliance cleared, and, as funded studies enrol, the forecast error.

Drift in IIS review rarely comes from the model. It comes from the inputs: a new version of the published research priorities, a new year's fair-market-value benchmarks, a reweighted scorecard, a registry that has not been refreshed. In the solution each of these is a versioned setting with its change in the audit trail, and a packet can be re-run against the new version. Treat every such change as a trigger to re-run the back-test on the golden set and compare the five figures with the last run.

The dials that change what 'accurate' means are versioned settings

A settings page headed: Rules the agents apply, who reviews, and how the committee decides. Every change is versioned in the audit trail. Rows: scorecard weights 30, 30, 20, 20; overlap alert with options 0.5, 0.6 and 0.7, 0.6 selected; minimum priority fit; overhead cap 25%; enrolment from site history, on; commercial-involvement screen, on; budget benchmarks, 2026 benchmarks by country v3; then scientific reviewers by therapeutic area.
Sample data. Weights, the overlap alert threshold, the overhead cap, the commercial-involvement screen and the benchmark set are all settings with a version history. A change to any of them is a reason to re-run the back-test.

What to hand compliance and QA, then: the evaluation protocol and the date its thresholds were set; the golden set and the planted cases; the five figures with their denominators; and, for each live cycle, the override rates and the holds. The design that makes these countable, cited findings, people who sign and versioned settings, is set out on the Investigator-Initiated Study Review page.

Questions people ask

Should reviewers see the AI's scores before they score a concept?

In live use, yes: the point is to save their time, and they confirm or change every score. During evaluation, no. Agreement can only be measured on scores reviewers gave without seeing the agent's, so blind scoring belongs to the back-test and the shadow cycle.

Can we use past committee decisions as ground truth?

Only partly. Decisions mix science with strategy and budget, and most concepts are declined, so agreement with decisions is easy to inflate. Use decisions to check rank agreement around the funding line, and use reviewer criterion scores, planted cases and actual enrolment for the rest.

Does an AI tool for IIS review need formal validation?

FDA's 2025 draft guidance on AI does not cover internal workflow tools that do not affect patient safety, drug quality or study results, so no specific regulation sets the bar. Your quality system and compliance programme do. A written protocol, measured results and a re-test whenever inputs change is what an auditor will expect to see.

How many past concepts does a back-test need?

Enough of each kind of error to count. To pin an overlap recall of about 90% within roughly ±10 points at 95% confidence takes some 35 known duplicates, more than most programmes decline in a year. That is why the overlap and compliance tests plant cases you construct, while score agreement can run on a few dozen ordinary past concepts.

Which measure should we run first?

Compliance sensitivity and overlap recall: both can be tested in days with planted cases, and both protect against the errors that cost most. Score agreement needs reviewer time, and forecast error needs studies that have finished enrolling.

Investigator-Initiated Study Review.
See it on your own files.

An agentic solution from Botminds. Scores every investigator's study concept on priority fit, scientific quality, feasibility and overlap with existing studies, ranks the cycle on one scorecard, and builds a cited packet for the committee to decide and sign. Book a demo, or explore how it works.