SystemResearch & Discovery

Evaluate · agents and their limits

AI competitive intelligence in pharma: what agents may and may not do

Agents are good at the part of competitive intelligence that is reading, and measurably weaker at the part that is judgement. This guide sets out what the research says about both, and the limits Competitive Pipeline Tracking puts on each of its own six agents — including the ones a buyer should insist on.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

AI competitive intelligence in pharma means handing the reading to software and keeping the judgement: agents compare a rival's trial record against the version captured last week, keep the exact sentence from a results call, pull the numbers out of a congress abstract and check three agencies' approval databases, while a named person decides what is significant and whether anything is sent. That division is not a philosophical preference, it is what the measurements support — language models extract background facts from trials reliably and outcome data much less so. Botminds' Competitive Pipeline Tracking is built on it: six agents each anchored to a published rule, every claim carrying the passage it came from, a confidence floor below which a change goes to review, and nothing reaching a programme team until an analyst approves it.

This page is written for the competitive intelligence lead being asked to put AI on the landscape, and for the legal or compliance colleague who will ask what it is allowed to decide. The accuracy figures below are from a published evaluation of language models, not from our product; the product figures are design settings and sample data, and are labelled as such.

What language models get right, and where they stop

81.6–92.4%

of background fields extracted from trial papers with no error, across three models

27.8–80.7%

of outcome fields extracted with no error, across the same three models

85%

confidence below which a proposed change goes to an analyst for review — a setting in the solution, not a measurement

The first two figures are mean no-error proportions measured by Oami and colleagues across ChatGPT-4o, Claude 3 Sonnet and Gemini 1.5 Pro on trials behind five clinical questions in a national sepsis guideline. The third is the confidence floor configured in Competitive Pipeline Tracking.Source: Oami et al., Frontiers in Digital Health, 2026

What does AI competitive intelligence in pharma actually do?

It does four jobs that are all forms of reading. It compares: today's registry record against last week's capture, field by field. It captures: the sentence in which a sponsor said something about a watched asset. It transcribes: the numbers in an abstract, into a table where they can be lined up against other numbers. And it checks: the same handful of agency databases, every week, for the same list of assets. None of those four is a decision, which is why all four can be automated without moving the accountability for a landscape.

What it does not do is tell you anything that is not public. No agent in this design contacts a person, logs into anything, or reads a source a visitor could not read. That is the same boundary the profession's own code draws, and it is worth stating plainly to a compliance colleague: the answer to "where does the data come from" is "pages anybody can open", and the code of ethics and the legal line explains what sits on the other side of it.

The six agents of Competitive Pipeline Tracking, and the two seats a person keeps

  1. Agent

    Registry watcher

    Compares each watched trial record with the last captured version — status, phase, study start, enrolment, dates, arms, sites — with the exact text before and after.

    Never: Never infers a status the record does not state

  2. Agent

    News watcher

    Reads sponsor releases, pipeline pages and earnings-call transcripts and captures any statement about a watched asset, keeping the exact sentence.

    Never: Public sources only; quotes rather than paraphrases

  3. Agent

    Abstract reader

    Pulls design, population, endpoints and grade 3 or higher adverse events from congress abstracts into a comparable table.

    Never: Numbers are copied, never computed; interim and final are flagged

  4. Agent

    Approval watcher

    Checks agency approval databases in the United States, the European Union and Japan for approvals, filing acceptances and label changes.

    Never: Copies the indication word for word

  5. Agent

    Change detector

    Merges findings into one change per real-world event, scores significance against the watch list rules, and explains in two sentences why it matters to our programme.

    Never: Below 85% confidence it goes to review

  6. A person decides

    Analyst confirms

    Reads the old and new text side by side and decides which changes are significant for this programme.

    Never: Significance is never settled by an agent

  7. Agent

    Brief writer

    Drafts one page per programme team with every claim linked to the passage it came from.

    Never: No forecast stated as fact; never sends

  8. A person decides

    Analyst approves and sends

    Approves the page as written, which also confirms the changes it carries, and sends it to the named readers, with the name and time recorded.

    Never: Nothing is sent before approval

The green steps are the two a named person owns. The 'never' line under each agent is its own stated limit, not a description of good practice.

Six reading jobs, and why none of them is a decision

The test for whether a job can be given to an agent is whether a second competent person, given the same sources, would produce the same output. Comparing two versions of a record passes that test: the fields that differ are the fields that differ. Copying a hazard ratio out of a table passes it. Deciding whether a rival's Phase 3 start changes your own trial design does not pass it, because two competent people will reasonably disagree, and the disagreement is the value of having them.

That is why significance stays with a person even though the agent proposes it. A change detector can apply written rules — approvals, filings and label changes; pivotal readouts; phase advances; new trials in our indication; sites in our shortlist countries — and it can score how confident it is. What it cannot do is decide that this quarter, for this programme, a competitor's site opening matters more than its phase advance.

How accurate is an agent at pulling a number out of a paper?

Accurate on the easy half, unreliable on the hard half, and the gap is large enough to design around. Evaluating three models on trials behind five clinical questions in a national sepsis guideline, Oami and colleagues found that For background data extraction, mean no-error proportions ranged from 81.6% ... to 92.4% ... across models. For outcome data extraction, mean no-error proportions ranged from 27.8% ... to 80.7%, and — importantly for anyone worried about invention — that Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon (Frontiers in Digital Health, 2026).

Two findings in the same study deserve more attention than they usually get. Reproducibility was worse than accuracy: inter-session consistency ranged from 76.3% to 91.3% on background fields and from 44.8% to 65.6% on outcome fields, so at the low end the same model, asked again in a new session, agreed with its own earlier outcome extraction less than half the time. And prompt engineering strategies made only modest differences. The authors' conclusion is the one to quote at a steering group: LLMs can reliably support background data extraction in systematic reviews. However, outcome data extraction remains challenging, emphasizing the continued need for human oversight.

No-error extraction, by what is being extracted

Background fields — best model
92.4%
Background fields — weakest model
81.6%
Outcome fields — best model
80.7%
Outcome fields — weakest model
27.8%
Mean no-error proportions from Oami and colleagues, 2026, across ChatGPT-4o, Claude 3 Sonnet and Gemini 1.5 Pro on full-text trial papers. The spread between the top and bottom bar is the entire argument for copying printed numbers rather than asking a model to read a result out of a paper.Source: Oami et al., Frontiers in Digital Health, 2026

The design consequence is specific. Where an agent must handle a number, it should copy a printed one rather than derive it, and the number should be shown next to the passage it was copied from so a person can check it in two seconds. The abstract reader in this solution is built that way — numbers are copied, never computed — and that rule is a direct response to the measurement above rather than a style preference.

The published rule each watcher is anchored to

An agent that reads a source without a rule behind it is guessing in a consistent voice. Each watcher here is anchored to something published, so that both its job and its limits can be argued with. The registry watcher's anchor is 42 CFR § 11.64, which fixes which registry fields carry a 30-day deadline and which drift under a twelve-month catch-all — the clocks that decide what a weekly comparison should look at, set out in the ClinicalTrials.gov update requirements (42 CFR § 11.64). The news watcher's anchor is the ethics code's requirement of public sources and no misrepresentation. The approval watcher's is the agency record itself, copied rather than summarised.

What each agent may do, what it may never do, and what it is anchored to

AgentMayMay neverAnchored to

AgentRegistry watcher

MayCompare every watched trial record against the last captured version and record each changed field with the text before and after.

May neverInfer a status the record does not state, or read any non-public source.

Anchored to42 CFR § 11.64, which fixes what must be updated and when

AgentNews watcher

MayCapture any statement about a watched asset from a sponsor release, pipeline page or earnings-call transcript, keeping the exact sentence.

May neverParaphrase a claim, or contact anybody.

Anchored toThe SCIP code: public sources, no misrepresentation

AgentAbstract reader

MayPull design, population, endpoints and adverse events from a congress abstract into a comparable table.

May neverCompute a number the abstract does not print, or leave interim and final data unlabelled.

Anchored toThe abstract as published, nothing derived from it

AgentApproval watcher

MayCheck agency approval databases in the United States, the European Union and Japan for approvals, filing acceptances and label changes.

May neverReword an approved indication.

Anchored toThe agency record, copied word for word

AgentChange detector

MayMerge findings into one change per real-world event, score significance against the watch list rules and explain in two sentences why it matters.

May neverSettle significance alone, or pass anything below 85% confidence without review.

Anchored toThe watch list’s own written significance rules

AgentBrief writer

MayDraft one page per programme team with every claim linked to the passage it came from.

May neverState a forecast as a fact, or send anything.

Anchored toEvery claim needs a source passage; an analyst approves first

The six agents of Competitive Pipeline Tracking. The first two columns are each agent's own stated scope and limits; the third names the published rule or source the limit is derived from.

What no agent here may do

The list is short and worth reading as a specification rather than a reassurance. No agent contacts a person, by any channel. No agent logs in anywhere, which also rules out anything behind a credential — the ethics code treats obtaining a password to a competitor's site without authorisation as illegal, and so as a breach of the code, whatever the reason (SCIP, Ethics FAQs). No agent infers a trial status the record does not state. No agent computes a number that was not printed. No agent decides that a change is significant. And no agent sends anything to anybody.

Those constraints also mark where this kind of software does not help. It will not tell you about a rival's unregistered phase 1 programme, because no public source carries it. It will not reconcile two sponsors' different definitions of the same endpoint — a person has to. It will not tell you what a competitor intends. And it is weakest exactly where an analyst most wants help: turning a results table into a comparison. Buying it to replace judgement is buying the wrong thing; buying it to remove days of reading is buying what it does.

Confidence, and the line where a person confirms

A confidence score is only useful if something happens at a threshold. Here, the change detector scores each proposed change and anything below 85% confidence goes to an analyst for review rather than onto the board; above it, the change is still proposed rather than settled, and a named analyst confirms the ones marked significant. In the sample week that produced 28 changes, 6 of them significant, and two changes still awaiting confirmation at the moment the analyst came to approve the brief — which the solution tells her before she approves, because approving the page confirms them as written.

A change the detector was 99% confident about, still waiting for a person

A change feed entry comparing this week's capture with last Monday's: status from Under review (priority) to Approved — United States; indication 2L+ metastatic PDAC, KRAS G12-mutant; OS 12.4 versus 6.8 months. The change is marked 99% confident with a two-sentence explanation of why it matters for our programme, and the approval letter and the sponsor's release sit beside it with the relevant passage highlighted. An analyst has confirmed it, with her name shown.
Sample data in Competitive Pipeline Tracking. High confidence does not bypass the person: it changes how long the person spends, not whether they are there.

Why every claim has to open its own passage

Because that is what makes an AI output reviewable at all. The NIST AI Risk Management Framework puts the general case bluntly — Deployment of AI systems which are inaccurate, unreliable, or poorly generalized to data and settings beyond their training creates and increases negative AI risks and reduces trustworthiness — and names what the remedy depends on: Transparency reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system (NIST AI 100-1).

In a weekly brief, transparency has a very concrete form: the claim links to the sentence, the date links to the record version, and the brief shows its own checks before anybody approves it. The sample brief carries 14 claims and 14 linked passages. That is not a quality metric so much as a precondition — a programme lead who cannot see the sentence has no way to disagree with the claim, and a brief nobody can disagree with is not intelligence.

A general assistant asked about a rival, and an agent anchored to a rule

A general chat assistantA watcher anchored to a rule and a source version
What it reads

A general chat assistant: Whatever its training and any live search returns, with no fixed list and no guarantee of coverage.

A watcher anchored to a rule and a source version: A named list of sources for a named list of assets, every week, with each page version kept.

What it compares against

A general chat assistant: Nothing. Each answer is generated fresh, so a change is invisible unless somebody already knew about it.

A watcher anchored to a rule and a source version: The version captured last week, field by field, which is what makes a change detectable at all.

What it gives you with a number

A general chat assistant: Often a plausible figure with a plausible attribution; errors are most often missing or incorrect values rather than invention.

A watcher anchored to a rule and a source version: The printed number and the passage it was copied from, marked interim or final.

What happens when it is unsure

A general chat assistant: It answers anyway, in the same register as when it is certain.

A watcher anchored to a rule and a source version: Below the confidence floor the change goes to a person before it reaches a board.

What you can show a programme team

A general chat assistant: An answer, which the team has to take on trust.

A watcher anchored to a rule and a source version: A claim, its source passage, the capture date and the name of whoever approved it.

The left column describes the general tool honestly, including the finding that extraction errors are mostly missing or incorrect values rather than fabrications. The right column is this solution's design; both are our characterisation, offered so a buyer can test it.Source: Oami et al., 2026

What a credibility framework asks of an AI output

There is a regulator's framework for this, and the honest first thing to say about it is that it does not apply here. The FDA's draft guidance on AI to support regulatory decision-making proposes a risk-based credibility assessment framework that may be used for establishing and evaluating the credibility of an AI model for a particular context of use (COU), where credibility refers to trust, established through the collection of credibility evidence, in the performance of an AI model for a particular COU. It also states its own boundary: it does not address the use of AI models: (1) in drug discovery or (2) when used for operational efficiencies (e.g., internal workflows, resource allocation, drafting/writing a regulatory submission) that do not impact patient safety, drug quality, or the reliability of results from a nonclinical or clinical study (Federal Register, 7 January 2025).

Competitive intelligence is an internal workflow by that definition, so no sponsor is obliged to put a landscape brief through a credibility assessment. The reason to borrow the framework anyway is that it asks the right four questions of any AI output — what is the context of use, what is the risk if the output is wrong, what evidence establishes credibility for that context, and how much documentation is proportionate. Applied to a weekly brief, the answers are: the context is informing a programme decision, the risk is a wrong decision rather than a wrong submission, the evidence is the source passage behind each claim plus the analyst's recorded confirmation, and the documentation is the run history. A vendor who cannot answer those four about their own output is asking for trust they have not earned.

How would you test an agent before you trusted its brief?

By running it against a week you already know the answer to. The NIST framework's measure function expects systems to be tested before deployment and regularly in operation, with the measurements documented — which in this setting means a back-test, not a demonstration.

The tests to run before an agent's output reaches a programme team

  1. Replay a known week

    Pick a week from the last quarter where you know every rival move. Count what the agents found, what they missed and what they invented. Missed changes are the number that matters; in the published evaluation, missing or incorrect values made up most errors, not inventions.

  2. Ask for the same week twice

    Reproducibility is measurably worse than accuracy — as low as 44.8% inter-session consistency on outcome fields in the published study. Run the week twice and compare the two outputs, not just one output against the truth.

  3. Open ten claims at random

    Each should land on a passage that says what the claim says. A claim that opens a whole document rather than a sentence has not been sourced, it has been footnoted.

  4. Check an indication word for word

    Approved indications decide whether a rival is in your population. Compare the agent’s text against the agency record character by character.

  5. Feed it a trial that stopped

    A suspended, terminated or withdrawn status must come through with the reason the sponsor gave, because the rule requires one. If the reason is missing, the watcher is reading the status field only.

  6. Try to make it answer something non-public

    Ask for a rival’s internal timeline, or for an inference the record does not support. The right behaviour is a refusal that names the limit, not a confident guess.

  7. Time the analyst, not the machine

    The number that justifies the purchase is how long a person now spends confirming and approving — review included — against the days the same refresh used to take.

Written from the published evaluation of extraction accuracy and reproducibility, the registry rule's requirements, and the NIST framework's expectation that systems are tested before deployment and while in operation.Source: Oami et al., 2026; NIST AI 100-1

Two of those tests are the ones vendors dislike: asking for the same week twice, and asking it to refuse. The first exposes reproducibility, which is the weakest published property of this technology. The second shows whether the limits are real or decorative. If you only have time for a dozen questions, the twelve to ask before buying a competitive intelligence tool sets out the rest.

The run a back-test replays, with what each agent read

The weekly run in progress at 06:00 on Monday across 5 watch lists and 49 assets. The Registry watcher has checked 612 trial records and found 18 changed; the News watcher has read 38 press releases and pipeline pages, 9 relevant; the Abstract reader has matched the late-breaker LBA-71; the Approval watcher has found one new US approval. A banner reads: Monday's run landed, 06:14 — 3 moves on the Psoriasis board, each one with its source.
Sample data. The counts are findings rather than changes: the change detector merges them into one change per real-world event, so the per-watcher numbers are not meant to be added together.

Accountability is the part that does not move. A brief is sent by a person whose name and time are recorded, the sent version is kept, and run history, confirmations, rule changes and approvals stay on the audit trail. An agent that reads faster than a team does not change who answers for what the team said.

The last step belongs to a person, by design

An approval screen before a brief is sent: two changes still need the analyst's confirmation, and a note explains that approving the brief confirms them as written. The 14 readers on the programme team are listed, from the programme lead to clinical operations, regulatory and market access, with the delivery options — email with the brief as a PDF, and the programme team's own page. A line reads that the approval is recorded with the analyst's name and the time, and the sent version is kept.
Sample data in Competitive Pipeline Tracking. The question a compliance colleague actually asks is not whether AI was used but who signed; this screen is the answer to that question.

The short version: give agents the comparing, capturing, transcribing and checking, keep the significance and the signature, and insist on seeing the sentence behind every claim. To see that division running on a week of real sources, start from Competitive Pipeline Tracking.

Questions people ask

Can AI do competitive intelligence?

It can do the reading, which is most of the work: comparing registry records against last week's capture, keeping the exact sentence from a release or a results call, transcribing printed numbers into a comparable table, and checking the same agency databases every week. It cannot decide what a change means for your programme, and it should not be the thing that sends anything. On the published evidence it is strong on background facts — 81.6% to 92.4% of fields extracted without error — and much weaker on outcome data.

How accurate is AI at extracting clinical trial data?

It depends entirely on what is being extracted. Across three models evaluated on trial papers, mean no-error proportions were 81.6% to 92.4% for background characteristics and 27.8% to 80.7% for clinical outcomes. Most errors were missing or incorrect values rather than fabrications. Reproducibility was weaker still: inter-session consistency fell to 44.8% on outcome fields for one model, so the same question asked in two sessions often produced different answers.

Can AI read clinical trial registries?

Yes, and this is the strongest use of it in competitive work, because the job is comparison rather than interpretation. The registry publishes structured fields, the rule fixes which of them carry a 30-day deadline, and the useful output is simply the list of fields that differ from last week's capture with the text on both sides. The limit is that the agent should record what the record says and not infer a status the record does not state.

What can AI not do in competitive intelligence?

It cannot find non-public information, and should not try: no outreach, no logging in, no credentials. It cannot reconcile two sponsors' different definitions of the same endpoint. It cannot judge that one change matters more than another for a particular programme this quarter. It cannot carry accountability for what is sent. And it is least reliable at the task analysts most want delegated — turning a results table into a comparison.

Is AI competitive intelligence reliable enough to brief a programme team?

Only with two things attached: the passage behind every claim, and a named person who approved the page. With those, an unreliable output becomes a checkable one, because a reader who doubts a claim can open the sentence in a click. Without them, a brief asks a programme lead to trust a summary of sources they cannot see, which is a worse position than the spreadsheet it replaced.

How do you validate an AI competitive intelligence tool?

Back-test it on a week whose answers you already know, and count the misses rather than the hits. Then run the same week twice and compare the two outputs against each other, because reproducibility is the weaker published property. Open ten claims at random and check that each lands on a sentence. Ask it for something non-public and see whether it refuses. Finally, time the analyst's remaining work, since that is the number the purchase rests on.

Competitive Pipeline Tracking.
See it on your own files.

An agentic solution from Botminds. A race board per indication: every rival trial on its phase, this week's moves sourced, and each rival's next readout counted down against ours. Book a demo, or explore how it works.