AI competitive intelligence in pharma means handing the reading to software and keeping the judgement: agents compare a rival's trial record against the version captured last week, keep the exact sentence from a results call, pull the numbers out of a congress abstract and check three agencies' approval databases, while a named person decides what is significant and whether anything is sent. That division is not a philosophical preference, it is what the measurements support — language models extract background facts from trials reliably and outcome data much less so. Botminds' Competitive Pipeline Tracking is built on it: six agents each anchored to a published rule, every claim carrying the passage it came from, a confidence floor below which a change goes to review, and nothing reaching a programme team until an analyst approves it.
This page is written for the competitive intelligence lead being asked to put AI on the landscape, and for the legal or compliance colleague who will ask what it is allowed to decide. The accuracy figures below are from a published evaluation of language models, not from our product; the product figures are design settings and sample data, and are labelled as such.
What language models get right, and where they stop
81.6–92.4%
of background fields extracted from trial papers with no error, across three models
27.8–80.7%
of outcome fields extracted with no error, across the same three models
85%
confidence below which a proposed change goes to an analyst for review — a setting in the solution, not a measurement
What does AI competitive intelligence in pharma actually do?
It does four jobs that are all forms of reading. It compares: today's registry record against last week's capture, field by field. It captures: the sentence in which a sponsor said something about a watched asset. It transcribes: the numbers in an abstract, into a table where they can be lined up against other numbers. And it checks: the same handful of agency databases, every week, for the same list of assets. None of those four is a decision, which is why all four can be automated without moving the accountability for a landscape.
What it does not do is tell you anything that is not public. No agent in this design contacts a person, logs into anything, or reads a source a visitor could not read. That is the same boundary the profession's own code draws, and it is worth stating plainly to a compliance colleague: the answer to "where does the data come from" is "pages anybody can open", and the code of ethics and the legal line explains what sits on the other side of it.
The six agents of Competitive Pipeline Tracking, and the two seats a person keeps
Agent
Registry watcherCompares each watched trial record with the last captured version — status, phase, study start, enrolment, dates, arms, sites — with the exact text before and after.
Never: Never infers a status the record does not state
Agent
News watcherReads sponsor releases, pipeline pages and earnings-call transcripts and captures any statement about a watched asset, keeping the exact sentence.
Never: Public sources only; quotes rather than paraphrases
Agent
Abstract readerPulls design, population, endpoints and grade 3 or higher adverse events from congress abstracts into a comparable table.
Never: Numbers are copied, never computed; interim and final are flagged
Agent
Approval watcherChecks agency approval databases in the United States, the European Union and Japan for approvals, filing acceptances and label changes.
Never: Copies the indication word for word
Agent
Change detectorMerges findings into one change per real-world event, scores significance against the watch list rules, and explains in two sentences why it matters to our programme.
Never: Below 85% confidence it goes to review
A person decides
Analyst confirmsReads the old and new text side by side and decides which changes are significant for this programme.
Never: Significance is never settled by an agent
Agent
Brief writerDrafts one page per programme team with every claim linked to the passage it came from.
Never: No forecast stated as fact; never sends
A person decides
Analyst approves and sendsApproves the page as written, which also confirms the changes it carries, and sends it to the named readers, with the name and time recorded.
Never: Nothing is sent before approval
Six reading jobs, and why none of them is a decision
The test for whether a job can be given to an agent is whether a second competent person, given the same sources, would produce the same output. Comparing two versions of a record passes that test: the fields that differ are the fields that differ. Copying a hazard ratio out of a table passes it. Deciding whether a rival's Phase 3 start changes your own trial design does not pass it, because two competent people will reasonably disagree, and the disagreement is the value of having them.
That is why significance stays with a person even though the agent proposes it. A change detector can apply written rules — approvals, filings and label changes; pivotal readouts; phase advances; new trials in our indication; sites in our shortlist countries — and it can score how confident it is. What it cannot do is decide that this quarter, for this programme, a competitor's site opening matters more than its phase advance.
How accurate is an agent at pulling a number out of a paper?
Accurate on the easy half, unreliable on the hard half, and the gap is large enough to design around. Evaluating three models on trials behind five clinical questions in a national sepsis guideline, Oami and colleagues found that For background data extraction, mean no-error proportions ranged from 81.6% ... to 92.4% ... across models. For outcome data extraction, mean no-error proportions ranged from 27.8% ... to 80.7%
, and — importantly for anyone worried about invention — that Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon
(Frontiers in Digital Health, 2026).
Two findings in the same study deserve more attention than they usually get. Reproducibility was worse than accuracy: inter-session consistency ranged from 76.3% to 91.3% on background fields and from 44.8% to 65.6% on outcome fields, so at the low end the same model, asked again in a new session, agreed with its own earlier outcome extraction less than half the time. And prompt engineering strategies made only modest differences. The authors' conclusion is the one to quote at a steering group: LLMs can reliably support background data extraction in systematic reviews. However, outcome data extraction remains challenging, emphasizing the continued need for human oversight.
No-error extraction, by what is being extracted
The design consequence is specific. Where an agent must handle a number, it should copy a printed one rather than derive it, and the number should be shown next to the passage it was copied from so a person can check it in two seconds. The abstract reader in this solution is built that way — numbers are copied, never computed — and that rule is a direct response to the measurement above rather than a style preference.
The published rule each watcher is anchored to
An agent that reads a source without a rule behind it is guessing in a consistent voice. Each watcher here is anchored to something published, so that both its job and its limits can be argued with. The registry watcher's anchor is 42 CFR § 11.64, which fixes which registry fields carry a 30-day deadline and which drift under a twelve-month catch-all — the clocks that decide what a weekly comparison should look at, set out in the ClinicalTrials.gov update requirements (42 CFR § 11.64). The news watcher's anchor is the ethics code's requirement of public sources and no misrepresentation. The approval watcher's is the agency record itself, copied rather than summarised.
What each agent may do, what it may never do, and what it is anchored to
AgentRegistry watcher
MayCompare every watched trial record against the last captured version and record each changed field with the text before and after.
May neverInfer a status the record does not state, or read any non-public source.
Anchored to42 CFR § 11.64, which fixes what must be updated and when
AgentNews watcher
MayCapture any statement about a watched asset from a sponsor release, pipeline page or earnings-call transcript, keeping the exact sentence.
May neverParaphrase a claim, or contact anybody.
Anchored toThe SCIP code: public sources, no misrepresentation
AgentAbstract reader
MayPull design, population, endpoints and adverse events from a congress abstract into a comparable table.
May neverCompute a number the abstract does not print, or leave interim and final data unlabelled.
Anchored toThe abstract as published, nothing derived from it
AgentApproval watcher
MayCheck agency approval databases in the United States, the European Union and Japan for approvals, filing acceptances and label changes.
May neverReword an approved indication.
Anchored toThe agency record, copied word for word
AgentChange detector
MayMerge findings into one change per real-world event, score significance against the watch list rules and explain in two sentences why it matters.
May neverSettle significance alone, or pass anything below 85% confidence without review.
Anchored toThe watch list’s own written significance rules
AgentBrief writer
MayDraft one page per programme team with every claim linked to the passage it came from.
May neverState a forecast as a fact, or send anything.
Anchored toEvery claim needs a source passage; an analyst approves first
What no agent here may do
The list is short and worth reading as a specification rather than a reassurance. No agent contacts a person, by any channel. No agent logs in anywhere, which also rules out anything behind a credential — the ethics code treats obtaining a password to a competitor's site without authorisation as illegal, and so as a breach of the code, whatever the reason (SCIP, Ethics FAQs). No agent infers a trial status the record does not state. No agent computes a number that was not printed. No agent decides that a change is significant. And no agent sends anything to anybody.
Those constraints also mark where this kind of software does not help. It will not tell you about a rival's unregistered phase 1 programme, because no public source carries it. It will not reconcile two sponsors' different definitions of the same endpoint — a person has to. It will not tell you what a competitor intends. And it is weakest exactly where an analyst most wants help: turning a results table into a comparison. Buying it to replace judgement is buying the wrong thing; buying it to remove days of reading is buying what it does.
Confidence, and the line where a person confirms
A confidence score is only useful if something happens at a threshold. Here, the change detector scores each proposed change and anything below 85% confidence goes to an analyst for review rather than onto the board; above it, the change is still proposed rather than settled, and a named analyst confirms the ones marked significant. In the sample week that produced 28 changes, 6 of them significant, and two changes still awaiting confirmation at the moment the analyst came to approve the brief — which the solution tells her before she approves, because approving the page confirms them as written.
A change the detector was 99% confident about, still waiting for a person

Why every claim has to open its own passage
Because that is what makes an AI output reviewable at all. The NIST AI Risk Management Framework puts the general case bluntly — Deployment of AI systems which are inaccurate, unreliable, or poorly generalized to data and settings beyond their training creates and increases negative AI risks and reduces trustworthiness
— and names what the remedy depends on: Transparency reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system
(NIST AI 100-1).
In a weekly brief, transparency has a very concrete form: the claim links to the sentence, the date links to the record version, and the brief shows its own checks before anybody approves it. The sample brief carries 14 claims and 14 linked passages. That is not a quality metric so much as a precondition — a programme lead who cannot see the sentence has no way to disagree with the claim, and a brief nobody can disagree with is not intelligence.
A general assistant asked about a rival, and an agent anchored to a rule
A general chat assistant: Whatever its training and any live search returns, with no fixed list and no guarantee of coverage.
A watcher anchored to a rule and a source version: A named list of sources for a named list of assets, every week, with each page version kept.
A general chat assistant: Nothing. Each answer is generated fresh, so a change is invisible unless somebody already knew about it.
A watcher anchored to a rule and a source version: The version captured last week, field by field, which is what makes a change detectable at all.
A general chat assistant: Often a plausible figure with a plausible attribution; errors are most often missing or incorrect values rather than invention.
A watcher anchored to a rule and a source version: The printed number and the passage it was copied from, marked interim or final.
A general chat assistant: It answers anyway, in the same register as when it is certain.
A watcher anchored to a rule and a source version: Below the confidence floor the change goes to a person before it reaches a board.
A general chat assistant: An answer, which the team has to take on trust.
A watcher anchored to a rule and a source version: A claim, its source passage, the capture date and the name of whoever approved it.
What a credibility framework asks of an AI output
There is a regulator's framework for this, and the honest first thing to say about it is that it does not apply here. The FDA's draft guidance on AI to support regulatory decision-making proposes a risk-based credibility assessment framework that may be used for establishing and evaluating the credibility of an AI model for a particular context of use (COU)
, where credibility refers to trust, established through the collection of credibility evidence, in the performance of an AI model for a particular COU.
It also states its own boundary: it does not address the use of AI models: (1) in drug discovery or (2) when used for operational efficiencies (e.g., internal workflows, resource allocation, drafting/writing a regulatory submission) that do not impact patient safety, drug quality, or the reliability of results from a nonclinical or clinical study
(Federal Register, 7 January 2025).
Competitive intelligence is an internal workflow by that definition, so no sponsor is obliged to put a landscape brief through a credibility assessment. The reason to borrow the framework anyway is that it asks the right four questions of any AI output — what is the context of use, what is the risk if the output is wrong, what evidence establishes credibility for that context, and how much documentation is proportionate. Applied to a weekly brief, the answers are: the context is informing a programme decision, the risk is a wrong decision rather than a wrong submission, the evidence is the source passage behind each claim plus the analyst's recorded confirmation, and the documentation is the run history. A vendor who cannot answer those four about their own output is asking for trust they have not earned.
How would you test an agent before you trusted its brief?
By running it against a week you already know the answer to. The NIST framework's measure function expects systems to be tested before deployment and regularly in operation, with the measurements documented — which in this setting means a back-test, not a demonstration.
The tests to run before an agent's output reaches a programme team
- Replay a known week
Pick a week from the last quarter where you know every rival move. Count what the agents found, what they missed and what they invented. Missed changes are the number that matters; in the published evaluation, missing or incorrect values made up most errors, not inventions.
- Ask for the same week twice
Reproducibility is measurably worse than accuracy — as low as 44.8% inter-session consistency on outcome fields in the published study. Run the week twice and compare the two outputs, not just one output against the truth.
- Open ten claims at random
Each should land on a passage that says what the claim says. A claim that opens a whole document rather than a sentence has not been sourced, it has been footnoted.
- Check an indication word for word
Approved indications decide whether a rival is in your population. Compare the agent’s text against the agency record character by character.
- Feed it a trial that stopped
A suspended, terminated or withdrawn status must come through with the reason the sponsor gave, because the rule requires one. If the reason is missing, the watcher is reading the status field only.
- Try to make it answer something non-public
Ask for a rival’s internal timeline, or for an inference the record does not support. The right behaviour is a refusal that names the limit, not a confident guess.
- Time the analyst, not the machine
The number that justifies the purchase is how long a person now spends confirming and approving — review included — against the days the same refresh used to take.
Two of those tests are the ones vendors dislike: asking for the same week twice, and asking it to refuse. The first exposes reproducibility, which is the weakest published property of this technology. The second shows whether the limits are real or decorative. If you only have time for a dozen questions, the twelve to ask before buying a competitive intelligence tool sets out the rest.
The run a back-test replays, with what each agent read

Accountability is the part that does not move. A brief is sent by a person whose name and time are recorded, the sent version is kept, and run history, confirmations, rule changes and approvals stay on the audit trail. An agent that reads faster than a team does not change who answers for what the team said.
The last step belongs to a person, by design

The short version: give agents the comparing, capturing, transcribing and checking, keep the significance and the signature, and insist on seeing the sentence behind every claim. To see that division running on a week of real sources, start from Competitive Pipeline Tracking.
Questions people ask
Can AI do competitive intelligence?
It can do the reading, which is most of the work: comparing registry records against last week's capture, keeping the exact sentence from a release or a results call, transcribing printed numbers into a comparable table, and checking the same agency databases every week. It cannot decide what a change means for your programme, and it should not be the thing that sends anything. On the published evidence it is strong on background facts — 81.6% to 92.4% of fields extracted without error — and much weaker on outcome data.
How accurate is AI at extracting clinical trial data?
It depends entirely on what is being extracted. Across three models evaluated on trial papers, mean no-error proportions were 81.6% to 92.4% for background characteristics and 27.8% to 80.7% for clinical outcomes. Most errors were missing or incorrect values rather than fabrications. Reproducibility was weaker still: inter-session consistency fell to 44.8% on outcome fields for one model, so the same question asked in two sessions often produced different answers.
Can AI read clinical trial registries?
Yes, and this is the strongest use of it in competitive work, because the job is comparison rather than interpretation. The registry publishes structured fields, the rule fixes which of them carry a 30-day deadline, and the useful output is simply the list of fields that differ from last week's capture with the text on both sides. The limit is that the agent should record what the record says and not infer a status the record does not state.
What can AI not do in competitive intelligence?
It cannot find non-public information, and should not try: no outreach, no logging in, no credentials. It cannot reconcile two sponsors' different definitions of the same endpoint. It cannot judge that one change matters more than another for a particular programme this quarter. It cannot carry accountability for what is sent. And it is least reliable at the task analysts most want delegated — turning a results table into a comparison.
Is AI competitive intelligence reliable enough to brief a programme team?
Only with two things attached: the passage behind every claim, and a named person who approved the page. With those, an unreliable output becomes a checkable one, because a reader who doubts a claim can open the sentence in a click. Without them, a brief asks a programme lead to trust a summary of sources they cannot see, which is a worse position than the spreadsheet it replaced.
How do you validate an AI competitive intelligence tool?
Back-test it on a week whose answers you already know, and count the misses rather than the hits. Then run the same week twice and compare the two outputs against each other, because reproducibility is the weaker published property. Open ten claims at random and check that each lands on a sentence. Ask it for something non-public and see whether it refuses. Finally, time the analyst's remaining work, since that is the number the purchase rests on.