Competitive intelligence tools for pharma are bought on demonstrations and judged, a year later, on whether a programme team believed the weekly findings. The gap between those two moments is where most selections go wrong, because a demonstration shows coverage and a programme team needs traceability. The twelve questions below are built so that each one can be answered with evidence rather than a claim: the public registries publish their own update deadlines, so you can test what freshness is physically possible; the compliance literature is published, so you can test whether a curated database can be complete; and a public measurement framework sets out what testing an automated system should involve. Botminds' Competitive Pipeline Tracking is one answer to those questions, and this guide says where it fits and where it does not.
Where the solution appears below it is running on fictional sample data — the watch lists, people and asset codes are invented for the demonstration. The research and the regulations are real and linked at each figure.
Three public facts that set the bar any tool has to clear
30days
the longest a sponsor may take to publish a change in recruitment status, so a weekly read is the shortest useful interval
63.8%
of trials due to report results under the United States rule had submitted them at any point, in a cohort study to September 2019
12months
the outer limit on refreshing registration information, which is why a current record is not a complete one
What should competitive intelligence tools for pharma actually do?
Three things, in this order. Read the sources a human analyst would read, on a schedule, without being asked. Turn what changed into a small number of findings that matter to a named programme, with the evidence attached. And leave a person in charge of what gets said, so a programme lead can act on a finding without first checking it.
Everything else — the visualisations, the alerts, the similarity scores, the natural-language querying — is in service of those three or it is decoration. A useful way to hold the line in a selection is to ask, of every feature demonstrated, which of the three it serves. A dashboard that shows a competitor's pipeline as a set of coloured markers serves none of them if the markers cannot each open the sentence they came from.
The twelve questions below follow the same order: what it reads, what it does with what it reads, what evidence it keeps, and who is in control.
Questions 1–3: sources, coverage and freshness
1. Name every source you read, and tell me how each one is obtained. Not "public sources" or "the leading databases" — the list, with how each is accessed. A vendor that reads a registry's own interface has a different freshness and a different failure mode from one that licenses a third-party extract. Ask whether any source is a competitor's or a customer's non-public material; the answer should be no, and the reason is in the competitive intelligence code of ethics and the legal line.
2. What is outside your coverage, and how would I know? Every tool has edges. The useful ones are specific about them: which registries, which languages, which congresses, which agencies, which company sizes. The World Health Organization's registry network lists eighteen primary registries around the world, from the Australian and New Zealand register to the Chinese, Indian, Japanese and Pan-African ones, each meeting "specific criteria for content, quality and validity, accessibility, unique identification, technical capacity and administration" (WHO Registry Network). A tool that reads two of them is not wrong; a tool that does not say which two is.
3. How soon after a change on the source do I see it, and how do you know? This is the question most often answered with a marketing word — "real time", "continuous", "always current". It has a factual answer, and the next section is how to get it.
The freshness test you can run in an afternoon
Freshness is the one claim a buyer can verify without the vendor's cooperation, because the sources publish their own clocks. Under the United States registry rule, a change in a trial's overall recruitment status must be published within 30 calendar days of the change; the same applies to any individual site's status; and registration information as a whole must be refreshed at least once every twelve months (42 CFR § 11.64). What those deadlines mean for a tool is simple: nothing a tool reads from that registry can be fresher than the sponsor's own posting, and a weekly read captures essentially everything a daily read would, a few days later.
So the test is a comparison, not a stopwatch. Take six watched assets. For each, find in the registry the most recent change and the date the record shows it was posted. Then ask the tool when it reported that change. Three outcomes are informative: the tool found it within a week of posting, which is as good as the source allows; the tool found it but weeks later, which tells you the read interval; or the tool never reported it, which tells you the change was filtered out, and the reason for that is question 6.
Run the same test on a sponsor announcement and on an agency approval, because those three source families move at different speeds and a tool that is excellent at one is often indifferent at another. The whole exercise takes an afternoon and it replaces every adjective in the proposal with a number.
Questions 4–6: change detection, de-duplication and significance
4. Show me a change, before and after, on a record you did not choose. The demonstration will have a beautiful example ready. Ask for one from your own watch list, found this week. What you are looking for is the previous version of the source held alongside the current one, with the changed field marked — not a sentence saying something changed.
5. One event reaches you three times. What do I see? A phase advance typically appears in a registry update, a pipeline page edit and a press release within days of each other. A tool that reports three findings has given you three times the reading and no more information. Ask how the merge is done, how often it is wrong, and what happens when two sources disagree about the same event — because they frequently do, and the disagreement is itself a finding.
6. Who decides what is significant, and can I see the rule? This is the question that decides whether the tool is usable in week six. Significance is not a property of a change; it is a property of a change in relation to your programme. A rival's new trial in your indication matters; the same rival's new trial in a different disease does not. If the rule is a model's judgement that cannot be inspected or edited, you have bought someone else's priorities.
What counts as significant, written as rules you can read and change

Questions 7–9: sourcing, confidence and who decides
7. Open the passage behind this claim. One click, from any statement in any output, to the sentence in the source that supports it — and to the version of that source as it stood when it was read. This is the question that most separates a working pharmaceutical tool from a general news-monitoring product, because a programme team cannot act on an unsourced claim and a compliance reviewer cannot defend one.
8. What is your confidence, and what happens below it? Any automated extraction has a distribution of reliability, and the honest tools say so and route the tail to a human. Ask for the threshold, ask what proportion of items fall below it in a typical week, and ask what the person on the other side is shown. A tool with no low-confidence pile is not more accurate; it is less candid.
9. Who approves what reaches a programme team, and is that recorded? Named approver, recorded time, kept version. If the answer is "it sends automatically", the tool is likely to be switched off the first time it sends something wrong to a therapy-area head.
A dashboard of logos against a sourced report a named person approved
What a demonstration usually shows: A feed of everything that moved, ranked by an unexplained relevance score.
What a programme team can act on: A small number of findings chosen by rules you wrote, with the rest listed below.
What a demonstration usually shows: A link to the source's home page, or to a current record that may since have changed.
What a programme team can act on: The passage, highlighted, in the version of the source as it stood when it was read.
What a demonstration usually shows: The same phase advance three times, from three source families.
What a programme team can act on: One event, with its three sources attached to it.
What a demonstration usually shows: A summary of the change, written about the rival.
What a programme team can act on: Two sentences on what it means for your asset, with a suggested owner for the action.
What a demonstration usually shows: Nobody. The feed is the product.
What a programme team can act on: A named analyst approved it, at a recorded time, and the version that went out is kept.
What a demonstration usually shows: The rules live in the vendor's configuration and in one analyst's head.
What a programme team can act on: The rules are settings, versioned, with an audit trail behind every change.
Questions 10–12: your rules, your approvers, your record
10. Can I change a significance rule myself, on a Tuesday, without a support ticket? Watch lists change when a programme changes, and the lag between a reorganisation and a tool catching up is where a weekly report quietly stops being read. Ask to do it, live, during the evaluation.
11. Who can see what, and can I prove it later? Pharmaceutical competitive intelligence mixes public findings with your own development plans. The planned readout dates you compare a rival against are not public, and the tool has to keep that boundary by configuration rather than by convention.
12. What does your audit trail hold, and for how long? Runs, confirmations, edits, rule changes, approvals and the versions sent. Ask for a retention period that matches your own records policy, in writing. A tool that can reconstruct why a particular claim was in a particular report in a particular week is a tool that can survive a question from legal.
The twelve questions, and the evidence to ask for with each
- Name every source, and how each is obtained
Evidence: the list, in writing, in the contract schedule. Not a slide.
- What is outside coverage, and how would I know?
Evidence: the registries, languages, congresses and agencies not read, named.
- How soon after a change do I see it?
Evidence: run the freshness test on six of your own assets against the registry's posting dates.
- Show me a before and after on a record I chose
Evidence: the previous version held beside the current one, with the changed field marked.
- One event, three sources — what do I see?
Evidence: a merged finding with its three sources, and a worked case where two sources disagreed.
- Who decides significance, and can I see the rule?
Evidence: the rule list on screen, editable, with a version history behind it.
- Open the passage behind this claim
Evidence: one click from a statement to the highlighted sentence in the stored source version.
- What is your confidence, and what happens below it?
Evidence: the threshold, the weekly proportion below it, and the review screen a person sees.
- Who approves what goes to a programme team?
Evidence: a named approver, a recorded time and the kept copy of what was sent.
- Can I change a rule myself, today?
Evidence: do it during the evaluation, and see the new version appear on the audit trail.
- Who can see what, and can I prove it?
Evidence: a permissions model that separates public findings from your own development plan.
- What does the audit trail hold, and for how long?
Evidence: the retained item types and the retention period, in the contract.
Six answers that separate a working tool from a demonstration
| The question | The answer that means yes | The answer that means no |
|---|---|---|
| Where did this date come from? | "From this sentence, in this version of this page, read on this date." The sentence opens. | "From our database." The link goes to a home page or to a record that has since moved on. |
| Why is this flagged as significant? | "Because rule four — a new trial in your indication — matched, and you can edit rule four." | "Our model scored it highly." No rule, no edit, no explanation. |
| How many findings will we get a week? | A number, from your own watch list, in a pilot, with the proportion a person confirmed. | A number from another customer, or "as many as there are changes." |
| What happens when two sources disagree? | "Both are kept and the disagreement is shown; the analyst decides which the report states." | "We take the most recent," with no record of what the other said. |
| Who sent last week's report? | A person, by name, at a recorded time, with the version kept. | The system did, automatically, overnight. |
| Can we add a significance rule ourselves? | Demonstrated live, with the change appearing as a new version on the audit trail. | "Raise a request and we will configure it." |
| How accurate is the extraction? | A measured figure, by field type, with the method and the sample described, and a confidence threshold below which items go to review. | A single accuracy percentage with no field breakdown and no stated method. |
How to evaluate an AI competitive intelligence tool before you buy
Every tool in this category now describes itself as AI-based, which makes the label useless as a discriminator and makes testing the only option. The United States national framework for managing AI risk is plain about what that means: systems are to be tested before deployment and regularly once in use, and measurement is to rest on rigorous testing with stated uncertainty, comparison against benchmarks, and results that are formally reported and documented (NIST AI RMF 1.0). Three of its observations apply almost word for word to a vendor selection.
The vendor's measurements may not be yours. The framework points out that the organisation that builds an AI system and the organisation that deploys it may measure risk in different ways, and that the builder may not disclose how it measured. An accuracy figure in a proposal is a measurement of something; ask what, on which fields, against which truth.
A single number can be gamed. The same document warns that a measurement approach can be oversimplified or gamed, can miss the nuance that matters, and can end up relied on in ways nobody intended. In this domain, an extraction accuracy averaged across easy fields and hard ones hides exactly the distinction a buyer needs: copying a trial identifier is not the same task as reading an endpoint out of a congress abstract.
Test it yourself. The framework credits independent review with making testing more effective and with reducing internal bias and conflicts of interest. The practical version for a buyer is a pilot on your own watch list, scored by your own analysts against sources they check by hand.
Two tests are worth building into the pilot specifically. Give the tool a week in which you already know, by hand, everything that happened on one watch list, and compare: what it missed, what it invented and what it ranked wrongly. Then take ten claims from its output and trace each to the source yourself. A tool that passes both has earned the benefit of the doubt on the rest. The wider question of what an automated agent may and may not be allowed to do in this work is covered in the guide to AI competitive intelligence in pharma.
What should a six-week pilot on one watch list prove?
A pilot that proves the tool works is worthless; every tool works. A pilot should answer the questions a contract cannot: how many findings a week your team can actually absorb, whether the significance rules can be written down, and whether the programme teams read the output. Six weeks is enough because a weekly cycle gives six observations and three chances to change the rules.
A six-week pilot on one watch list, and what each week has to show
- One watch list, your assets, your sources, your approver namedweek 1
- First run against a week you already know by hand: what was missed, what was inventedweek 2
- Rewrite the significance rules yourselves and watch the volume changeweek 3
- Trace ten claims to their sources by hand; count how many open the right passageweek 4
- Send the report to one real programme team and ask whether they acted on anythingweek 5
- Measure the analyst hours the cycle took, and compare with the same week done by handweek 6
At the end of six weeks you should be able to state four numbers: findings per week on one watch list, the share a person had to correct, the share of claims that opened the right passage, and the hours the cycle took against the hours the same week takes by hand. Those four decide the business case, and none of them can be obtained from a demonstration.
The four numbers a pilot has to produce, on one screen

What to leave out of the requirements document
Requirements documents in this category are usually too long and insufficiently specific, which is the worst combination because it makes every vendor answer yes. Four things are safe to delete.
Source counts. "Monitors 50,000+ sources" is a number about the vendor's crawler, not about your watch list. What matters is whether the four or five source families your analysts read are read properly, and the European portal's own description of what it publishes — the trial number, the therapeutic area, the sponsor's details, and "Start and end dates of participant recruitment and of the trial itself" (EMA, Clinical Trials Information System) — is a better specification than any count.
Alerting latency in minutes. The sources do not move in minutes. A sponsor has thirty days to post a recruitment-status change, and for most events in this field the difference between a five-minute alert and a weekly read is negligible; the exceptions, such as an approval, are what same-day alerts are for.
Named visualisations. Asking for a specific chart buys you that chart. Ask instead for the question it is supposed to answer — who reads out before us, which of our sites a rival has opened — and let the vendor show how it answers it.
Accuracy as a single percentage. For the reasons the measurement framework gives, require it broken down by field type, with the method and the sample stated, and with a stated confidence threshold below which an item goes to a person.
What belongs in the document instead is short: the source list, the evidence requirement, the significance rules as settings, the named approver, the permissions boundary around your own plans, and the retention period.
Scoring the twelve answers: four blocks, and the three that are pass or fail
Sources, coverage and freshness (questions 1–3)
The foundation. A tool that reads the wrong things cannot be fixed by a better interface.
Question 1 is pass or fail: if the vendor will not name its sources, nothing below is testable.
Change detection, de-duplication and significance (questions 4–6)
The engineering. This is where a weekly feed either becomes six findings or stays six hundred.
Question 6 is pass or fail: significance rules you cannot see are significance rules you cannot defend.
Sourcing, confidence and who decides (questions 7–9)
The part a compliance reviewer will read. Every claim traceable, every low-confidence item visible.
Question 7 is pass or fail: a claim without the passage behind it cannot go into a report.
Your rules, your approvers, your record (questions 10–12)
The part that decides whether the tool survives a reorganisation and an audit.
None individually fatal, but two weak answers here usually mean the tool is run by the vendor, not by you.
Where a weekly cycle leaves the tool and becomes a person's job
The last thing to establish in a selection is where the automation stops, because that boundary is what the function is actually buying. In the sample solution it stops in three places: an analyst confirms what is significant, anything below a confidence threshold goes to review, and nothing is sent until a named person approves it, with the name, the time and the sent version kept.
The moment the tool stops and a person decides

If the make-or-buy question behind the selection is still open, what a drug pipeline database can and cannot give you sets out what a curated subscription adds over reading the public sources yourself. To run the twelve questions against a working system on one of your own watch lists, start from Competitive Pipeline Tracking.
Questions people ask
How do you choose competitive intelligence software?
Score it on what it reads, what it does with what it reads, what evidence it keeps and who stays in control, and treat three answers as pass or fail: a named source list, significance rules you can see and edit, and one click from any claim to the passage behind it. Then run a pilot on one of your own watch lists, because the four numbers that decide the business case cannot come from a demonstration.
What is the difference between a competitive intelligence tool and a pipeline database?
A database is a curated store of what has been published, organised so that you can search it. A tool watches your assets, tells you what changed since last week and says what it means for your programme. The two are complementary and are often confused in requirements documents, which is why a selection that specifies coverage alone tends to end with a subscription nobody reads.
How long does it take to implement competitive intelligence software?
The implementation is rarely the constraint; agreeing the watch lists, the significance rules and the approvers is. Expect the configuration of one watch list to take days and the agreement about what counts as significant to take weeks, because it is a conversation between the intelligence function and each programme team rather than a technical task.
Should a competitive intelligence tool send findings automatically?
Not to programme teams. Same-day alerts for a narrow class of events — an approval, a pivotal readout — are worth having, and even those are better sent to the intelligence function than to a therapy-area head. For the weekly cycle, a named person approving before it goes out is what makes the output trustworthy enough to be acted on without checking.
What should we measure in a pilot?
Four things: findings per week on one watch list, the share an analyst had to correct, the share of claims that opened the correct source passage, and the hours the cycle took compared with the same week done by hand. Measure them on a week your team has already worked by hand, so you have a truth to compare against rather than the tool's own account of itself.
How accurate should automated extraction be?
Ask for the figure by field type rather than as one number. Copying an identifier, a phase or a date is a different task from reading an endpoint with its confidence interval out of a congress abstract, and a single average hides the difference. What matters as much as the figure is whether there is a confidence threshold below which an item is sent to a person, and what proportion of a typical week falls below it.