Regulatory intelligence software watches health authorities for new and changed guidance, regulations, notices and safety communications, and tells a regulatory affairs department what each one means for its own products. That is the promise across the whole market, and it hides an enormous range: at one end a keyword alert on a few agency pages, at the other a system that compares a revised guideline with the version it replaces, works out which of your licences and planned submissions it touches, and routes it to a named person who signs before anyone is told. Both are sold with the same three words. This page is eighteen questions that tell them apart, grouped into the four jobs that decide whether work actually moves — and an honest account of what Botminds' Regulatory Intelligence does and does not do against them.
The questions are written so that any supplier, including us, can answer them with a plain no. A no is useful. A question only one product can pass is a feature list pretending to be a standard, and you should be suspicious of any buyer's guide that reads that way — including this one, which is why every question below ends in evidence you can ask to see rather than a capability you can tick.
An alert feed and a system that assigns work: what each one leaves on your desk
An alert feed: A link and a title, often the page title rather than the document’s.
A system that assigns the work: The document, its original file, its publication date and the passage that matters.
An alert feed: Told that a revision exists. Finding the old version and comparing them is your afternoon.
A system that assigns the work: The replaced version named by the agency, compared paragraph by paragraph, with editorial changes separated from substantive ones.
An alert feed: A title in the original language, or a machine translation with no original beside it.
A system that assigns the work: Paragraph-level translation with the source text kept alongside, and uncertain passages flagged.
An alert feed: Keyword matches. You decide whether it touches your portfolio, every morning, for everything.
A system that assigns the work: Compared against your own product and licence list, labelling plan and submission calendar, with a proposed level and a reason.
An alert feed: Everyone on the distribution list, or one coordinator who forwards.
A system that assigns the work: The named owner of the affected product area, with the paragraphs that land on them.
An alert feed: An inbox, and whatever anybody copied into a spreadsheet.
A system that assigns the work: Who decided what, when, on which passage, and which teams were told.
What does regulatory intelligence software actually do?
Three jobs, and it is worth naming them separately because products are usually strong at one and weak at the others. Collection is finding what published, in every language your markets file in, without missing a page that changed quietly. Analysis is working out what it says, what changed against the version before it, and what that means for your products, your labelling and your filing dates. Dissemination is getting the right paragraphs to the person who can act, with enough context that they act rather than forward.
A subscription database is mostly collection. A consultancy retainer is mostly analysis. A distribution list is dissemination with nothing in front of it. The reason the three have to be bought together is that the value only appears at the end: a collection that nobody analyses is a backlog, and an analysis that reaches the wrong person is a document.
There is a fourth thing, which is not a job but a property: the record. In a regulated department, the question "how did we know, and who decided?" has to have an answer eighteen months later, and that answer is either designed in or it does not exist.
The four questions behind the eighteen
Everything below is a way of asking one of four things. Did you see it? — coverage. What changed? — comparison. What does it mean for us? — impact. Who signed? — the record. If a demonstration answers all four on one of your own documents, the rest is commercial negotiation. If it answers two, you are buying a feed and staffing the other two yourself, which may well be the right decision as long as you make it deliberately.
Asking in groups also makes the rubric easy. GAO's accountability framework for AI is organised the same way: it is built around four principles covering governance, data, performance and monitoring, and for each practice it supplies "a set of questions for entities, auditors, and third-party assessors to consider, as well as procedures for auditors and third-party assessors." A question list with evidence attached to each question is a far better instrument than a scoring sheet with adjectives.
Coverage: six questions about sources, and the evidence to ask for
1. What exactly is on the source list, and may I have it as a list? Not "global coverage" — the file. Count the sources that matter to you: the agencies you file with, the ones your partners file with, and the national competent authorities in the markets where you hold a licence. Evidence: the list, with a row per source.
2. How often is each source checked, and what happens when a site changes? Agencies redesign their sites, move PDFs and change feed formats without notice. Ask how a break is detected, how fast it is fixed, and whether you are told that a source was not successfully checked. A system that silently stops watching a page is worse than no system, because it buys confidence it has not earned.
3. What does it do with a document that is not in English? The answer you want has three parts: the original is kept and shown; translation happens at paragraph level so a passage can be quoted against its source; and the system says which passages it is unsure of. "We translate" with no original beside it means nobody downstream can check a quotation.
4. What does it miss, and how would I know? Every honest supplier can answer this. Portals behind logins, documents only announced by email to registered holders, agency social accounts, paper letters. Ask them to name their gaps, and ask how a gap is recorded so your own procedure can cover it another way.
5. Can we add our own sources? Most departments have a handful nobody else watches — a trade-association technical committee page, a partner's regulatory bulletin, a market where you are the only one filing. Ask whether adding one is a configuration change you make or a service request with a lead time.
6. Does the same document arrive twice, and how do you know it is the same? A document that appears on an agency's news page, its guidance list and a feed is one document. De-duplication sounds trivial until a morning's list has 41 items that are really 27, and people start skimming.
Comparison: four questions about revised documents
7. When a guideline is revised, how do you find the version it replaces? There is only one safe answer: from the document itself, because agencies name what they supersede on the cover. A system that guesses by title similarity will eventually compare a 2026 revision against the wrong 2019 document, and the difference will look like a substantive change that nobody made.
8. Do you show me the old and the new wording, or a description of the difference? This is the single question that most reliably separates products. A description of a change to a legal text is a paraphrase, and a paraphrase is the thing you are not allowed to rely on when you write the justification in a submission. Ask to see the two texts side by side, at paragraph level.
9. Do you separate editorial changes from substantive ones, and can I see your working? In a long revised guideline most changed paragraphs are formatting, numbering and house style. The value is in the three that are not. Ask how the separation is made and whether you can open the editorial ones anyway — because sometimes an "editorial" renumbering changes a cross-reference you rely on.
10. What happens when the agency does not publish a clean previous version? It happens, particularly with national authorities and with documents that have been reissued rather than revised. Ask whether the system says so plainly, or quietly compares against something close enough.
The comparison, as a buyer should ask to see it: old wording and new, word for word

Impact: four questions about your products, markets and plans
11. What of ours does it compare each update against? Relevance is not a property of the document; it is a relation between the document and your portfolio. The three inputs that make the relation real are a product and licence list with markets, a labelling plan, and a submission calendar. Ask which of the three the system can take, in what format, and how often it is refreshed.
12. Does it propose a level per product and market, or one label for the whole update? A guideline revision is commonly High for one presentation and not relevant for another. One label for the document pushes the real work back onto the reader.
13. Can I see the reason and the passage behind every proposed level? Insist on this one. A general-purpose language model reading regulatory text can produce a fluent, confident statement that nothing in the document supports, and that is the failure a reader is least likely to catch. A citation to the passage is the only check that costs one click, and NIST's guidance on AI risk points the same way: accuracy claims mean little without the test set and method behind them. A level with no passage beside it is an opinion you cannot audit.
14. Can we set rules that the system is not allowed to argue with? Every department has a few. Any labelling change on a marketed product is at least High; anything touching a product under a post-authorisation commitment goes to a named person regardless of level. Ask whether those are settings you control, when a change takes effect, and whether a rule can only raise a level or also lower one.
Signature and record: four questions about governance
15. What exactly does an electronic signature store? There is a right answer, and it is in the regulation. A signed electronic record must show the signer's printed name, the date and time of signing, and the meaning of the signature — review, approval, responsibility or authorship. Those three items carry the same controls as the record itself and must appear on any human-readable form of it — so they have to appear on the printout, not only in a database.
16. Who is allowed to confirm what, and can that differ by product area? Biologics, devices, small molecules and an affiliate market usually have different approvers, and the person who may confirm a Japanese labelling change is not the person who may confirm a European device change. Ask whether approvers are per area and how a delegation during leave is recorded.
17. Is anything sent before a human confirms it? A yes is not disqualifying — low-importance updates filed automatically is a reasonable design, and sampling them is a reasonable control — but you need to know exactly where the line is and who drew it. Ask what proportion is filed without review, and what the sampling rate is.
18. How would you validate this, and what would you give us to do it with? The regulation is a good prompt: it asks for systems validated for accuracy, reliability and consistent intended performance, and able to tell an invalid or altered record from a good one. In practice ask for: a written description of what the system does and does not do, a test set you keep and the supplier does not see, a measurement with its methodology, and a monitoring plan after go-live. NIST makes the test set non-negotiable: an accuracy figure should always come with a realistic test set that represents real use, and with its method.
One more thing worth knowing before you ask question 18: what the European regulator actually comments on when AI is used in regulatory evidence. A systematic analysis of EMA advice and public reports, published in Clinical Pharmacology & Therapeutics, found that the agency's comments "varied according to the context of use and covered aspects of documentation, methodology, validation, and lifecycle management." Those four words are a decent shape for the evidence pack you ask a supplier for.
The evidence pack to ask for before the contract, not after
- The source list as a file
One row per source: the agency, the pages watched, the language, the schedule, and who fixes it when it breaks.
- A written description of what the system does, and does not
The specific job, inside which process, with how much influence on which decision. If a supplier cannot write this paragraph, nobody can validate it.
- One real comparison on one of your own revised guidelines
A document you have already analysed by hand, so you can mark their output rather than admire it.
- An accuracy measurement with its test set and method
What was measured, on which documents, scored by whom, and the errors broken down by type rather than one percentage.
- A signature record, printed
Name, date and time, meaning — on the human-readable form, as the regulation requires, not only in a database field.
- The settings catalogue
Which of the decisions you make are configuration you own, which are a service request, and when a change takes effect.
- The monitoring plan after go-live
What is watched, how often, what number triggers a review, and who sees it.
- Named gaps
What the supplier knows it does not cover. A list of gaps is the strongest signal of an honest answer to everything else.
How do you choose regulatory intelligence software?
Score the four groups separately and refuse to average them. A product that scores well on coverage and badly on impact is a feed, and should be priced like one. The rubric below is the one we would use; the point is less the exact wording than that a 2 requires evidence, not a demonstration.
A scoring rubric where a 2 has to be evidenced
| Group | 0 — not really | 1 — partly | 2 — evidenced |
|---|---|---|---|
| Coverage · did you see it | Keyword alerts on a handful of pages; no list; nothing about failures. | A source list and a schedule, but no signal when a source is not successfully checked. | The list, per-source schedules, failure alerts, named gaps, and your own sources addable by you. |
| Comparison · what changed | Tells you a revision exists. | Shows a generated description of the difference. | Old and new wording paragraph by paragraph, against the version the agency names as replaced, editorial changes separated. |
| Impact · what it means for us | One relevance label for the whole document. | A level per product, with no reason you can audit. | A level per product and market, with the reason, the passage and a confidence, compared against your own licence list, labelling plan and calendar. |
| Record · who signed | An activity log. | An approval step with a user ID and a timestamp. | Name, date and time and meaning on the record and on its printout; approvers per product area; nothing sent before confirmation. |
The demo test: one revised guideline, from publication to a signed note
Ninety minutes, one document of yours, four questions. Pick a revised guideline from an agency you file with that your team has already analysed by hand, so you know the right answer. Bring the three inputs. Then ask the supplier to drive while somebody of yours takes the time at each step.
A ninety-minute demonstration, and the test to pass at each step
- Show the source list, then find my document in what it collected15 min
- Name the replaced version from the document10 min
- Old and new wording, paragraph by paragraph, against my hand analysis15 min
- Load my product and licence list, labelling plan and calendar15 min
- A level per product with its reason and passage — mark it against mine15 min
- Change an approver and a rule live, in front of me10 min
- Confirm one impact, then show me the signed record printed10 min
Two rules make the exercise honest. Use a document nobody prepared for, and keep a test set the supplier never sees. NIST's framing of third-party risk explains why the second matters: third-party systems can accelerate things and also "complicate risk measurement", because the measures the builder uses need not be the measures you would use. The only measure you can fully trust is the one you built.
The settings a buyer should ask to see changed live

Red flags, and what a good answer sounds like
None of the sentences on the left are lies. They are answers that stop a conversation, and each has a version that opens one.
Six answers that should make you ask again
"We cover every agency."
Nobody does. The answer you want is a list, a schedule per source, and what happens when a site changes shape.
"Here is the list. These 14 are checked hourly, these 50 daily, and these 6 we get from a feed we do not control."
"Our accuracy is 95%."
A percentage with no test set behind it is a marketing number. Ask what was measured, on which documents, against whose answers.
"On 200 documents from three agencies, scored by two reviewers, we were right on 180. Here is the error breakdown by type."
"The AI decides the impact."
Then your signature is on a decision you did not make, and the accountability has not moved anywhere.
"It proposes a level with a reason and the passage. Your named approver confirms it, and nothing is sent before they do."
"It summarises the change for you."
A summary of a revision is a paraphrase of a legal text. The question is whether you can see the old and new wording.
"We show the replaced version paragraph by paragraph against the new one, and we never guess which version is replaced."
"Everything is in the audit trail."
Ask to see one. Specifically, ask what an electronic signature stores.
"Name, date and time, and the meaning of the signature, on the same record, exportable in a human-readable form."
"We will tune it during the pilot."
Fine — but then the pilot has no baseline, and you cannot tell improvement from fitting to your test.
"We will set it up, then run a fixed test set you keep, and show you the same measure before and after."
There is a seventh, quieter one: a demonstration on the supplier's data. It is always smoother, and it tells you nothing, because the hard part of this work is the collision between a published document and your own particular portfolio. Insist on your documents, even if it means a second meeting.
What this software cannot do for you
It cannot decide. The judgement about whether a changed expectation moves a filing, what to say to an assessor, and how much risk a wording carries is regulatory work, and it carries a signature that has to be a person's. NIST says the same thing about thresholds generally: people, not the system, should choose the trustworthiness metrics and set their thresholds. Somebody has to decide what counts as High for your portfolio, and no supplier can do it for you.
It cannot fix inputs you do not have. If your product and licence list is a spreadsheet nobody owns, or your labelling plan lives in three places, no amount of monitoring will produce a reliable impact assessment — the system will compare a real document against an imaginary portfolio. Sorting that out is usually the longest part of a programme and the part no supplier can sell you.
And it cannot tell you what will be published next. Horizon scanning is a discipline of agendas, consultations and effective dates, not prediction; a product that offers to tell you what a regulator is about to do is describing something nobody can deliver.
What the four jobs look like when they are joined up

For the wider decision — whether to buy a subscription, hire people, or put agents on it — see what a database, a team and agents each cover; for the governance questions behind group four, see who signs the impact. To put these eighteen questions to us on your own documents, start at Regulatory Intelligence.
Questions people ask
What is the difference between regulatory intelligence software and a RIM system?
A regulatory information management system is a system of record for your own regulatory data — registrations, submissions, commitments, health authority correspondence. Regulatory intelligence software looks outward: it watches what agencies publish and works out what it means for that portfolio. The two meet at the inputs, because the intelligence tool needs your product and licence list and your submission dates to say anything useful. Ask any supplier how it reads them, and whether it writes anything back.
What should a regulatory intelligence RFP include?
The source list as a file; per-source schedules and what happens when a source breaks; how a revision's replaced version is identified; whether old and new wording are shown; which of your own inputs are read and how; whether a level is proposed per product and market with a reason and passage; who may confirm and what the signature stores; which decisions are settings you control; an accuracy measurement with its test set and method; and the supplier's named gaps. Ask for a test on one of your own documents as part of the response, not after it.
How do you validate regulatory intelligence software?
Define what it does and does not do in one paragraph, build a test set from your own documents with answers you already know, measure error types separately rather than reporting one percentage, write down the human step and who is accountable for it, and keep monitoring after go-live with a number that triggers a review. Where it touches regulated records, the usual expectations apply — accuracy, reliability, consistent intended performance, and the ability to tell an invalid or altered record from a good one.
Is regulatory intelligence software worth it?
It depends entirely on which of the four jobs you are short of. If your problem is that nobody has time to read, a feed helps. If your problem is that things are read but reach the owning team a week late, a feed will not help at all and may make it worse by adding volume. Work out where your days actually go before you shortlist — count, on ten real updates, the hours from publication to the owning team and where they went.
How many sources should a regulatory intelligence system watch?
As many as your markets require and no more. The useful count is not the supplier's total but the overlap with your own list: the agencies you file with, the authorities in every market where you hold a licence, and the few specialist sources your therapeutic area depends on. A product watching hundreds of sources, none of which is the national authority in your largest market, covers nothing you need.
Can a small regulatory team use this kind of software?
Yes, and small teams often get more from it, because the three jobs are the same size whatever the headcount and a department of two has no slack. The constraint is not the tool but the inputs: a current product and licence list, a labelling plan and a submission calendar. A small team that has those can start; one that does not should spend the first weeks building them, because every impact assessment depends on them.