AI in drug manufacturing is already governed, in draft, by texts most buyers have not read: a new EU GMP Annex 22 on artificial intelligence, a revised Annex 11 and Chapter 4 alongside it, the EMA's reflection paper on AI across the medicinal product lifecycle, and FDA's draft guidance on AI used to support regulatory decision-making. The difficulty is that they do not answer the question people ask — "is AI allowed?" — but a sharper one: what decision does the model make on its own, and what does a qualified person still decide? Botminds' Batch Record Search is built for the side of that line where the model finds and cites, and a named reviewer reads the source and signs; this guide explains where the line is drawn and how to test any vendor against it.
Two cautions before the clauses. Annex 22, the revised Annex 11 and the revised Chapter 4 are consultation texts, not law: the published EudraLex Volume 4 still lists Chapter 4 as its January 2011 version and Annex 11 as its January 2011 revision, and carries no Annex 22 at all. Everything quoted from them below is a draft, and is marked as one each time. The product examples come from a fictional sample network inside the solution, with invented plants, products, people and lots.
Can you use AI in GMP manufacturing?
Yes, within limits the regulators are in the middle of writing down, and the limits depend entirely on what the model is being asked to decide. The EMA's position for manufacturing is that the question is when, not whether: its reflection paper anticipates more use of AI and machine learning across process design and scale-up, process optimisation, in-process quality control and batch release in the years ahead (EMA reflection paper, section 2.3.6), and expects model development, performance assessment and lifecycle management to follow quality risk management principles, taking patient safety, data integrity and product quality into account.
What is not open is the idea that an AI output can stand on its own in a GMP decision. The texts come at it from different directions, but none of them lets a model's output stand in for a qualified person's judgement on a GMP decision. The difference between a tool that can be deployed next quarter and one that cannot is almost always that person's role: whether they are a genuine checker with the source in front of them, or a signature at the end of a workflow.
Where authority sits when Batch Record Search answers a question
A person decides
The question is askedA quality reviewer, MSAT lead, site quality or regulatory colleague asks in plain words about a lot, a step, a parameter or a plant.
Agent
Record, page and step foundLocates the plant, product, lot, version and language, and maps the named step to its page in the scanned record.
Never: Never edits a source record — it is read where it lives, read only
Agent
The entry is readReads the handwritten entries and both sets of initials, and puts the original step beside its English translation with units and step numbers unchanged.
Never: Never changes a number, a unit or a step number
Agent
Criticality and filing checkedFinds the parameter in the effective criticality analysis and its registered range in the dossier, and states the difference if there is one.
Never: Never clears a difference from the filing — it always goes to a person
Agent
The answer is writtenStates the finding with the document, page and step cited for every claim, or says it was not found.
Never: Never rounds, converts or merges a process range
A person decides
A reviewer verifies at the pageOpens the cited step on the scanned record and checks the answer against the source rather than against a summary of it.
A person decides
A reviewer signs, with a meaningChooses what the signature attests to and where the answer may be used, confirmed by single sign-on, and the meaning is kept with the cited versions.
Never: No agent releases a lot or closes a deviation
What the draft Annex 22 says, and what it does not cover
Annex 22 is a proposed new annex to EU GMP covering AI models used in critical applications in the manufacture of medicinal products and active substances. Its scope is narrower than its title suggests, and the narrowing is the most consequential paragraph in the draft. It applies to machine learning models that obtained their functionality through training rather than explicit programming; to static models, not ones that keep learning in use; and to models with deterministic output, which given identical inputs return identical outputs.
Then comes the sentence that decides most buying conversations. The scope section says the annex does not apply to generative AI and large language models, and that such models should not be used in critical GMP applications; where they are used in non-critical ones — applications with no direct impact on patient safety, product quality or data integrity — suitably qualified and trained personnel should always be responsible for checking that each output is fit for its intended use, which the draft calls a human-in-the-loop (draft Annex 22, section 1).
Read it carefully, because it is easy to misread in both directions. It does not ban language models from GMP sites. It says they are outside this annex's scope, should not be used in critical applications, and may be used in non-critical ones provided a qualified person is responsible for the suitability of each output. That is a workable rule, and it puts the burden exactly where a careful buyer would want it: on being able to say, clause by clause, which applications at your site are critical and which are not.
Critical and non-critical GMP applications: the line that decides everything
The draft's own definition of critical is direct impact on patient safety, product quality or data integrity. A model that classifies vials as accept or reject, or that decides whether a batch meets specification, is plainly on the critical side: its output is the decision. A model that finds the page a parameter is written on, and states what the page says with a citation, so that a qualified person can read the page and decide, is on the other side — provided the person really does read the page, and the system makes that possible rather than merely permitted.
A critical GMP application and a non-critical one, under the draft Annex 22
Critical: the model's output is the decision: Automated visual inspection that accepts or rejects units; a model that classifies a batch as in or out of specification.
Non-critical: the model finds and cites, a person decides: Finding the step in a 318-page scanned record, reading the entry, and citing the page it is on.
Critical: the model's output is the decision: On patient safety, product quality or data integrity — the output goes into the outcome.
Non-critical: the model finds and cites, a person decides: On neither, so long as nothing is written back to a record and no decision closes without a person.
Critical: the model's output is the decision: The draft says such models should not be used here.
Non-critical: the model finds and cites, a person decides: The draft allows it in non-critical applications, with a human-in-the-loop responsible for the output’s suitability.
Critical: the model's output is the decision: Full scope of the annex: intended use, acceptance criteria, independent test data, explainability, confidence thresholds, change control.
Non-critical: the model finds and cites, a person decides: The annex’s principles "may be considered where applicable"; the discipline is worth adopting anyway, and it is what a serious buyer should ask for.
Critical: the model's output is the decision: Approval of test results and features, and the governance around retraining and monitoring.
Non-critical: the model finds and cites, a person decides: Named, qualified, trained, and responsible for whether each output is suitable for its intended use.
Critical: the model's output is the decision: Could the output reach a patient without a person reading the evidence?
Non-critical: the model finds and cites, a person decides: Does the person actually have the evidence in front of them, in a form they can check in seconds?
Human-in-the-loop is a defined role, not a disclaimer
"Human-in-the-loop" has become a line in sales decks, which is unfortunate, because the draft gives it content. Clause 3.3 requires that where a model gives an input to a decision made by a human operator, and the effort to test the model has been reduced on that basis, the description of the intended use must include the operator's responsibility — and the operator's training and performance then have to be monitored the way any other manual process would be.
Clause 10.5 adds the record-keeping half: in that same situation, records should be kept of the human review process, and depending on criticality and the level of testing, that may mean reviewing or testing every single output the model produces, under a written procedure. In other words, if you are leaning on the person to make the system safe, the person becomes part of the validated process and has to be treated like one. A tool whose design makes checking slow or optional cannot carry that weight.
This is the clause to hold a vendor against when they describe an approval step. Ask what the reviewer sees, how long a real check takes, what is recorded about it, and what happens when the reviewer disagrees. In the sample network, a verification means opening the cited step on the scanned page itself, and the signature records what it attests to — that the cited steps match the sources, or that the answer is fit for use in a deviation investigation or a change control — rather than a bare approval.
Each step shown as it runs, so a reviewer can see what was read

What does the EMA reflection paper ask of manufacturers?
The EMA's reflection paper covers the whole medicinal product lifecycle and is explicitly a reflection rather than a guideline, but two of its paragraphs bear directly on this kind of tool. The manufacturing paragraph, quoted above, routes everything through quality risk management and points at ICH Q8, Q9 and Q10 while GMP standards are revised.
The sharper paragraph is about generated text. On product information the EMA warns that generative language models tend to produce output that sounds right but is wrong or incomplete, and asks for quality review mechanisms that confirm every piece of model-generated text is factually and syntactically correct before it goes to a regulator (EMA reflection paper, section 2.3.5). That is a statement about a failure mode, not about a document type, and the failure mode travels: a fluent, wrong answer about a hold time is exactly as dangerous as a fluent, wrong sentence in a submission.
The mitigation the EMA names is a quality review mechanism. For an answering tool the practical form of that is citation: every claim tied to a document, page and step, so that checking is reading rather than re-researching. If citations can be switched off, the review mechanism is optional, and an optional control is not a control.
The FDA's credibility framework and the idea of a context of use
FDA's draft guidance approaches the same problem from a different angle, and its vocabulary is worth borrowing even outside the United States. In its Federal Register notice the agency describes a risk-based framework for establishing and evaluating whether an AI model is credible for a particular context of use, and defines credibility as trust in the model's performance for that use, built by collecting evidence (FDA, Federal Register notice of 7 January 2025).
Two things follow. First, credibility is not a property of a model but of a model-plus-use: the same model can be credible for one question and not for another, which is why "is your AI validated?" is a badly formed question. Second, the scope note draws its own line. The draft guidance covers AI used in the nonclinical, clinical, postmarketing and manufacturing phases where the use is to produce information or data supporting regulatory decisions about safety, effectiveness or quality — and it explicitly leaves out drug discovery and AI used purely to make internal work more efficient — workflows, resource allocation, drafting a submission — where the use has no bearing on patient safety, drug quality or the reliability of study results.
A tool that helps a reviewer find a page faster is an operational efficiency. The moment its output is relied on as the evidence in a deviation investigation or a release decision, it is producing information supporting a quality decision, and the framework applies. The honest way to buy is to assume the second case, because that is what users will do with it in month three.
Who is accountable for an output a model produced?
The draft revision of EU GMP Chapter 4 answers this in one sentence, and it is the sentence to put at the top of any governance paper: "The accountability for the integrity of documents, records or (raw) data produced or processed with artificial intelligence or any other automatic means (e.g. validation scripts) rests with the regulated user." Not the vendor, not the model, not the person who approved the purchase.
That has a practical consequence for software selection. If the evidence behind an answer lives only inside the vendor's system, or the answer cannot be traced to the document version it was read from, then the regulated user has accepted accountability for something they cannot inspect. The test is simple: can a reviewer, at any later date, open the exact page and version each claim came from, from their own document system, without the vendor's help?
The draft's clause 4.3, headed "No decrease", sets the bar for any model that takes over manual work: its acceptance criteria should be at least as demanding as the performance of the process it replaces — which presupposes that the performance of that process is known (draft Annex 22, clause 4.3).
That presupposition is the part with teeth. Few sites know the measured performance of the manual process a model would replace — nobody has scored how often a person reading a 318-page scanned record in a second language gets the hold time, the criticality and the registered range all correct. A buyer who takes the clause seriously has to measure the manual baseline first, and that measurement is useful whether or not anything is ever bought.
Testing a model that answers questions: metrics, held-out data and "undecided"
The draft's testing clauses were written with classifiers in mind, but they translate. Clause 4.1 asks for suitable, case-dependent metrics defined against the intended use. For a question-answering tool the useful metrics are not a single accuracy figure but a set: whether the cited page is the right page, whether the value read matches the value written, whether the version cited was the effective one on the date in question, and how often the system declines to answer when it should.
Clause 6.1 is the one easiest to skip. It requires effective technical or procedural controls to ensure that test data was not used during development, training or validation, either by capturing test data only after training is complete or by splitting it out before training starts — and clause 6.2 adds that, where test data is split from a common pool, the people who developed and trained the model must never have had access to it. For a buyer that means asking who wrote the question set, when, and whether the vendor saw it.
Clause 9.2 is the one that should be in the product rather than the protocol: where a model's confidence is very low, the draft asks whether it should mark the outcome "undecided" instead of making an unreliable prediction (draft Annex 22, clause 9.2). In the sample network that idea appears as a setting and as an agent's stated limit — say "not found" instead of guessing — which is the same discipline in the product's own words. A tool that always produces an answer has no way to tell a reviewer that it is out of its depth.
The draft Annex 22 clauses a buyer should be able to evidence
| Draft clause | What it asks for | What a buyer should be able to show |
|---|---|---|
| 3.1 Intended use | A detailed description of the intended use and the input sample space, with limitations and possible erroneous inputs identified, approved by a process expert before testing. | A written statement of which questions the tool is for, on which document types, in which languages, and which it is not for. |
| 3.3 Human-in-the-loop | Where the model informs a human decision, the intended use must state the operator’s responsibility, and the operator’s training and performance are monitored. | A named role, its qualification, and how its checking is monitored — not a generic approval step. |
| 4.3 No decrease | Acceptance criteria at least as high as the process the model replaces, which means that process’s performance has to be known. | A measured baseline of the manual work, taken before the tool arrives. |
| 6.1 Test data independence | Controls ensuring test data was never used in development, training or validation. | Who wrote the question set, when it was written, and that the vendor had no sight of it. |
| 8.1 Explainability | Capture of the features that contributed to an outcome, using feature attribution or visual tools where applicable. | For an answering tool: the document, page and step behind every claim, visible to the reviewer. |
| 9.2 Threshold | A confidence threshold, and consideration of flagging an outcome as "undecided" rather than producing an unreliable one. | What the tool does when it cannot find the answer, and how often that happened in testing. |
| 10.1 Change control | The model, the system and the whole process under change control before deployment, with any decision not to retest justified. | What counts as a change, who approves it, and what triggers a re-test. |
| 10.5 Human review | Records kept of the human review process, possibly a review of every output, according to a procedure. | The audit trail of checks and signatures, with what each signature meant. |
Nine questions to put to a vendor before anything touches a record
The questions below are ordered so that a weak answer to an early one makes the later ones academic. None of them asks whether the product uses AI, and none asks for an accuracy figure in isolation — a score with no stated question set, no independence and no baseline is a number, not evidence.
Nine questions, each with what a weak answer sounds like
- What decision does your system make without a person?
A weak answer describes an approval screen. A good one names the decisions that are structurally impossible for the software to take.
- Is this a critical or a non-critical GMP application, in your words, and why?
A vendor who has not read the draft and cannot place their own product on that line has not done the work.
- What is the measured performance of the process we would be replacing?
"Customers tell us it saves time" is not a baseline. The draft expects the replaced process to have a known performance.
- Who wrote the test questions, and did you see them first?
If the vendor wrote and marked their own exam, the result tells you about the exam.
- Can every claim be traced to a document, page, step and version?
A weak answer offers a list of sources for the whole response. The clause-level answer is per claim.
- What happens when the answer is not in the documents?
Listen for whether the system can say so. A tool that always answers cannot tell you when it is guessing.
- Can citations be turned off, for speed or for tidiness?
If they can, they will be, and the review mechanism the EMA asks for goes with them.
- Does anything get written back into a GMP record?
Read-only is a far easier conversation with an inspector than write-with-audit-trail. Know which one you are buying.
- If we leave, what remains ours?
Accountability for records processed by automatic means rests with the regulated user, so the evidence cannot live only in the vendor’s system.
Where Batch Record Search sits on the line, and what it will not do
The honest placement is on the non-critical side, by design and by construction. It answers questions from master and executed batch records, criticality analyses, registered filings and hold time studies; it reads those documents where they live and never edits them; and it hands every answer to a named person with the page open. Its agents' own stated limits are the three that matter under the drafts: never round, convert or merge a process range; cite document, page and step for every claim and say "not found" rather than guess; and never clear a difference from the filing, which always goes to a person.
The governance around it is built the same way. Citations are required for every claim and cannot be turned off. Only effective versions are cited unless a question names a past one. A batch-record range outside the registered range is flagged with the size of the difference and routed to Quality and Regulatory CMC. Answers used in a deviation or change control need a named reviewer's signed check, and every question, answer, page opened and signature stays on the audit trail with the versions cited. No agent releases a lot, closes a deviation, or edits a source record.
The answer, the suggested next questions, and the page it all points at

What it will not do is as much a part of the specification as what it does. It will not decide that a deviation is closed, that a lot may be released, or that a difference from the filing is acceptable. It will not translate a number into a different unit to make two plants agree. It will not quietly use a superseded version because that version answered more neatly. Those are the places where, on any reading of the drafts, a qualified person has to be the one who decides.
If you are evaluating this class of tool, the two pages worth reading next are what an FDA investigator actually asks for, and how fast, which is where the evidence gets tested, and what an electronic batch record programme leaves behind, which is where most of the documents an answering tool has to read will still be living in five years. To see the sequence end to end on sample records, start from Batch Record Search.
Questions people ask
What does EU GMP Annex 22 say?
It is a proposed new annex, out for consultation and not in force, covering AI models used in critical GMP applications in the manufacture of medicinal products and active substances. It applies to static, deterministic machine learning models and sets requirements for intended use, acceptance criteria, independent test data, test execution, explainability, confidence and operation. It states that it does not apply to generative AI and large language models, and that such models should not be used in critical GMP applications.
Are large language models allowed in GMP?
In non-critical applications, under the draft, with a qualified human in the loop responsible for ensuring each output is suitable for its intended use. In critical applications — those with direct impact on patient safety, product quality or data integrity — the draft says they should not be used. Since the text is a consultation draft, a site adopting it now is adopting a direction of travel rather than a rule in force.
What is the FDA credibility assessment framework for AI?
A risk-based approach in FDA's draft guidance for establishing trust in an AI model for a particular context of use — the specific role and scope of the model in answering a question of interest. Credibility is established by collecting credibility evidence proportionate to model risk, so the same model can be credible for one use and not another. The draft covers nonclinical, clinical, postmarketing and manufacturing uses that support regulatory decisions, and excludes purely operational uses.
How do you validate an AI model in pharma?
By defining the intended use and its input space first, setting metrics and acceptance criteria against the process being replaced, testing on data the developers never saw, recording what features or sources drove each output, setting a confidence threshold with a route for "undecided", and putting the model, the system and the surrounding process under change control before deployment. The draft Annex 22 sets those out as clauses 3 to 10.
What is human-in-the-loop in GMP?
A defined arrangement, not a disclaimer. Under the draft, where a model gives an input to a decision made by a human operator and the testing effort has been reduced on that basis, the intended use must describe the operator's responsibility, the operator's training and performance are monitored like any other manual process, and records are kept of the review — possibly of every output, according to a procedure.
Who is accountable for an AI output in a GMP record?
The regulated user. The draft revision of EU GMP Chapter 4 states that accountability for the integrity of documents, records or raw data produced or processed with artificial intelligence or any other automatic means rests with the regulated user. That makes traceability a procurement requirement rather than a nice feature: if the evidence cannot be opened in your own document system, you are accountable for something you cannot inspect.