Automated redaction software should be bought on evidence, and the evidence that matters is recall on documents like yours — the share of the identifiers actually present that the tool finds — measured by scoring its output against a package you have already published and know the answers to. Accuracy, the number most often quoted, is close to meaningless here: it averages two errors whose costs are nothing alike. A false positive costs a reviewer a few seconds to reject. One missed subject number in a 2,000-page clinical study report can undo the anonymisation of the whole package. Botminds' Clinical Trial Disclosure & Redaction is built on that asymmetry — it proposes rather than applies, sends anything below a confidence bar to a person one at a time, and links every identifier across the package so a miss in one place shows up as a disagreement with another.
The study, sponsor, medicine and people in the examples below are invented sample data used to demonstrate the solution. The measured figures come from the published studies linked beside them, which are not measurements of this or any other product.
How accurate is automated redaction, really?
The honest short answer, from the published evaluations, is: good enough to be useful, nowhere near good enough to be unsupervised. One large clinical evaluation built a gold standard of "30 815 protected health information elements and more than one million tokens" from 3,503 clinical notes across 22 note types at a large paediatric hospital, and reported that "The most accurate NLP method had 91.92% sensitivity (R) and 95.08% precision (P) overall" (Deleger and colleagues, Journal of the American Medical Informatics Association, 2013). Sensitivity there is recall: about eight in every hundred identifiers were not found.
More recent measurements of general-purpose anonymisation tools on unstructured clinical text are lower, and they are the reason the word "accuracy" has to be handled carefully. A 2025 evaluation ran 200 neurosurgical documents spanning ten years through two open tools against a manually established ground truth. A median of eight identifiers per document were found by hand; each tool found a median of six. The authors report that "Each tool de-identified PHI with an accuracy of 96%", and then, in the same results: "Presidio demonstrated precision of 0.51 and a recall of 0.74, while Philter had precision and recall of 0.35 and 0.79" (Alrazihi, Biswas and George, Surgical Neurology International, 2025).
Read those two sentences together, because a buyer will meet them separately. Ninety-six per cent accuracy and a recall of 0.74 describe the same run. Roughly a quarter of the identifiers present were not found by one tool, and a fifth by the other, and the figure in the headline did not say so.
Recall in published evaluations: what was actually found
Recall, precision, and why they are not interchangeable here
Two measures, one package, opposite consequences. Recall asks what share of the identifiers that were there did the tool find. Precision asks what share of the things it marked were really identifiers. Both are worth measuring; only one of them can fail an anonymisation.
The two numbers, written over the same study report
Recall
found ÷ (found + missed)
Of every identifier genuinely in the package, the share the tool proposed. The only number with a compliance consequence.
Scored against a gold standard you built
Precision
real ÷ (real + false alarms)
Of everything the tool proposed, the share that was genuinely an identifier. Low precision costs reviewer time.
Scored on the same package
Accuracy
agreement over all decisions
Dominated by the vast majority of spans that are not identifiers and were never going to be marked. Reported alongside a recall of 0.74 in one 2025 evaluation.
Report it if you must; never buy on it
Page-level leakage
pages with at least one miss ÷ pages
The measure that matches the regulatory failure: a page is unsafe if one identifier survives on it.
The number to put in a pilot report
There is a second reason recall dominates, specific to clinical disclosure. A missed identifier is not merely an error in one place: it changes the risk arithmetic for the subject it belongs to, because it puts them back in a smaller group than the published document was supposed to leave them in. A re-identification risk assessment computed over the decisions you made cannot see the ones you never made.
What accuracy should you require from redaction software?
Neither EMA, under Policy 0070 or for CTIS, nor Health Canada publishes a required recall figure, so a vendor quoting one as a standard is quoting itself. What the regulators say is narrower and more useful: automation is permitted, and the sponsor remains answerable for the result. EMA's CTIS guidance puts it plainly — redaction of pre-specified identifiers "can be done manually and/or automatically with software functionalities which enable the user to identify the pre-specified identifiers intended for redaction" (EMA, Protection of personal data and CCI while using CTIS, v2.2, §3.3.1). Nothing there about how well the software must do it.
So the requirement has to be written by the buyer, and the sensible way to write it is as a pair of numbers with different jobs. Set a recall target per identifier class, measured on your own documents, and treat it as a screening threshold rather than a guarantee: a tool that finds 85% of subject numbers in your listings is not safe to publish from, but it is a tool worth reviewing behind. Set a separate, much softer precision target, because precision only buys back review time. And set a page-level target, because that is the one that matches how a package fails.
What to measure in a pilot, and what to do with each number
| Measure | How to score it | What a result means |
|---|---|---|
| Recall by identifier class | Separately for subject numbers, dates, ages, sites, investigator names, batch numbers. Pooled recall hides the class that matters. | The class with the worst recall sets your review effort, and a pooled figure can hide it behind the easy classes. Score batch and serial numbers separately: they look like data rather than like a person. |
| Precision by identifier class | False proposals as a share of all proposals in the class. | When most proposals in a class are wrong, reviewers risk rejecting by reflex, which is how a real identifier gets rejected too. |
| Page-level leakage | The share of pages carrying at least one missed identifier. | The honest headline for a pilot. One miss makes a page unsafe, so this number is always worse than recall and always more relevant. |
| Document-level leakage | The share of documents with at least one missed identifier anywhere. | In a long document it can approach one even at good recall, because a single miss anywhere counts. That is not a reason to reject automation; it is the reason a person still accepts every redaction. |
| Consistency failures | Identifiers hidden in one place and readable in another, counted across the whole package. | The failure mode unique to multi-document packages, and invisible to any per-page score. |
| Reviewer minutes per 100 pages | Timed, on the same pages, by the same reviewers. | The only number that tells you whether the tool pays for itself. Measure it after reviewers have stopped learning the interface. |
How good are people at this? The human baseline
The comparison that changes most buying conversations is not software against perfection. It is software against the people doing the work today. In the JAMIA evaluation, two human annotators were scored on the same text as the automated systems. Their recall was 92.15% and 94.55%, with precision of 93.95% and 88.45%; the best system reached 92.91% recall and 95.73% precision on that same subsample, and the authors concluded that its performance "was indistinguishable from that of human annotators" (Deleger et al., JAMIA 2013).
Two readings follow, and they point the same way. The first is that a manual process is not a baseline of 100%: a team marking 2,140 pages by hand misses identifiers too, and nobody is counting. The second is that if careful humans miss roughly one identifier in fifteen, a single pass by anyone — person or software — is not a control. What makes a disclosure package safe is a second, different look: a check that compares every occurrence of an identifier with every other occurrence, which neither a tired reviewer nor a per-page model performs by default.
Page-level and document-level leakage
Entity-level scores flatter every system, because most identifiers are easy and the hard ones are rare — and because a page fails if any one of its identifiers survives. The arithmetic is simple enough to do on the back of a pilot plan. Take a recall of 0.92, close to the best figure in the clinical-notes evaluation above, and a listing page carrying ten identifiers. If misses fell independently, the chance that all ten are found is 0.92 multiplied by itself ten times, about 0.43: some 57% of such pages would still carry at least one readable identifier. At twenty identifiers a page the share rises to about 81%.
That calculation is illustrative — real misses cluster on particular layouts and particular identifier forms rather than falling independently — but the direction does not change: a respectable entity-level recall is compatible with most pages being unsafe to release on their own. The densest pages are the listings, and the hardest are the scanned signature pages, consent documentation and annotated case report form pages, where text recognition has to succeed before detection can. Any pilot report that stops at entity-level recall is reporting the flattering number.
One overnight run on the sample package, measured the way a pilot should be
218
minutes — 02:10 to 05:48, across 2,140 pages and three documents
6,214
redactions proposed, each matched to one of 9 rules with its reference
214
proposals below the 0.90 confidence bar, each sent to a person one at a time
14
identifiers hidden in one place and readable in another, found by comparison rather than by detection
The back-test: scoring a package you have already published
The only test worth running before a purchase is the one where you already know the answers. Take a package you published a year ago. You have the original documents, the published versions, and the decisions in between. Run the tool over the originals, and score what it proposes against what was actually redacted — treating your own published package as the gold standard, with the caveat that it is a human gold standard and therefore imperfect in the direction that flatters the tool.
How to design a back-test that tells you something
- Choose a package you already published, not a sample the vendor prepared
Vendor samples are selected; your own package has the scanned pages, the split table cells and the listing footnotes that make this work hard.
- Include Appendix 16.2, not only the report body
The listings hold most of the identifiers and all of the awkward ones. A back-test on the body alone measures the easy half.
- Build the gold standard from what was published, then have someone re-check a sample of pages
Your previous redaction is the reference; a fresh look at, say, 50 random pages tells you how good the reference itself is.
- Decide in advance what counts as a miss
An identifier found but given the wrong rule, a date offset the wrong way, an ID found in one place and not another — agree the taxonomy before anyone sees the result.
- Score at three levels: identifier, page, document
Report all three. Expect them to tell three different stories, and let the page-level number lead.
- Score the things that are not detection at all
Consistency across documents, deterministic recoding, metadata removal, scanned-page handling. These are pass or fail, not percentages.
- Time the review on the same pages, with your own reviewers
A tool that finds 95% and costs more reviewer time than it saves has failed the only test that decides renewal.
- Repeat the run to confirm it is deterministic
Same input, same output. If a second run proposes something different, the recoding and the audit trail are both in question.
The run, with its timings and what it left for a person

What a confidence threshold is really doing
Every automated redaction tool attaches a number to each proposal, and buyers routinely mistake it for a measurement. It is not. A confidence score is the model's own estimate, calibrated — if at all — on data that is not yours. The honest way to use one is as a routing rule rather than a quality claim: above the bar, proposals can be accepted a page at a time, because the reviewer is sampling a population that is mostly right; below it, each one goes to a person on its own.
In the invented sample configuration the bar sits at 0.90 and 214 of 6,214 proposals fell below it. That is a workload statement — 214 individual decisions — and nothing more. It does not mean 97% of proposals were correct, and a vendor who presents a confidence distribution as evidence of recall is answering a different question from the one you asked. Ask instead what the bar costs you at two settings: raise it and more proposals come to a person one at a time, lower it and more are accepted in bulk. That trade is what you are buying.
The bar, and the rest of the settings it sits among

False positives cost reviewer time, not compliance
A tool that over-marks is annoying; a tool that under-marks is dangerous. That asymmetry should shape the settings, and it has a cost worth naming. EMA's own guidance pushes against blanket redaction: not every indirect identifier needs to be redacted or transformed, and the decision depends on the number of indirect identifiers per participant, group sizes, the re-identification potential of specific identifiers, data utility and the size of the reference population (EMA Policy 0070 external guidance, v1.5, §3.5.3.2.2). Over-redaction costs the published report its usefulness, which is the thing publication exists for.
There is a subtler cost too, and it is the reason precision deserves a target even though it has no compliance consequence. A reviewer working through a queue where half the proposals are wrong stops reading each one. Rejection becomes a rhythm, and somewhere in that rhythm a real identifier gets rejected with the rest. A low-precision tool does not merely waste time; it degrades the human check you are relying on to catch what the tool missed.
What to write into a pilot before it starts
Most pilots fail to settle anything because nobody agreed in advance what would count as a pass. Four things should be written down before the first page is processed. The documents: which package, which appendices, how many pages, including the scanned ones. The gold standard: what it is, who built it, and how it will itself be sampled for error. The scoring: the measures, the levels, and the definition of a miss. And the decision rule: what result leads to a purchase, what result leads to a second pilot, and what result ends it.
Add two acceptance tests that are not statistical and are worth more than any percentage. Hand the tool a package where you have deliberately left one subject number readable in a listing footnote, and see whether it is found — that is the listings problem in miniature. And ask for the same package to be processed twice, to confirm that identical input gives identical output, because a recoding scheme that is not deterministic cannot produce two documents that agree.
The check that is a comparison, not a detection

Judged that way, the question stops being how accurate the software is and becomes how much of the work it can take off a person without taking the decision away from them. To see proposals with their rule and their confidence, and the places a person still decides, start from Clinical Trial Disclosure & Redaction. The capability questions behind the measurement — excision, scanned pages, metadata — are in what to look for in document redaction software.
Questions people ask
What is recall in automated redaction, and why does it matter more than precision?
Recall is the share of the identifiers genuinely present that the tool proposed; precision is the share of its proposals that were genuinely identifiers. Only recall has a compliance consequence, because a missed identifier can be published while a false alarm is simply rejected by a reviewer. Published evaluations of clinical text report recall between roughly 0.74 and 0.92 depending on the system and the corpus.
How do you validate redaction software for a regulated process?
Back-test it on a package you have already published, with the previous redaction as the gold standard, and score at identifier, page and document level separately. Add the non-statistical checks: deterministic output on a repeated run, metadata removal, scanned-page handling, and consistency across documents. Then write the result, including what was missed, into your validation record rather than quoting a vendor's figure.
Is 96% accuracy good for a redaction tool?
Not as stated. In the 2025 evaluation of two open anonymisation tools, an accuracy of 96% sat alongside recall of 0.74 and 0.79 on the same documents — a fifth to a quarter of the identifiers were not found. Accuracy is dominated by the enormous number of spans that are not identifiers and were never going to be marked, so ask for recall, per identifier class, on documents like yours.
Do humans do better than automated redaction?
Not dramatically. In a large evaluation on clinical notes, two human annotators reached 92.15% and 94.55% recall while the best automated system reached 91.92% overall and 92.91% on the same subsample, which the authors described as indistinguishable. The useful conclusion is that a single pass by anyone is not a control, and that the second look should be a different kind of check rather than more of the same.
What is page-level leakage?
The share of pages that still carry at least one identifier after redaction — the measure that matches how a package actually fails, since a page is unsafe if even one identifier survives. Because one miss is enough, it is always worse than entity-level recall: if misses were independent, a recall of 0.92 on pages carrying ten identifiers would leave roughly 57% of them with at least one readable.
Should a confidence score be treated as a measure of accuracy?
No. It is the model's own estimate, usually calibrated on data that is not yours, and it is best used to route work: above the bar, accept a page at a time; below it, decide one at a time. In the sample configuration a 0.90 bar left 214 of 6,214 proposals for individual decisions — a statement about workload, not about correctness.