Document tampering detection dataset: test on your own claims first

A document tampering detection dataset is a collection of document images or files in which some have been deliberately altered, with the altered regions marked, so that a detector can be trained and scored. Several are public, from about a thousand scanned receipts to 170,000 synthetic document images. None is made of insurance claim documents, few contain realistic forgeries, and almost all predate the AI tools that now change a figure in a photo in under a second. For a carrier choosing or tuning a detector, that makes public benchmarks a starting point, not an answer. The test that predicts performance on your claims is a seeded one: take your own past invoices, medical bills and estimates, plant known edits of each kind, and count what the tool finds. Document Fraud Detection, Botminds' solution for testing claim documents before payment, is evaluated this way, and the method below works for any tool.

Is there a dataset for document tampering detection? Several, and none of them is your claims

Yes. The public sets differ in size, in how the forgeries were made and in how close they come to a claims mailroom. The closest to an insurer's work is a receipt set from the University of La Rochelle, presented at ICDAR 2023. Its authors start from the problem every buyer meets: "research in this field struggles with the lack of publicly available realistic data" (Martínez Tornés et al.). They had 19 volunteers forge receipts with ordinary software, including Paint and GIMP, and named insurance among their scenarios: proof of purchase for an insurance claim.

Each set is honest about its limits, and the limits are the lesson. DocTamper's images are generated by the authors' own synthesis method. The Real Text Manipulation set was built because, its authors write, most datasets rely on synthetic tampering, "which inadequately replicates real-world tampering attributes" (arXiv 2312.06934). The receipt set is realistic but small: 163 forged documents is enough to compare methods and too few to set a threshold for a book of claims.

Do public forgery datasets include AI edits? Almost none do

Almost none. A 2026 benchmark that ran 14 detection methods across eight public document datasets, a preprint not yet peer reviewed, notes that "all eight datasets predate the era of generative AI editing" (DOCFORGE-BENCH). The one public set built on AI edits, AIForge-Doc, changed a single number on each of 4,061 receipts and forms with two commercial image models, and found detectors that score well on older forgeries close to chance on these. Its results are set out in can AI make fake receipts.

The consequence for testing is direct. A tool's published score on a pre-AI dataset tells you how it handles the edits of five years ago. Your seeded test has to include AI-made edits, made the way a claimant would make them: a phone photo of a receipt with the total changed by an editing app, and a PDF invoice regenerated with a new figure.

Why a benchmark score does not transfer to your invoices

Scores fall when the documents change, and claim documents are a different population from any benchmark. Two findings show why. First, the edits are tiny. In document datasets the tampered region covers between 0.27% and 4.17% of the pixels, against 10 to 30% in the natural-photo benchmarks most detectors were built on (DOCFORGE-BENCH). A detector tuned to find a pasted object in a photograph is looking for something a hundred times larger than a changed total.

Second, a detector can rank pixels well and still make poor decisions, because its decision threshold was set on other data. The same benchmark found methods with moderate pixel-level AUC of 0.76 or more producing near-zero F1 at the standard threshold, and called it "a pervasive calibration failure". Its most useful finding for an insurer is the fix: "adapting a single threshold on N=10 domain images recovers 39-55% of the Oracle-F1 gap" in the high-AUC cases it tested (DOCFORGE-BENCH). The authors' conclusion is that threshold adaptation, not retraining, is the missing step, and their test needed only ten documents from the target domain. That is the case for testing on your own claims, in one sentence.

Real documents bring a third problem: honest marks. In the receipt set, 500 of the 988 receipts carried handwritten notes, highlights or stamps that are not forgeries, and 54 carried digital annotations. The authors kept them deliberately, because a detector has to learn "to distinguish between a fraudulent modification" and an innocent one (Martínez Tornés et al.). A claim file is full of these: an adjuster's stamp, a claim number written on a receipt, an invoice re-issued with a corrected address.

What is seeded error testing? Plant known edits in your own past documents

Seeded error testing means planting known errors in real work and counting how many a reviewer, person or machine, finds. The answer key exists before the test starts, so recall can be measured exactly, and the documents are your own, so the score describes your intake. The same method is used to test legal bill review, where auditors plant the errors and count what comes back, and claims triage, where testers plant injuries in first notices.

For document fraud the edit kinds should follow how documents are actually altered, and each needs its own count, because a tool that is excellent at one may be blind to another. NIST's review of digital forensics explains why one overall rate misleads: "Digital processes tend to have systematic errors rather than random errors. Therefore, an error mitigation analysis provides more information and is the correct way to manage uncertainty. An error rate is only useful where there are random errors." (NIST IR 8354). A tool does not miss one reprinted invoice in fifty at random. It misses reprinted invoices.

How many forged documents do you need to test a model?

Enough of each edit kind that the recall you measure has a usefully narrow range, which in practice means tens per kind, not a handful. The arithmetic is standard. NIST's statistics handbook recommends the Wilson method for a confidence interval on a proportion, the form used for "the proportion of defectives" (NIST/SEMATECH e-Handbook). Applied to a seeded test, the proportion is the share of planted edits caught.

Read the table as a budget. Ten planted edits of a kind tell you almost nothing: a tool that caught 9 could have a true recall anywhere from 60% to 98%. Fifty per kind narrows that to about 16 points. Eight edit kinds at 50 each is 400 planted documents, mixed into a base set several times larger so that the testers also measure how often honest documents are questioned.

What is recall in document fraud detection, and which false positives matter?

Recall is the share of altered documents the tool flags; precision is the share of flagged documents that were actually altered. In claims both have a price, but not the same one. A missed edit is paid. A false flag costs an adjuster's question, a delay on an honest claim and, sometimes, a vendor relationship. NIST's review adds a caution about reading either: "When error rates are provided, it is important for the user to understand the context of the numbers." (NIST IR 8354).

Measure false positives on the documents that cause them, not on a random sample. Re-issued invoices, scanned faxes, documents with handwritten claim numbers and invoices from vendors who use unusual templates will all raise more flags than a clean PDF export. Their rate tells you how much adjuster time the tool will cost. In the solution, a finding from the file alone raises a question rather than a refer, which limits what a false flag can do: it asks for a look, it does not stop a payment by itself.

Live operation keeps producing the evidence. The sample carrier's 30-day view lists template clusters, and one of them, three claims from one pharmacy with a different receipt font, is marked closed after a printer change was confirmed. That is a false positive found, explained and recorded, which is what a test-and-learn loop should produce.

Thirty days of findings, including a cluster closed as innocent

How often should document fraud models be retested?

Retest whenever something that drives the score changes, and on a calendar in between. The triggers are easy to name: a new version of the tests or thresholds, a new document source such as a vendor portal or a bill feed, a catastrophe that brings in vendors you have never paid, and new editing tools in the hands of the public. The last one moved fastest in 2025 and 2026, as image models learned to replace a number in a photographed receipt.

No insurance rule sets the interval. The NIST AI Risk Management Framework, which is voluntary, applies trustworthiness considerations to the use of an AI system as well as to its design and development, which argues for testing that continues after go-live rather than a one-off acceptance test (NIST AI RMF). For a document test the practical rhythm is a small seeded run with each version change, a full run at least yearly, and a standing count of questions cleared as innocent, which costs nothing because it comes from live work.

In the solution, versions make retesting tractable. Each change to a test level or threshold is published as a new version and applies only to documents received after it, so a seeded run can be tied to the exact rules it scored. The fraud manager owns the versions; the scores are her evidence for each change.

Each Document Fraud Detection rule change is a dated version you can test

Run a seeded test on Document Fraud Detection, with your own documents

The offer is the method on this page. Botminds runs Document Fraud Detection on a sample of the carrier's own paid and open claim documents, and the seeded set can be built from the same sample. The scores come back by edit kind, with every finding pinned on the page, so the carrier's SIU can check each one rather than trust a summary. The design target is that every claim document over $1,000 is tested before payment; a seeded test is how a carrier learns what that testing catches in its own mix.

Two cautions from the research belong in the test plan. Keep the people who plant the edits away from the people who tune the tool, or the test measures their memory. And include the edits the tool's builders did not think of: the AIForge-Doc authors found that detectors trained on traditional edits did not carry over to AI-made ones. For how the file evidence itself is read, see PDF forensics tools, inside the agent.

Questions people ask

Where can I download a document forgery detection dataset?

The receipt set from ICDAR 2023, DocTamper, Real Text Manipulation and AIForge-Doc are each released by their authors; the papers give the links and terms. Check the licence before using any of them commercially, and remember that none contains insurance claim documents.

How do you test document fraud detection on real claims?

Take closed, paid documents with no SIU history, have someone outside the test team plant known edits in a share of them, run the tool blind and score what it finds by kind of edit. Measure false flags on the untouched documents in the same run.

What is a good recall for document fraud detection?

There is no industry standard, and a single figure hides the edit kinds a tool misses. Ask for recall by edit kind, with a confidence range; with 50 planted edits of a kind, a measured 92% is consistent with a true recall between about 81% and 97%.

Is synthetic training data good enough?

For building a detector it helps; for judging one it is not enough. The Real Text Manipulation authors built their set because synthetic tampering inadequately replicates real-world tampering, and detectors trained on older edits fell close to chance on AI-made ones in the AIForge-Doc preprint.

How do you measure false positives on invoices?

Run the tool on untouched documents in the same mix you receive, including re-issued invoices, scans and documents with handwritten notes, and count every flag that turns out innocent. In the ICDAR 2023 receipt set, half of the 988 receipts carried honest handwritten marks or stamps, the kind of change that produces false flags.