LLM for data extraction: plant six errors and count what comes back

An LLM for data extraction reads documents and returns structured values: for a research team, the revenue, segment results, cash flow and KPIs in a company's 10-Q, release and deck. Large language models can do it, and published benchmarks show where they still fail: they locate numbers well, combine them less reliably, invent values that are not there, and break down on wide schemas. The honest way to decide whether a given system is good enough is a seeded-error test: take last quarter's real filings, plant the six errors that most often corrupt a model — a scale slip, year-to-date cash flow entered as the quarter, a recast history, a segment gap, a flipped sign and a relabeled KPI — and count, by type, which ones the system flags before an analyst sees them, which it misses, and how many false alarms it raises on clean cells. Botminds' Financial Data Extraction is designed around those six: separate agents read, convert, map, foot and compare history, and anything they cannot settle goes to a person with its page.

The benchmark figures below are from published research, labelled peer reviewed or preprint. The worked figures are fictional sample data from the solution, or our arithmetic, marked as such.

Can LLMs extract data accurately?

They can find and copy numbers accurately much of the time, and they become unreliable as soon as a task needs several steps, many fields or arithmetic. The research on financial documents agrees on the shape even where the numbers differ. TableQuest, built from financial reports, found seven leading models good at locating facts and much weaker once a question needed multi-step calculation (Pan et al., NeurIPS 2024 workshop). ExtractBench, a 2026 preprint on PDF-to-JSON extraction, found performance falling sharply as schemas widened, to no valid output at all on its widest financial reporting schema (arXiv 2602.12247, not peer reviewed).

FinanceBench is worth reading for its design as much as its result. Its authors built its questions about public companies to be simple and unambiguous, a floor rather than a hard exam (Islam et al., 2023). The models of 2023 fell well short of that floor; feeding more of the document improved results, but the authors judged that "unrealistic for enterprise settings due to increased latency" and unable to "support larger financial documents". Models have changed a great deal since 2023, which is the point of testing your own system on your own documents rather than quoting anyone's leaderboard. A broader review of the "LLMs beat analysts" literature, including two well-known papers since withdrawn by their authors, is in financial statement analysis with large language models, tested.

Do LLMs hallucinate numbers?

Yes, and in extraction the dangerous hallucination is a plausible number in the right place. FinanceBench's authors found hallucinations in every model they tested, enough to limit their use in a business (what FinanceBench measured). ExtractBench's method treats the problem as a scoring rule: omission must be told apart from hallucination, because a blank and an invented value are different failures (ExtractBench).

The opposite failure matters as much. Asked to verify financial statements, models can see errors that are not there. FinVerBench, a 2026 preprint built from S&P 500 companies' XBRL filings, reported that under its original prompt nine of fourteen complete model runs flagged errors on 95% to 100% of the clean statements, while one run raised none (Panda, arXiv 2605.29586, not peer reviewed). A checker that cries wolf on every cell is as useless as one that never speaks. A seeded test therefore has to count false alarms on clean cells, not just catches on planted ones.

What errors do LLMs make on financial tables?

The costly ones are context errors more than character errors: the right digits under the wrong scale, the right column for the wrong period, the right label with a changed meaning. They are also the errors a research associate checks for by hand every night of earnings season, which is why they make the best seeds. The table below sets out six, how to plant each, and what a good system does.

A seventh family is worth adding if the team covers non-US names: number formats. A German quarterly prints 412.386 for four hundred twelve thousand and 1,31 for one point three one; plant a value where the separator decides the magnitude. The guide to PDF table extraction for filings lists the table traps these seeds are drawn from.

How do you test data extraction accuracy? Seed the errors

Replay a past quarter with an answer key. Choose a set of covered companies that reported last quarter, including a few small caps, one or two non-US filers and at least one scanned document. Take the filings as they were, make copies, plant the six seeds in the copies, and give the system the planted versions. The answer key, written before the run, lists every planted error and every clean value the analysts' models held.

Then count three things by seed type. Caught: the system flagged the planted error, with the right page and a sensible proposal. Missed: the error reached the model, or would have, without a flag. False alarms: clean values the system flagged as wrong. Time matters too: how long from the filing to a model a person could accept.

Sample size decides what a clean result means. If a system catches every one of n independent planted errors of one type, the 95% upper bound on its true miss rate is 1 − 0.05^(1/n): about 26% for 10 seeds, 14% for 20, 9.5% for 30 and 4.9% for 60, by our arithmetic. Twenty seeds of each type is a reasonable floor for a decision; a handful proves only that the system is not useless. The same logic is applied to loan spreading in a twelve-error test for financial spreading software.

Reading the result: what a pass and a fail look like

A pass is a system that flags nearly every seed of every type, with the right page, while sending a manageable number of clean cells to people. A fail comes in three shapes, and each points to a different fix. A system that misses one seed type entirely, say every recast, lacks a check, and no amount of tuning will find what it never compares. A system that catches everything but flags a large share of clean cells has thresholds set too tight, or a checker that cannot tell a real break from noise, the failure FinVerBench describes. A system whose catches cite the wrong page has not really caught anything: the person reviewing it will spend the saved time finding the evidence.

Two numbers decide whether the result is usable. The first is unflagged errors that would have reached a published model: the target is none, and the ones that occur say where to add a check. The second is people's time: how many cells a day the system sends for review at the chosen thresholds. A test that ends with both numbers known is one a head of research data can take to a decision, and one that can be repeated after every change of model, prompt or threshold.

Running the test on Financial Data Extraction

The solution's first-time setup already contains the replay. A team connects an analyst's workbook, reviews the line mapping the agents propose, picks the covered names, sets the checks and runs a trial on last quarter, compared with what the team typed, before anything is written into a live model. Seeding is a matter of giving that trial the planted copies instead of the originals and keeping the answer key outside the system.

Each catch arrives where a person would see it in production: a footing failure on the cell, with the reconciling line proposed and its page; a recast in the exceptions list with old and new values; a relabeled metric as a question with both choices. Every step and decision is on the audit trail, and every edit, rejection or return carries a reason code, so the trial's own record becomes the scoring sheet. The thresholds under test, the 0.92 mapping floor, the 0.85 scan floor and the footing tolerance, are settings that a second administrator must approve to change, which keeps the system that passed the test the same system that runs afterwards.

What is the best LLM for data extraction? Usually the wrong question

The model matters less than what surrounds it. A single model asked to read 84 pages and return every value in one pass is being asked to do the wide-schema task ExtractBench found models fail at. The same model asked to do one narrow thing at a time, read this table and record each value with its page, row, column and printed units, is doing the task TableQuest found models do reasonably well. Everything that needs reasoning across values, conversion, derivation, footing, comparison with history, can then be done by steps that show their arithmetic and can be checked.

That is how Financial Data Extraction is built. Seven agents each have one job and written limits: the Table Reader never infers a number that is not printed; Units and Periods never converts with a rate other than the house rate and shows its arithmetic on every derived value; the Mapper asks below 0.92 confidence and always on a changed definition; Footing never plugs a difference; the Restatement agent never overwrites published history. In the fictional sample run on Corvane Industrial, Footing runs 46 checks, 45 pass, and the one that fails becomes the associate's first decision.

The thresholds a seeded test exercises, as settings in Financial Data Extraction

How do you validate extracted financial data in production?

With checks that cannot be argued with, run on every filing, and a person on everything they flag. Footing is the strongest: segments to consolidated revenue, statement subtotals, the balance sheet, the cash flow roll-forward and every non-GAAP reconciliation back to its GAAP figure. Cross-checks come next: the SEC's structured data for the face statements of US filers, which the SEC itself warns it "cannot guarantee" (SEC, Financial Statement Data Sets), and a vendor dataset where one exists, both as second opinions rather than sources. History comparison catches recasts. Estimates catch the improbable: a value far from the analyst's estimate is highlighted for a look.

The control around the checks matters as much as the checks. Among the points FINRA's 2026 oversight report suggests firms using generative AI consider is monitoring through "validation and human-in-the-loop review of model outputs, including performing regular checks for errors or bias" (FINRA, 2026 Annual Regulatory Oversight Report). In the solution, an associate accepts each column, a different senior analyst checks recasts and footing decisions, every override needs a reason code, and the thresholds themselves change only with a second administrator's approval.

What the agents and people did in a season, in Financial Data Extraction's reports

Questions people ask

What is the best way to evaluate an LLM for data extraction?

Run it on your own past filings with planted errors and an answer key, and score caught, missed and false alarms separately for each error type. Published benchmarks are useful for direction but measure different tasks, models and documents from yours.

Can an LLM extract data from a PDF into Excel?

Yes, but the output needs its page, row, column and units kept with each value, and units converted explicitly, or no one can check it later. For financial models it should write into the existing workbook under the analyst's line names rather than into a new sheet.

How many test documents do you need?

Enough to plant about twenty errors of each type across varied companies and pages. With zero misses in twenty independent seeds, the true miss rate could still be as high as about 14% at 95% confidence; with sixty, about 5%.

Should extraction run without a person in the loop?

Not for numbers that reach a published model. Even a high-confidence read can sit in a table that does not foot; in the solution's sample, net sales read at 0.99 confidence still went to the associate because the segments did not add up.

Do I need a different LLM for each language?

Not necessarily, but each language and market needs its own seeds: number formats, scale words and local report types. Reading the original rather than a translation avoids a known failure, since machine translation is unreliable on numbers.