Data extraction from PDF using an LLM means splitting a document into pages, finding the pages most likely to hold the figure you need, giving the model only those pages with precise instructions, and asking it to return the value with its unit, year and the page it came from. On corporate sustainability reports this works well enough to be useful and badly enough to need checking: in the central banks' Project Gaia, the model's extracted values matched a human's in 79.8% of a random sample, and the misses came from complex tables, mismatched definitions and missing years. One model reading a table is therefore a good guess, not a figure you can publish. Botminds' ESG Data Extraction is built around that finding: two agents read every table independently, one blind to the other, a units agent writes the conversion out, and a person confirms every issuer before a different person approves it.
The research results below come from central banks and peer-reviewed studies, linked where they appear. The examples come from sample data on a fictional asset manager, Quillfield Capital, and its holdings, as they run through the solution.
What one checkable value carries, beyond the number
Value
1,421,604
Scope 1, in template units
Unit and gas
tCO₂e
The table says thousand tCO₂eq; the conversion is written out
Year
FY2025
Fiscal year January 1 to December 31, 2025
Boundary
Consolidated
Per template rule B-01; the parent-only figure kept beside it
Source
p. 87, Table 4-2
The cropped Korean table, opened in one click
Readers
Two, agree
The second read the cell blind, by a different method
Checks
Plausible
+3.0% on last year; inside the peer range; vendor estimate 17% lower
Signatures
Analyst, then steward
Two different people before it is published
How does data extraction from PDF using an LLM work?
A working pipeline has four steps: split the document into units small enough for the model, retrieve the units most likely to contain each target figure, prompt the model with those units and a definition of the figure, and parse its answer into a value with its source. Project Gaia, a collaboration between the BIS Innovation Hub's Eurosystem Centre, the Bank of Spain, the Deutsche Bundesbank and the European Central Bank, documents each step for climate disclosures: reports are split into pages, each page is stored with a semantic vector, and a similarity search picks the five pages most likely to hold a given KPI (BIS, Project Gaia, March 2024).
The prompt carries more weight than most buyers expect. In a peer-reviewed test on the 2023 sustainability reports of 16 European airline groups, "incorporating explicit KPI terms into prompts increased accuracy from below 30% to above 70%", and HTML versions of reports gave "superior extraction results compared to PDFs" (Martín-Domingo et al., Transportation Research Interdisciplinary Perspectives, 2025). The definition of the figure, written into the prompt, is as much a part of the system as the model.
The last step, the source, is what makes the output usable. Gaia's prompt asks for "The sources used, including the page number and a short quotation from that page", because the central-bank working group behind it "emphasised the need to be able to trace an extracted KPI to the source". A value without a page has to be found again by a person before anyone can trust it, which undoes most of the saving.
How accurate is LLM data extraction from PDF? What the published tests found
Published tests on sustainability reports put field-level accuracy in the high 70s to around 80%, with careful design. None is a benchmark for any particular product, and each used different reports, fields and models, but they agree on the order of magnitude and on where the errors come from.
Three published tests of LLM extraction from sustainability reports
79.8%
of Gaia's extracted values matched a human's manual check (104 cases)
76.9%
accuracy in data extraction for ESGReveal on 166 Hong Kong-listed companies' reports
<30% → >70%
airline KPI accuracy once explicit KPI terms were added to prompts
Gaia's test is the most transparent. The team drew 163 random cases and checked them by hand. In the 104 where Gaia returned a value, 79.8% were correct, 18.3% diverged and 1.9% could not be decided even by the human tester. In the 59 where Gaia found nothing, the tester confirmed in 98.3% of cases that the information really was not in the documents (BIS, Project Gaia).
Gaia's manual cross-check: right when it answers four times in five, and honest when it doesn't
Two readings of those numbers matter for a buyer. A one-in-five error rate on extracted values is too high to publish unchecked and far too useful to ignore. And the high score on "not found" shows that a model told to say so when information is missing will mostly say so, which is the behaviour a regulated statement needs: a blank is honest, a guess is not.
Where LLM extraction fails on sustainability reports
The failures are specific, and each has a design answer. Gaia's team named three: "overly complicated table structures that the LLM was not able to interpret"; reports deviating from the required definition, such as a "total emissions" that is Scope 1 and 2 only; and missing years, where the model "incorrectly reported the value for the last available year". It also noted that figures inside infographics were not extracted. ESGReveal's authors say the same of their system: it does "not process pictorial information".
Units are the fourth failure and the one that reaches print. When Gaia's team compared its Scope 1 values with commercial data sources, it had to define a match as "within 2% after aligning the order of magnitude (correcting for diverging metric prefixes: eg kg/t or GWh/kWh)": the same quantity turns up in different scales across reports and data sets. The supervisors' review of PAI statements found the same class of error in published numbers: CO2 reported instead of CO2 equivalent, and one statement in US dollars instead of euros (ESAs, JC 2025 26). Scale and gas are where a correct reading of the digits still produces a wrong figure.
How to stop LLM hallucination in data extraction
Hallucination in extraction is a model producing a value the document does not contain, and the defence is to make every value prove where it came from. Gaia met a vivid case. During testing, many extracted KPIs came back as the same figure, 1,500,000 tonnes of CO2 equivalent, quoted from two reports named "Corporate_Report_2020" and "Annual_Report_2020". Neither report existed; the model had invented them, with convincing quotations. The team reduced the problem with a newer model, a prompt "instructing the LLM only to refer to information given within the provided context and explicitly stating what to respond if no information is found", and a temperature of zero (BIS, Project Gaia).
Those three measures lower the rate; they do not prove a particular value. Proof needs the source to be checkable by someone other than the model. The ChatReport system, presented at EMNLP 2023, made the same choice for analysing sustainability reports: it tied each answer to its source so that invented content could be caught, and kept domain experts in the loop (Ni et al., EMNLP 2023 System Demonstrations). A cited page turns a hallucination from an invisible error into a failed check.
Gaia's report also names the subtler failure: overconfidence. Asked whether a company had partnerships with environmental organisations, the model often concluded that it had none, with "a high level of certainty", because nothing in the pages it was given said so. In extraction terms, "not found in these pages" is not "not disclosed by the company". The fix is to keep a list of the pages searched beside every "not disclosed".
Two readers, one table: what a second blind reading buys
A second reading by a different method turns an unknown error rate into a visible disagreement. If two readers make independent mistakes, the cases where both are wrong in the same way are far rarer than the cases where one is, so agreement becomes evidence and disagreement becomes a queue. The condition is independence: the second reader must not see the first reading, and must not be the same method run twice.
One reader and a spot check, against two blind readers and a rule
One model, then a sample check: A sample, chosen after the reading.
Two blind readers, then a person on every disagreement: Every disagreement, every outlier and every unit or boundary choice, plus a confirmation of each issuer.
One model, then a sample check: Found only if the sampled value is checked against the page.
Two blind readers, then a person on every disagreement: Caught when the readers or the units step disagree, or when the figure jumps against last year.
One model, then a sample check: Looks plausible; usually passes.
Two blind readers, then a person on every disagreement: The year is part of the reading; two readers must agree on it too.
One model, then a sample check: Possible to publish if it is plausible.
Two blind readers, then a person on every disagreement: Cannot be confirmed: it has no page to open.
One model, then a sample check: A reviewer's hours on the sample.
Two blind readers, then a person on every disagreement: Machine time for the second read; a person's time only where the readers disagree.
On the fictional sample, the overnight run shows the economics. The second reader read 4,812 figures blind and sent 41 disagreements to analysts: under 1% of figures needed a person to choose between readings, and each came with both readings and the page. The rest still pass an analyst's confirmation, but as a check of cited values rather than a re-read of the report.
Asking ESG Data Extraction's agents about a value, with the page behind every answer
Which is the best LLM for PDF extraction?
The published research does not crown one, and the evidence says the question is the wrong one. Gaia found its newer model "adopts a more conservative approach instead of providing potentially incorrect values" compared with its predecessor; the airline study compared three models and found that prompt specificity and document format moved accuracy most. Models change every few months; the design around them changes slowly.
The useful test is the model on your own documents, inside your own design. Choose the fields, take reports whose correct values you already know, and measure the right answer, the wrong answer and the honest "not found" separately, by field and by language. A model that is 2 points better on English annual reports and 10 points worse on scanned Indonesian ones is worse for a global book. How to run that test on last season's reports is set out in its own guide.
What to ask before you buy an LLM extraction tool
Eight questions for any LLM extraction tool, with the answer to listen for
- Does every value come with its page, table and cell?
Listen for: yes, opened in one click, in the original language. A page number alone is not enough on a 300-page report.
- Is "not disclosed" an answer, and does it list the pages searched?
A blank and a zero are different answers; the supervisors criticised zeros used for missing data.
- Is there an independent second reading?
By a different method, blind to the first. The same model run twice is not independent.
- Who writes the units, fiscal-year and boundary rules?
Your template's owner, versioned. The tool applies them and writes the arithmetic out; it never changes them.
- What happens to scanned pages, charts and infographics?
Listen for a low-confidence queue, not a silent best guess.
- How are disagreements and outliers decided?
By a named person with a reason code, never released by the tool itself.
- Can you test it on your own reports before you buy?
On issuers your analysts already did by hand, compared field by field.
- What is logged, and for how long?
Every reading, conversion, decision and model version, read-only, long enough for your audit cycle.
The responsibility question sits under all eight. ESMA's 2024 statement on AI in investment services is plain that a firm's management body answers for its decisions however they were reached, by people or by AI, and it names the ESG assessment of investment products among the uses of AI on unstructured data (ESMA, May 30, 2024). A tool that cannot show who confirmed a figure leaves the firm unable to show that it took that responsibility.
Where ESG Data Extraction fits, and where it does not
The solution fits a recurring job: a defined template, filled from issuer reports in any language, every season, with sign-off. Seven agents divide it. An intake agent matches reports to issuers; an extraction agent and a second reader read every table, blind to each other; a units agent applies the template's rules; a plausibility agent holds outliers and looks for the issuer's own explanation; an assurance agent proposes the data-quality score; a publishing agent loads only what a steward approved and reads every value back.
ESG Data Extraction's audit trail: every reading, conversion and signature, with the agent's version
It is not a research assistant. Open questions about a company's strategy or controversies belong to analysis, the territory of ESG Issuer Research (ESG analysis of a company, one cited claim at a time). It is not worth deploying for a handful of PDFs a year. It reads charts and scanned tables with lower confidence and sends them to a person rather than guessing. And it does not decide definitions: if a firm has no written rule for fiscal years or boundaries, the agents will flag every case until someone writes one. For the economics of the choice, see buying estimates or reading reports; the solution itself is at ESG Data Extraction.
Questions people ask
Can ChatGPT extract data from a PDF?
A general chatbot can read a PDF and return figures, and for a single document that may be enough. For a template filled from thousands of reports, the missing parts are the page citation for every value, an independent second reading, written unit and boundary rules and a named person who confirms. Gaia's team found that even a carefully prompted model returned a correct value in about four of five checked cases.
Should you use OCR before the LLM?
For scanned reports, text and table layout must be recovered before any model can read them, by OCR or by a model that reads page images. Gaia and ESGReveal both reported that figures in images or infographics were missed. Treat scanned and image-heavy pages as a separate, lower-confidence path that a person reviews.
How do you measure extraction accuracy?
Against values you already know, field by field: count correct values, wrong values and correct "not found" answers separately, by field and by language. Gaia's comparison with commercial data counted a match within 2% after aligning the order of magnitude; for a published statement, also count scale errors, because a value right in its digits but wrong by a thousand is still wrong.
Is a confidence score enough to skip human review?
No. Gaia's report describes a model concluding with high certainty from incomplete pages. Confidence is useful for sorting work; agreement between two independent readers, plus plausibility checks against last year and peers, is a better reason to pre-confirm, and a person still confirms the issuer.
Can an LLM extract data from reports in Korean or Japanese?
Gaia's page search works across languages, and models read many scripts. The risk is in headers and units: a Korean header in thousands of tonnes, or a Japanese table in t-CO2, changes every figure under it. Keep the original table beside the English so a reader of the language can confirm it.