Financial statement analysis with large language models, tested

Financial statement analysis with large language models is a real and useful practice, but the strongest public claim made for it no longer stands. The working paper of that exact title, by Alex Kim, Maximilian Muhn and Valeri Nikolaev, which drew wide attention in 2024, was withdrawn by its authors on February 20, 2025, after a co-author found inconsistencies while trying to replicate its analyses; a second paper by the same authors on summarising disclosures was withdrawn in October 2025. Neither can be cited for its findings. The evidence that remains is less dramatic and more useful: on questions about real filings, models fail often unless they are given the right pages, and in the largest peer-reviewed study of AI against analysts the best forecasts came from combining the two. Botminds' Equity Research Automation is designed around that evidence: a model reads each figure, a second model re-reads it, every value is tied to its page, and the analyst decides.

Research results on this page are linked where they appear, with their status: peer reviewed, working paper, or not peer reviewed. Product examples are fictional sample data from the solution.

"A co-author identified inconsistencies in the data and analyses while attempting to replicate past analyses from the working paper. Accordingly, we have temporarily withdrawn the working paper from circulation while we review the research findings."

Withdrawal notice for arXiv 2407.17866, 'Financial Statement Analysis with Large Language Models', version 3, February 20, 2025

Can ChatGPT analyze financial statements?

It can read them, extract figures and explain them, and it is often right; the problem is the cases where it is wrong and sounds the same. A model asked about a filing it has not been given will answer from what it absorbed in training, which may be stale or invented. A model given the filing still has to find the right page, read the right column and keep the units straight. FINRA's 2026 oversight report defines the failure that matters: "Hallucinations refer to instances where the model generates information that is inaccurate or misleading, yet is presented as factual information" (FINRA, 2026 Annual Regulatory Oversight Report, December 9, 2025).

So the useful question is not whether a model can analyze financial statements, but under what conditions its output can be trusted without re-doing the work. The rest of this page goes through what the research says about that, starting with the paper most people have heard of.

Was the financial statement analysis with large language models paper withdrawn?

Yes. The working paper "Financial Statement Analysis with Large Language Models" (arXiv 2407.17866) was first posted on July 25, 2024, revised on November 10, 2024, and withdrawn by Valeri Nikolaev on February 20, 2025. The notice, quoted above, gives the reason: inconsistencies in the data and analyses that a co-author found while trying to replicate them (arXiv 2407.17866, version 3). The withdrawal is described as temporary, pending the authors' review; on October 27, 2026 the arXiv record still shows it withdrawn.

The withdrawal is not a footnote. The paper's results were widely repeated as evidence that language models could already do an analyst's core job, and they still circulate in presentations and vendor material. A withdrawn working paper carries no findings that can be relied on; until the authors publish a reviewed version, the honest statement is that the claim has no standing evidence behind it. That is how this page treats it.

A second withdrawal: "Bloated Disclosures"

The same authors' earlier working paper, "Bloated Disclosures: Can ChatGPT Help Investors Process Information?", went through four versions from June 2023 to March 2025 and was withdrawn on October 9, 2025. Its notice is blunter: "A co-author attempted to independently replicate key results from the working paper using the described approach, but the analyses did not yield results supporting the reported findings. Accordingly, we have withdrawn the working paper from circulation" (arXiv 2306.10224, version 5).

Two lessons follow for anyone evaluating AI for research. First, working papers are drafts; the field moves fast, and a widely shared draft can be wrong. Second, the authors' own replication caught the problems. That is the standard to hold any claim to, including a vendor's: can the result be reproduced, on data you control, by someone other than the person who made it?

How accurate are LLMs on financial questions?

On simple, factual questions about real filings, accuracy in published tests ranged from poor to good depending far more on how the model was given the documents than on which model it was. FinanceBench, a test of "10,231 questions about publicly traded companies", ran 16 model configurations on a sample of 150 cases and had the 2,400 answers reviewed by hand (Islam et al., arXiv 2311.11944, November 2023; not peer reviewed). The questions were "intended to be clear-cut and straightforward to answer to serve as a minimum performance standard".

Two patterns in the table matter more than the ranking. First, retrieval design moved the same model from 19% to 50% to 79%. Second, the failures differed in kind: the authors note that Llama2 with vector stores "had a much higher percentage of incorrect answers" than refusals, while GPT-4-Turbo refused far more often than it answered wrongly. For a research team, a wrong answer that looks right is the costly failure; a refusal at least sends the question to a person.

Is FinanceBench enough to judge a model? Three limits

FinanceBench is an open benchmark of questions about US public companies' filings, with answers and evidence pages, built to test whether language models can answer financial questions from documents. Its authors concluded that "all models examined exhibit weaknesses, such as hallucinations, that limit their suitability for use by enterprises". It is the most useful public test of the question this page asks, and it has three limits a reader should keep in mind; how the benchmark was built and graded is covered in FinanceBench and the defensible answer.

It tests 2023 models, and models have changed since; a benchmark result is a dated observation, not a permanent property. It tests question answering, not the full task of updating a model and writing a note. And it tests models on their own, without the checks a production system can add: a second reader, a tie to the page, a hold when a figure does not reconcile. A tool's accuracy on your work depends on all three of those, which is why it has to be measured on your own releases; the pilot guide shows one way.

Does AI outperform analysts? The peer-reviewed evidence

On some tasks a purpose-built AI system beats most analysts, and analysts using AI do better still; that is the finding of the main peer-reviewed study, which is not withdrawn. Cao, Jiang, Wang and Yang report that "an AI analyst we build to digest corporate financial information, qualitative disclosure and macroeconomic indicators is able to beat the majority of human analysts in stock price forecasts and generate excess returns compared to following human analyst" (NBER Working Paper 28800, 2021; published in the Journal of Financial Economics 160, 2024).

Note what the study built and tested. It is not a general chatbot reading statements; it is a model trained for a specific forecasting task on structured and textual data, and its advantage was largest where information was voluminous and transparent. The authors also found the AI's lead over analysts shrank when analysts gained access to alternative data and their own AI tools, and their conclusion is a combination: the strongest forecasts came from pairing the machine's processing with the analyst's understanding of soft information. The guide to the equity research process shows how that division of labour looks task by task.

Can ChatGPT predict stock returns?

Research suggests language models can read the direction of news, but that the trading value of doing so falls as more investors do it. Lopez-Lira and Tang tested GPT-4 on news headlines published after its training cut-off and report that "GPT-4 scores also significantly predict the subsequent drift, especially for small stocks and negative news", and that the returns to trading on those scores fell as more investors adopted language models, which they read as prices becoming more efficient (arXiv 2304.07619, version 6, October 28, 2025; not peer reviewed).

Headlines are not financial statements, and reading the sign of a headline is not the same as analyzing a company. The result is a reminder that whatever edge comes from reading faster is competed away as the reading becomes common. What does not compete away is getting the figures right and making a defensible judgment about what they mean, which is the work an analyst signs.

Five tests before trusting a model with a filing

The evidence points to a short list of conditions under which a model's reading of a filing can be relied on. Each is something a research team can check for itself, on its own documents, before and after a tool goes live, as CFA Institute Standard V(A) asks members to test the output of the models and analytical tools they use (CFA Institute Standard V(A)).

Two readers and a tie: how Equity Research Automation uses models

In the solution, no figure reaches a note on one model's reading. The Extractor reads the release, deck and filing in the original language and writes each actual into the analyst's own model with its page and a confidence; values below 97% confidence go to the research associate. The Tie-out Reviewer then re-reads every value with a different AI model, re-adds segments and subtotals, compares history with the model and looks for recast tables. A tie it cannot explain holds the note, and the agent's written limit is that it "never fixes a value by guessing".

On the sample morning that sequence did its job. The Extractor wrote 64 values; the Tie-out Reviewer found three that did not tie, held the note, found the recast on page 6, rebased 24 history cells with the page cited, and asked the analyst to confirm. Only then did the Variance agent compute the beat. Across the sample season, 37 failed ties were caught, and no note was sent with an untied figure.

In Equity Research Automation one model writes, a second re-reads, and a failed tie holds the note

The design does not depend on any model being right. It depends on two readings agreeing, on every figure reconciling, and on a person deciding when they do not. That is the narrower claim the evidence supports, and it is enough to take the keying and checking off an analyst's morning. To see it run on your own releases, start from Equity Research Automation.

Questions people ask

Can LLMs beat financial analysts?

At some forecasting tasks, a purpose-built AI system beat most analysts in Cao, Jiang, Wang and Yang's peer-reviewed study, and analysts with AI did better still. The most cited general claim that a chatbot beats analysts at financial statement analysis came from a working paper that its authors withdrew in February 2025.

Why was the Kim, Muhn and Nikolaev paper withdrawn?

The withdrawal notice says a co-author found inconsistencies in the data and analyses while trying to replicate past analyses, and that the authors withdrew the paper while they review the findings. Their earlier "Bloated Disclosures" paper was withdrawn in October 2025 after an independent replication did not support its results.

Is FinanceBench peer reviewed?

The FinanceBench paper cited here is an arXiv preprint from November 2023 and is not peer reviewed; its dataset is open. Its results describe 2023 models, so they show how retrieval design affects accuracy rather than how today's best models perform.

Which LLM is best for financial statement analysis?

The evidence says the setup matters more than the model. In FinanceBench the same GPT-4-Turbo answered 19% of questions correctly with one shared document store and 79% with the filing in its context window. Test any model on your own filings, with page references and a second check.

Can AI read financial statements in other languages?

Yes, and the same checks apply with more force, because a translated figure must still match the original digit for digit. In the solution, releases are read in the original language, shown beside the English, and each English figure is checked against the original and cited to its page.

Should a research team cite a withdrawn paper?

Not for its findings. A withdrawn working paper has no standing results; at most, a team can note that it was withdrawn and why, which is itself useful when the paper still circulates in vendor material or presentations.