FinanceBench is an open benchmark for testing whether large language models can answer questions about public companies from their filings: 10,231 questions drawn from 361 10-Ks, 10-Qs, 8-Ks and earnings reports of 40 US companies, each with an answer, the evidence passage and its page. Its 2023 results showed that the same model could answer correctly a fifth of the time or four-fifths of the time depending only on how the filing reached it, and that many wrong answers looked as well reasoned as right ones. For an investment research team the lesson is not a leaderboard position. It is that an AI answer is only defensible when each sentence carries a passage that someone, or something independent of the writer, has re-read. That is the standard Botminds' Investment Research Assistant applies to every answer it gives across a firm's notes, licensed broker research, filings and expert calls.
What FinanceBench is made of
10,231
questions about 40 US public companies, each with an answer, an evidence passage and a page
361
filings and earnings reports, 2015 to 2023, that the questions are drawn from
2,400
model answers checked by hand: 150 sample questions across 16 model setups
66%
of the labelled questions need numerical reasoning, not just finding a figure
What is FinanceBench? A floor test built from 361 filings
FinanceBench is a question-and-answer test set for financial documents, designed by its authors as a minimum standard rather than a hard exam. It has three kinds of question. Twenty-five "domain-relevant" questions are asked of most companies, such as whether a dividend was paid or whether operating margins were consistent. Annotators wrote 1,323 company-specific questions meant to need some reasoning. And 7,983 questions were generated from 18 base metrics taken from the three main statements, from which ratios such as net income margin are derived.
Its most useful design choice for a research team is the evidence. Each entry carries "an evidence string (which contains the information needed to verify that the answer to the question is correct) and a page number from the relevant document" (Islam et al., 2023). That is the shape of a checkable answer: a claim, the words it rests on, and where those words sit. The authors also labelled 8,908 questions by type: 28% are pure extraction, 66% involve numerical reasoning and 6% logical reasoning. Two-thirds of the test is arithmetic on figures the model first has to find.
The dataset was built by a team of 20 annotators with finance backgrounds, and the authors note that five left early "due to quality issues" and their work was discarded. Even a benchmark of simple questions took supervised experts and spot checks to get right, which is worth remembering when a team builds its own.
How accurate are LLMs on financial questions? It depends on the setup
On FinanceBench, accuracy depended more on how the filing reached the model than on which model it was. The authors ran 2023 models in five setups: with no documents at all, with one search index over every filing, with one index per filing, with the whole filing pasted into a long context window, and an "oracle" setup handed the exact evidence pages. They then checked every answer by hand and sorted it as correct, incorrect, or a failure to answer.
FinanceBench: how each setup's 150 answers split between right, wrong and no answer
GPT-4-Turbo, no documents
- Correct · 9%
- Wrong · 3%
- No answer · 88%
Llama2, one index for all filings
- Correct · 19%
- Wrong · 70%
- No answer · 11%
GPT-4-Turbo, one index for all filings
- Correct · 19%
- Wrong · 13%
- No answer · 68%
GPT-4-Turbo, one index per filing
- Correct · 50%
- Wrong · 11%
- No answer · 39%
GPT-4-Turbo, whole filing in context
- Correct · 79%
- Wrong · 17%
- No answer · 4%
GPT-4-Turbo, given the evidence pages
- Correct · 85%
- Wrong · 15%
The realistic retrieval setting, one shared index over every filing, is where most enterprise search tools sit, and it was the weakest of the document setups: 19% correct for both models tested. Indexing each filing separately helped because the search had fewer places to go wrong. Pasting the whole filing into the context window did best among the practical setups, but the authors note that long context is slow and costly and cannot hold the largest documents, some of which run to 250 pages and had to be cut. A research question that spans twenty filings and a broker's model is not a problem a context window solves.
These are 2023 models, and the scores have dated. The structural findings have not: retrieval decides what the model gets to read, and the model cannot be more right than what it read. A wider review of language models on financial statements, including two widely cited papers since withdrawn by their authors, is in financial statement analysis with large language models, tested.
A wrong answer and a refusal are different failures
For a research team, a model that says "I cannot find this" is a nuisance; a model that confidently gives a wrong figure is a risk. FinanceBench's split shows the two are not the same thing and do not move together. In the shared-index setup, Llama2 was wrong on 70% of questions and silent on 11%; GPT-4-Turbo was silent on 68% and wrong on 13%. The same correct rate hides opposite behaviours, and only one of them reaches a memo unnoticed.
The authors' grading rule is stricter than most internal pilots, and it is the right one. "If a model gives the right answer but with logic or calculations that explicitly contradict the evidence in the gold standard answer, we label it Incorrect" (Islam et al., 2023). A correct number reached by the wrong route is luck, and luck does not repeat on the next filing. Any test a research team runs should grade the reasoning and the source, not only the final figure.
The prompt itself moved results too. In the long-context setup, putting the filing before the question rather than after it lifted GPT-4-Turbo from 25% to 78% correct and Claude2 from 37% to 76%, a gap the authors attribute to models losing track of the question after thousands of tokens of evidence.
Where the question sits in the prompt changed accuracy threefold
Why do LLMs hallucinate on financial documents?
Language models hallucinate on financial documents because they generate the most plausible text, and a plausible figure, unit or period is easy to produce and hard to spot. FinanceBench describes the failure precisely: "superficially coherent and seemingly well-justified answers, sometimes with extensive calculations and reasoning steps, which were still wrong" (Islam et al., 2023). The calculations are the trap. A visible working looks like evidence of care.
NIST's generative AI profile names the same behaviour confabulation and notes that it extends to the justification: "LLMs sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect" (NIST AI 600-1, July 2024). Filings add their own hazards. The same revenue line appears in the income statement, the segment note and an earnings release reconciliation, sometimes on different bases; a model that pulls the wrong one gives a real number from the wrong place.
Position in a long input matters as well. Liu and colleagues found that language models perform best "when relevant information occurs at the beginning or end of the input context", and that performance "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., Transactions of the ACL, 2024). A 200-page annual report has a great deal of middle.
What FinanceBench does not test, and your research desk does
FinanceBench tests one public company at a time, one question at a time, from public filings, and its authors say so. "FinanceBench was designed to answer questions about a single company, rather than to compare figures between two companies," they write, partly because comparing companies "means that models have to handle two separate documents, which is much harder" (Islam et al., 2023). They also left out private documents because they "create issues around commercial sensitivity and privacy".
A buy-side research question usually breaks all three limits at once. It spans several companies, runs over several turns, and leans on the documents FinanceBench excluded: the firm's own notes and broker research licensed to named readers. A good FinanceBench result is therefore a necessary sign for a research tool, never a sufficient one.
What FinanceBench measures, and what a research desk's question adds
FinanceBench: One company per question.
A buy-side research question: Many at once: which of our holdings does a proposed duty hit, ranked.
FinanceBench: Single question and answer.
A buy-side research question: Follow-ups: why does the broker see twice our hit, add another name, show the credit view.
FinanceBench: Public US filings and earnings reports.
A buy-side research question: Internal notes, licensed broker research, filings in their original languages, cleared expert calls, holdings.
FinanceBench: None: everything is public.
A buy-side research question: Each broker report is licensed to named readers; some issuers are restricted; some calls are on hold.
FinanceBench: One gold answer per question.
A buy-side research question: A broker and the firm can both be right under different assumptions; the analyst chooses.
FinanceBench: Correct, incorrect or no answer, checked by hand.
A buy-side research question: Every sentence must trace to a passage the reader may see, and someone signs the memo.
What is a cited answer, and how do you verify an AI answer?
A cited answer is one in which every sentence points to the passage it rests on, and verifying it means re-reading each passage to confirm it says what the sentence says. Citation alone is not verification. A human audit of AI search engines found that about half of their generated sentences were fully supported by the citations attached to them (Liu, Zhang and Liang, Findings of EMNLP 2023): the tools cited fluently and often wrongly.
The fix is structural rather than a better prompt. The step that checks support should be separate from the step that wrote the sentence, should read the passage itself, and should be allowed to delete. FINRA's 2026 oversight report lists, among the practices firms consider, "storing prompt and output logs for accountability and troubleshooting" and "tracking which model version was used and when" (FINRA, 2026 Annual Regulatory Oversight Report). CFA Institute's Standard V(B) adds the analyst's own duty, to keep what a source states apart from the analyst's opinion of it in any research presented (CFA Institute, Standard V(B)).
In Investment Research Assistant that separate step is the Verifier. Take the sample firm's tariff question: the draft answer had 16 sentences. The Verifier re-read every cited passage on its own and struck two. One, a claim that a holding planned a further 3% price rise, had no supporting passage anywhere in the sources. The other said a company had already moved its boot lines to Mexico, when the source said it was qualifying a plant for 2028; it came back rewritten in the source's own terms. Both strikes stay visible under the answer.
What Investment Research Assistant's Verifier struck, and why
The defensible answer: seven things it must prove
A defensible answer is one a portfolio manager, a compliance officer and an examiner could each re-check and reach the same conclusion. The checklist below turns that into tests that can be run on any research tool's output, including a human analyst's.
Seven things a defensible AI research answer must prove
- Every sentence has a passage
Not a document, a passage: the words the claim rests on, on a numbered page.
- The passage says what the sentence says
Checked by a step that did not write the sentence, with the power to delete or send it back.
- Numbers carry their period, unit and basis
FY or calendar, reported or adjusted, thousands or millions, and which statement or note.
- Translations sit beside the original
A figure from a German or Japanese annual report is shown with the original sentence and page.
- Disagreements are shown, not averaged
Where a broker and the firm differ, both quotes appear with the reason, and a named person chooses.
- The reader was allowed to see every source
Licensed text appears only for licensed readers; anything withheld is declared, not silently dropped.
- The whole exchange is kept
Question, plan, passages, deletions, decisions and viewers, retained with the firm's research records.
The fourth check is where many tools are silent. On the sample question, one holding's share of footwear made in Vietnam came from its German annual report: the footnote opens page 47 of 214 with the original sentence beside the English, and the Verifier's match score of 97%. A broker had estimated about 22% two weeks earlier; the answer used the company's newer 34% and said why.
An Investment Research Assistant footnote that opens the German annual report on its page
How to build your own FinanceBench on your research
The most informative test of a research tool is a small FinanceBench built from your own questions, sources and permissions. FinanceBench's authors end with the same advice for analysts: "robustly evaluate their models before using them in high-stakes live settings" and "double check results and triangulate findings by using multiple sources of evidence" (Islam et al., 2023).
A workable version takes thirty to fifty questions your team has already answered, with the memo that answered them as the answer key. Mix the three FinanceBench types, then add the four things it leaves out: a cross-company ranking, a follow-up that changes scope, a question touching a broker your test user is not licensed for, and one about an issuer on the restricted list. Grade every sentence for support, not just the final figure, and count wrong answers and refusals separately. The fuller acceptance test, and how to go live team by team once it passes, is in how to choose the best AI for investment research on your own sources; why a public chatbot fails the licence and restricted-list questions by design is in how to use ChatGPT for investment research.
Investment Research Assistant is built to be graded this way: every sentence footnoted, a Verifier that deletes rather than hedges, withheld sources declared, and every run kept so the test can be repeated after any change to an agent or model. To run it against your own questions, start from Investment Research Assistant.
Questions people ask
Is FinanceBench peer reviewed?
No. The FinanceBench paper is an arXiv preprint from November 2023 by researchers at Patronus AI, Contextual AI and Stanford University, and its 150-case evaluation sample is published openly. Treat its scores as describing 2023 models and its design as the durable contribution.
What models did FinanceBench test?
Four 2023 models from three providers, GPT-4, GPT-4-Turbo, Claude2 and Llama2, in 16 configurations: no documents, a shared search index, an index per filing, a long context window and an evidence-pages reference setup. Experts checked all 2,400 answers by hand.
Does retrieval-augmented generation fix hallucination on filings?
It reduces it and changes its shape, but it does not remove it. In FinanceBench, Llama2 with a shared index answered 70% of questions wrongly. A separate step that re-reads each cited passage and deletes unsupported sentences is what turns retrieval into a defensible answer.
How many questions should an internal AI accuracy test have?
FinanceBench's authors judged 150 questions enough to compare 16 setups when every answer was checked by hand. For a single team's acceptance test, thirty to fifty questions with known answers, graded sentence by sentence, usually show the failure patterns; add trap questions for licences and the restricted list.
What counts as a hallucination in a financial answer?
Any statement not supported by its source: an invented figure, a real figure from the wrong period or statement, an overstated claim, or a citation that does not say what the sentence says. FinanceBench also graded a right number reached by contradictory reasoning as incorrect, a stricter and more useful rule.