Evaluate · chat answer or cited answer

Generative AI in financial reporting: why chat misreads instructions

Generative AI in financial reporting is good at reading and drafting, and unreliable at saying which rule applies on which date. A chat answer to a Call Report question comes from memory, with no page, no revision date and no bank precedent; Call Report Instructions Search answers only from passages it can quote, and routes the rest.

Updated · 14 min read

Generative AI in financial reporting is already used to summarize filings, draft explanations and answer staff questions about reporting rules. On the Call Report, the last use is where general-purpose chat assistants fail in a specific, predictable way: they answer from what the model absorbed in training, so they cannot show the instruction sentence that decides a line, cannot tell which revision of a page was in force on the report date, and know nothing of the bank's own approved positions. The FFIEC instructions changed in three of the last five quarters, and a statute changed the reporting of reciprocal and custodial deposits on July 11, 2026; a model trained before that date cannot know it. Research on large language models points the same way: they read numbers well and link them to the right category in a large taxonomy much less reliably. Botminds' Call Report Instructions Search is a different design: agents that answer only from passages they can quote, in the instruction books for the form and quarter asked, check each page's revision date against the bank's memo, show what the bank filed before, and send what they cannot cite to a person.

The same Call Report question, answered two ways

A chat assistant answering from memoryCall Report Instructions Search
The question

A chat assistant answering from memory: Where do $240 million of equipment leases to counties and school districts go on the FFIEC 031?

Call Report Instructions Search: The same question, on the FFIEC 031, for September 30, 2026.

Where the answer comes from

A chat assistant answering from memory: Patterns in training text, which may include older instructions, forum posts and other banks' practices.

Call Report Instructions Search: Passages in the quarter's instruction books, Glossary and supplemental instructions, quoted exactly.

The line

A chat assistant answering from memory: Plausibly item 8, because the lessee is a municipality, or item 10, without a way to tell which is right.

Call Report Instructions Search: Item 10.b, All other leases; item 8 struck through, because it covers obligations "other than leases".

The date of the rule

A chat assistant answering from memory: Unknown; the model cannot say which revision of the page it learned.

Call Report Instructions Search: RC-C-20a (12-24), checked against the bank's memo and the 2026 supplemental instructions.

The bank's precedent

A chat assistant answering from memory: None; the model has never seen the bank's memos or filings.

Call Report Instructions Search: The approved memo and the amounts filed in item 10.b for three quarters.

When the sources do not settle it

A chat assistant answering from memory: Still answers, in the same confident tone.

Call Report Instructions Search: Says so, and routes the question to the Policy Lead with a clock.

Left column: the failure modes of an assistant that answers from model memory without retrieving the instructions, as described in the research cited on this page; not a transcript of any product. Right column: Call Report Instructions Search on sample data about a fictional bank.

How is generative AI used in financial reporting?

In three broad ways: reading documents, drafting text and answering questions about rules. For a bank's regulatory reporting team, reading means extracting figures from schedules and statements; drafting means first versions of edit explanations, memos and management commentary; answering means telling a preparer where a balance goes. The first two produce something a person then reviews line by line. The third is riskier, because the answer is often acted on directly by someone who asked precisely because they did not know. How agents draft and check Call Report explanations is covered in Call Report Analysis's guide to AI in regulatory reporting.

Supervisors use AI on the same reports, with a person deciding. The Government Accountability Office reported in May 2025 that the NCUA uses AI to predict values of items in credit unions' Call Reports and flag those outside the predicted range, and that officials of the regulators using AI said it did not decide anything by itself and was never their only source (GAO-25-107197). The agencies also asked banks, in their December 2025 request for information, about "current or emerging uses of technologies" they should consider for the Call Report forms and instructions (Request for Information: Streamlining the Call Report).

Can ChatGPT answer Call Report questions?

It can produce an answer; it cannot show you that the answer is right for your report date. A general chat assistant without access to the current instruction books answers from training data, so its answer has no page number, no revision date and no quotation that a reviewer can check against the FFIEC text. Even when it names the right line, the reviewer has to do the research again to know that, which removes most of the benefit.

The timing problem is concrete. The September 2026 Call Report changed how reciprocal deposits are reported under the 21st Century ROAD to Housing Act, effective July 11, 2026, and gave eligible institutions a choice, for September 30, 2026 only, between estimating custodial deposits under the new law "on a best-efforts basis" and continuing to report them as brokered under the old instructions (Supplemental Instructions, September 2026). It also lowered the community bank leverage ratio requirement to 8 percent and changed the weekend due-date rule (FIL-62-2026). An assistant whose training ended before July 2026 will answer those questions under the old rules, fluently.

Is generative AI accurate for accounting research?

Accurate at reading and extraction, much less so at classification against a large rulebook, and poorly calibrated about its own errors, on the research available so far. FinTagging, a benchmark of models tagging financial statement facts to the US GAAP taxonomy, reports that mapping facts to over 10,000 concepts "remains costly and error prone" and that models which extract well struggle when they must link each fact to the right concept (Wang et al., FinTagging, arXiv preprint, 2025–2026). That is the shape of a Call Report placement question: the amount is easy, the line is hard.

Calibration is the second problem. FinVerBench, built from SEC 10-K XBRL filings for 43 S&P 500 companies, asked models whether financial statements were internally consistent; under its original prompt, "nine of fourteen complete LLM runs produce 95-100% false positives on clean statements, while one run achieves 0% observed false positives" (Panda, FinVerBench, arXiv preprint, 2026). Neither study is peer reviewed, and neither tested Call Report instructions. Both support the same design conclusion: a model's output on a reporting question needs its source shown and a person's sign-off, not trust.

How often language models flagged clean financial statements as wrong

95–100% false positives on clean statements
9 of 14 runs
Between 0% and 95%
4 of 14 runs
0% observed false positives
1 of 14 runs
FinVerBench (arXiv 2605.29586, May 2026, preprint, not peer reviewed): fourteen complete model runs on 43 clean statements from SEC 10-K XBRL filings, original guided-checklist prompt. The middle bar is our arithmetic (14 − 9 − 1).

What are the risks of generative AI in regulatory reporting?

The main risk is a wrong line that looks right, because a misplaced amount is a reporting error with consequences. The General Instructions let a bank's supervisor require an amended report when a significant amount sits on the wrong line (General Instructions, page 10a). Each failure mode of a chat answer maps to a control that a reporting team already understands.

Five ways a generated Call Report answer goes wrong, and the control for each

FailureWhat it looks likeControl that catches it
Answer from memoryA fluent line number with no sourceNo citation, no answer: every layer cites a passage
Stale ruleCorrect under last year's page, wrong todayThe cited page's revision date checked against the report date and the bank's memo
Invented or altered quotationA sentence the instructions never containedEvery quoted sentence matched to its source; mismatches removed
Ignoring the bank's positionA different line from the one the bank has filed for yearsThe approved memo and filed values shown beside the instruction
False confidenceAn answer where the books are silentA threshold below which the question goes to a person
Our table. The controls are the answer rules of Call Report Instructions Search on sample data about a fictional bank.

Can AI cite the Call Report instructions?

Yes, if it answers from the instruction text itself rather than from memory, and if something checks the citation. Call Report Instructions Search loads each quarter's FFIEC 031 and 041 and FR Y-9C instructions, Glossary, supplemental instructions and the bank's approved memos as one corpus version, approved by the Regulatory Reporting Manager. The Answer agent finds the passages that decide a question; a citation check then matches every quoted sentence to its source, removes any that do not match, and blocks an answer with an uncited layer. On the sample question about leases to municipalities, the agents searched 1,410 pages, found nine passages, and the citation check matched seven of seven quoted sentences.

The agents' plan as they work; nothing is shown until every layer is cited

An answer in progress for 'Where do we report the $240M of leases to municipalities?' A line says the agents are working and nothing is shown until every layer is cited. Steps with timings: the Answer agent read the question and the form, FFIEC 031 and FR Y-9C, Q3 2026, in 0.9 seconds, then searched 1,410 pages and found 9 passages, RC-C items 10, 10.b and 8, in 6.2 seconds; the Change agent found RC-C-20a revised in December 2024, before the bank's memo, and no mention of item 10 in the March, June and September 2026 supplemental instructions; the Precedent agent found the memo approved December 15, 2025 and filings in item 10.b in three quarters; the Mapping agent found general ledger 14520 and 14525 mapped to RC-C 10.b and HC-C 10.b with a September 30 balance of $240.6M; the citation check, 7 of 7 quoted sentences match their sources.
Call Report Instructions Search on sample data about a fictional bank.

The citation opens the page itself. The reviewer sees RC-C-20a (12-24), revised December 2024, with "Include all leases to states and political subdivisions in the U.S. in this item" highlighted, which is the sentence the FFIEC printed in the item 10 instructions (Schedule RC-C instructions). Checking the answer takes as long as reading two sentences.

Every citation opens the instruction page, quoted exactly, with its date

The instruction page behind a citation: Items 10, 10.a and 10.b, Lease financing receivables, from the FFIEC 031 and 041 instructions, Schedule RC-C Part I, RC-C-20a (12-24), revised December 2024, marked 'Quoted exactly, Citation check'. The sentence 'Include all leases to states and political subdivisions in the U.S. in this item.' is highlighted, and the first sentence of item 10.b, All other leases. A side panel shows the document, page, revision date, corpus version 2026-Q3 v2 and the answer the page supports, with seven related passages.
Call Report Instructions Search on sample data about a fictional bank; the page text is the FFIEC's.

Do regulators allow generative AI for reporting?

No rule forbids it, and none excuses it: the bank answers for the figures however they were produced. The General Instructions keep that responsibility inside the bank and outside any vendor's reach. Federal Reserve Governor Michelle W. Bowman put the supervisory stance plainly in November 2024: AI gets no exemption from the law and rules that already apply, and its use gets no exemption from scrutiny (Bowman, November 22, 2024; quoted in how to use AI for regulatory compliance on wires). The Treasury Department's December 2024 report recommended that firms review AI use cases for compliance with existing laws and regulations before deployment (U.S. Treasury, AI in Financial Services).

Is generative AI covered by model risk guidance?

Not by the current interagency guidance, which leaves it out on purpose. The model risk management guidance the agencies issued in April 2026, which replaced SR 11-7, states that generative and agentic AI models are novel and rapidly evolving and are not within its scope; it also excludes simple spreadsheet arithmetic and "deterministic rule-based processes and software" with no statistical, economic or financial theory behind them, and says it is expected to be most relevant to organizations with over $30 billion in assets (SR 26-2). For a Call Report answer tool, that means two things. The deterministic parts, such as matching a quotation to its source or comparing two dates, are ordinary software controls. The generative part, choosing and phrasing the answer, falls to the bank's own AI governance until the agencies say more, which is one more reason to keep a person deciding whatever the sources do not settle.

How do you test an AI answer on the Call Report?

With your own past questions, including the hard ones, and by checking every citation rather than the tone. The test that matters is not how often the tool sounds right but how often it is right with a source, how often it says "I cannot cite this," and whether those two groups are the right ones. Call Report Instructions Search approves each quarter's corpus only after 120 golden questions with known answers are re-run; a bank evaluating any tool can build the same kind of set from its own policy lead's inbox.

A test set for any tool that answers Call Report questions

  1. Last quarter's real questions, with the answers the bank gave

    The baseline: same line, with a citation a reviewer accepts.

  2. Questions where the bank's memo and the instruction disagree

    The tool should say so, not side with either.

  3. Questions the books do not answer

    New products and instruments. The right result is a routed question, with what was searched.

  4. Questions about a page revised this quarter

    The answer should cite the new page and flag memos that relied on the old one.

  5. Questions on the FR Y-9C as well as the Call Report

    The parallel line in Schedule HC-C or HC-R, from the FR Y-9C book.

  6. Every quotation checked against the PDF

    One altered sentence is a failure, however right the line.

Our checklist. The golden-question approval is how Call Report Instructions Search loads each quarter on its sample configuration.

One cited answer, on the solution's sample data

34sec

from the question to an answer cited in all three layers

7

sources behind the answer; 7 of 7 quoted sentences matched by the citation check

0.97

confidence; the bank's threshold for answering without a person is 0.80

0.55

the score a passage needs before the agents will rely on it at all

Call Report Instructions Search on sample data about a fictional bank: the municipal leases question, October 22, 2026, and the answer rules then in force. The design target is a median cited answer in under one minute.

Where generative AI belongs in an instructions answer

In reading the question and writing the answer, not in deciding what the rule is. A language model is well suited to understanding a preparer's wording ("leases to municipalities") and to phrasing a verdict line in plain words. Which passages decide the question is a search over the current corpus; whether a quotation matches its source and whether a page postdates a memo are deterministic checks; whether a disagreement is resolved is a person's ruling. When none of the passages scores high enough, the answer is "No citation, no answer," and the question goes to the Policy Lead with what was searched.

When nothing in the instructions decides it, no line is suggested

An answer screen for 'How do we report tokenized deposits we hold for a fintech partner?' headed No citation, no answer. The text says the agents searched the whole Q3 2026 corpus and no passage addresses the question closely enough to quote; the nearest, the deposit definitions for Schedule RC-E, scored 0.41, below the 0.55 an answer needs, so a guess would be a misreported line and the question goes to the Policy Lead. Counters show 1,410 pages searched, 180 memos and 0 passages above 0.55, and the question has been routed with a due time.
Call Report Instructions Search on sample data about a fictional bank.

That division of labour is also what makes the product easy to evaluate. Every answer can be checked against a page, every routed question can be checked against what was searched, and every ruling has a person's name and a reason code on it. The rule for routing rather than answering and how a stale position memo is caught are covered in their own guides.

Questions people ask

Can generative AI prepare the Call Report?

It can help with parts of the work, such as drafting explanations or answering placement questions, but the figures come from the bank's general ledger and reporting system, and the CFO and directors attest to them. The agencies' instructions keep responsibility for accuracy with the bank.

Why not just paste the instruction PDF into a chatbot?

That helps with the page you pasted, but a Call Report question often turns on two or three passages in different places, the Glossary and the quarter's supplemental instructions, and on the bank's own memo. It also leaves no check that each quoted sentence is really in the source, and nothing kept for the reviewer.

Does the model risk guidance apply to an AI answer tool?

The April 2026 interagency guidance says generative and agentic AI models are not within its scope, and it excludes deterministic rule-based software. Banks therefore govern such tools under their own AI and third-party risk policies, and a person should decide anything the sources do not settle.

How fresh does an AI tool's knowledge of the Call Report need to be?

Current to the report date being filed. In September 2026 the instructions changed for reciprocal and custodial deposits, the community bank leverage ratio and the weekend due-date rule, so any answer resting on earlier text could be wrong for September 30, 2026.

What is the difference between AI search and a chatbot for reporting rules?

A search-based answer retrieves passages from a known, dated set of documents and quotes them; a chatbot without retrieval writes from what it learned in training. Only the first can show the page and revision date a reviewer needs.

Call Report Instructions Search.
See it on your own files.

An agentic solution from Botminds. Ask where something goes on the Call Report or the FR Y-9C; get the line, the instruction that decides it, what changed since, and how your bank reported it before, every layer cited Book a demo, or explore how it works.