ChatGPT for credit analysis: what a chatbot gets wrong on filings

You can use ChatGPT for credit analysis as a reader of documents you paste into it and a drafter of text you then check; you cannot use it as the source of a credit view. In the best-known test of chatbots on company filings, a 2023 GPT-4 model answering from its own knowledge got 9% of questions right, and even when the right filing was placed in front of it, about one answer in five was still wrong or missing. Credit work adds harder problems: definitions that run for pages, amendments that change the numbers, and private information that must not leave the firm. Botminds' AI Credit Analysis was built for exactly those problems, so this guide sets the chat window and the cited agents side by side on one question.

The side-by-side uses a fictional issuer, Halvorsen Packaging, and a fictional fund group, Quillfield, as they run through the solution: sample data on fictional people and companies. The test results come from published studies and regulators, linked where they appear.

Can I use ChatGPT for credit analysis?

Yes, for the parts where you can check every output against a document you already hold: summarising a public 10-Q you pasted in, drafting the structure of a memo, explaining a term to a new analyst. No, for anything a portfolio manager will size on, unless every sentence is traced back to its page by someone or something other than the model that wrote it.

The reason is not that the models are weak writers. It is that credit questions are open-book questions with long, cross-referenced books, and the answer depends on finding the right clause in the right version. A chat window does not know which amendment is current, does not refuse a sentence it cannot source, and does not know what your firm's information barrier allows you to read.

What a chat window does with a priming question, and what cited agents do

Take one sample question: can Halvorsen Packaging put new debt ahead of the 7.625% notes due 2029 that three Quillfield funds hold? In a chat window the analyst gets an answer shaped by whatever she pasted. In AI Credit Analysis the same question becomes a plan of four sub-questions over 9 documents, read by separate agents, checked sentence by sentence, and pinned to the capital structure.

Is ChatGPT accurate for financial analysis?

It is accurate roughly in proportion to how precisely the right pages are put in front of it. The FinanceBench authors tested models on 150 questions about public filings and reviewed 2,400 answers by hand. A 2023 GPT-4-Turbo working with no documents was right 9% of the time. Given a retrieval system over all the filings it was right 19% of the time; with a store per filing, 50%; with the whole filing in its context, 79%; and handed the exact evidence pages, 85% (Islam et al., FinanceBench, arXiv 2311.11944).

Two lessons follow for credit. First, a chatbot asked from memory is guessing; the 9% row is the "what do you know about Halvorsen's covenants?" prompt. Second, even the 85% row leaves a 15% error rate on questions the authors built to be simple and unambiguous, and credit questions are neither. The authors also judged that feeding whole filings into the prompt is impractical for firms, because it is slow and the largest financial documents do not fit.

The study is a preprint about 2023 models, and newer models score higher. One widely shared claim that a language model out-analysed human analysts on financial statements is no longer a result at all: its authors withdrew the working paper in February 2025, saying a co-author had found inconsistencies in its data and analyses (arXiv 2407.17866, withdrawn; what the claim did and did not show is examined in financial statement analysis with large language models, tested). The honest position is to test on your own documents, as a two-week pilot on your own notes sets out.

Can ChatGPT read a credit agreement?

It can read the text you give it, but a credit agreement is built to defeat a casual reading. Ivashina and Vallée report that the main text of one 2017 credit agreement ran to 170 pages and that "the definition of EBITDA alone takes 1,733 words" (NBER Working Paper 27316). The capacity questions an analyst asks depend on that definition, on the covenant, on its carve-outs and on every amendment since closing.

Length is a problem in its own right. Researchers testing long-context models found that accuracy peaks when the needed fact is near the start or the end of the input and drops when it is buried mid-document, a pattern that held even for models sold on long context (Liu et al., 'Lost in the Middle', Transactions of the ACL; what that means for answers that must be defended is covered in FinanceBench and the defensible answer). In a 318-page agreement, almost everything is in the middle.

The amendments make it worse. In the sample, the house's own June 12 note put Halvorsen's secured capacity under $150 million, though Amendment No. 2 had raised the free-and-clear basket from $250 million to the greater of $350 million and 57% of EBITDA on March 3. A model given the original agreement and asked the same question would have repeated the house's error with confidence. Questions about lender protections specifically are covered in AI agents for credit agreement review.

Can ChatGPT read a 10-K?

It can summarise a 10-K you upload, and that is the safest use of a chatbot in credit work, because the document is public and the output can be checked. The traps are tables and versions: a debt footnote spread across pages, a figure restated in a later filing, an 8-K that changed a credit agreement after the 10-K was filed. FinanceBench's own questions came from public-company filings, and the error rates above are for that material.

For credit, the 10-K is also rarely enough. The Halvorsen answer needed a Norwegian subsidiary's annual report, read in Norwegian, to show that the company holding the group's patents was not a guarantor. A chatbot may translate the page, but it will not show the original beside the English, flag a translation below a confidence line, or tie the sentence to page 47.

A cited page, original beside the English

Why does ChatGPT make up numbers?

Because generating plausible text is what it was built to do, and a plausible number is a different thing from a true one. NIST's generative AI profile calls the risk confabulation, confident output that is false, and notes that it matters most for open-ended questions that need long answers and in fields that depend on context and specialist expertise (NIST AI 600-1, July 2024). A covenant capacity figure is both.

Credit adds a second mechanism: the user's premise. In the legal study above, the authors found that models frequently go along with a user's mistaken legal premise instead of correcting it (Dahl et al., Journal of Legal Analysis). Ask "given the $250 million basket, how much can they add?" after the basket has been amended, and the arithmetic will be faultless and wrong.

Is it allowed to put MNPI into ChatGPT?

That is your compliance department's call, and in most credit funds the answer for a consumer chatbot is no. The legal duty is the adviser's: Section 204A of the Investment Advisers Act requires written policies "reasonably designed" to prevent misuse of material nonpublic information (15 U.S.C. 80b-4a), and lender-site material often arrives under confidentiality terms of its own. Pasting a compliance certificate into a tool outside the firm's controls is hard to square with either.

The statement "extends to situations involving the use by firm staff of third-party AI technologies (such as Chat GPT, Google Bard, and others) with or without the direct knowledge and approval of senior management."

ESMA, Public Statement on AI and investment services, 30 May 2024, paragraph 4

European supervisors have said so explicitly, and FINRA lists the same risk for agents: one working on sensitive data may store, disclose or misuse confidential or proprietary information without meaning to (FINRA, 2026 Annual Regulatory Oversight Report). The working rule is simple: if you could not email the document to an outside party, do not paste it into an outside chatbot. The rules behind the wall are in private side vs public side.

The wall applied before the readers start

ChatGPT prompts for credit analysis that cut the error rate

If you use a general chatbot on public documents, the prompts that help are the ones that move it from the 9% row toward the 85% row: give it the pages, make it show its sources, and check what it says against them. None of them makes it a source.

What is the best AI for credit analysis?

The best AI for credit analysis is the one that is right on your documents and can prove it sentence by sentence, which no leaderboard measures. Look for four things: it reads your agreements as amended rather than a vendor's summary, it refuses sentences it cannot cite, it applies your information barrier before it reads, and it keeps the question, the passages and the decisions where an examiner can see them. How AI Credit Analysis enforces the second of those is shown in the citation checker that blocks uncited sentences.

Where a chatbot fits is the edge of the work: learning, drafting, summarising public text you will check anyway. Where it does not fit is the centre: the number a portfolio manager sizes on and the sentence a senior analyst signs. That centre is what AI Credit Analysis is built for.

Questions people ask

Can ChatGPT do a full credit analysis of a company?

It can draft one from documents you supply, but it will not know which amendments apply, will not recompute its own figures and will not refuse unsupported sentences. In FinanceBench, even handed the exact evidence pages, the best 2023 configuration got 15% of straightforward filing questions wrong.

Is ChatGPT good at reading financial statements?

It is better at reading statements you give it than at recalling them: 79% correct with the whole filing in its context against 9% from memory, in the 2023 FinanceBench test. Tables split across pages and later restatements remain common sources of error.

Can I upload a credit agreement PDF to ChatGPT?

Technically yes; whether you may is a compliance question. A credit agreement that is material to an SEC-reporting issuer is filed publicly as an exhibit under Item 601(b)(10) of Regulation S-K; lender-site versions, amendments in negotiation and compliance certificates are usually confidential and belong on the private side of the wall.

Why does ChatGPT give a different answer each time?

Generation is probabilistic, so the same prompt can produce different wording and sometimes different numbers. That is one reason FINRA's 2026 report points firms to keeping prompt and output logs and tracking which model version was used.

Was there a study showing AI beats analysts at financial statement analysis?

The best-known one, by Kim, Muhn and Nikolaev, was withdrawn by its authors in February 2025 after a co-author found inconsistencies in the data and analyses. Its findings should not be relied on while the authors review them.