An AI hallucination in legal citations is an assertion a model presents as resting on a legal source when the source does not say it — an invented case or clause at one extreme, and at the other, and far more often, a genuine clause correctly cited beside a sentence it does not support. For a loan desk the second kind is the dangerous one, because the citation is what makes a reader stop checking. Published evaluations put numbers on both. Botminds' Loan Document Search is built around the assumption that they apply to it too: every sentence of an answer is read back against the clause it names, and anything the clause does not support is rewritten or removed before a servicing officer sees it. This guide is about how to test that claim, or anybody else's, on your own executed documents.
The measured figures below come from a peer-reviewed study and a preregistered evaluation; the definitions are from the US standards body; the worked example is a fictional sample facility. No measured accuracy figure for our own answers appears on this page, because we have not published one.
What the published evaluations found, and the number nobody has
58%
minimum rate at which general-purpose models hallucinated on legal queries (Journal of Legal Analysis, 2024)
33%
top of the 17% to 33% range at which purpose-built legal research tools hallucinated, in a preregistered evaluation (2024)
Yours
the rate on your own loan documents, for any tool — nobody has measured it for you
What is an AI hallucination in legal citations?
The clearest definition comes from the US standards body, which records commenters’ concern that the word hallucination anthropomorphises the failure, and names the category confabulation instead:
Confabulation: The production of confidently stated but erroneous or false content (known colloquially as "hallucinations" or "fabrications") by which users may be misled or deceived.
NIST AI 600-1, Generative Artificial Intelligence Profile, July 2024
Two words in that sentence do the work. Confidently stated is the mechanism: the same output would be harmless if it arrived hedged. Misled or deceived is the harm, and the same document is explicit about how it happens: the risk arises when users believe false content, often because the response sounds sure of itself, and then act on it or pass it on.
A citation is a confidence amplifier. Put a clause reference beside a sentence and a reader's natural response is to treat the sentence as already checked by somebody. That is reasonable behaviour when citations are reliable and exactly wrong when they are not — which is why the useful taxonomy for a loan desk is not "hallucinated or not" but the four distinct ways a citation can fail.
How often does AI hallucinate legal citations?
Two studies give the orders of magnitude. The first profiled general-purpose models on legal questions and reported that, "using OpenAI's ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users' incorrect legal assumptions" (Dahl and others, Journal of Legal Analysis, 2024). The last clause of that sentence is as important as the number: a model that accepts a false premise in the question will build a well-cited answer on top of it.
The second tested purpose-built legal research tools rather than chatbots, and was preregistered — the design and the scoring were published before the results were known, which rules out choosing the measure that flatters. Its authors report that "we design and report on the first preregistered empirical evaluation of AI-driven legal research tools. We demonstrate that the providers' claims are overstated", and they put the hallucination rate of the purpose-built tools they tested between 17% and 33% (Magesh and others, 2024).
What each evaluation tested, and what its number means
| Evaluation | What was tested | What counted as a failure | Reported rate |
|---|---|---|---|
| Dahl, Magesh, Suzgun and Ho, Journal of Legal Analysis, 2024 | General-purpose public models on questions about US federal case law, across jurisdictions, courts and time periods. | Output inconsistent with legal facts — including accepting a false premise contained in the question. | At least 58% |
| Magesh, Surani, Dahl, Suzgun, Manning and Ho, preregistered evaluation, 2024 | Purpose-built legal research tools built on retrieval over a legal corpus. | A response that is incorrect or misgrounded — a source that does not support the proposition it is cited for. | 17% to 33% |
| Your own loan documents | The questions your desk actually answers, on your executed agreements and amendments. | Whatever you decide before you start, written down first. | Nobody has measured it for you |
Neither figure transfers to a loan book directly, and it would be dishonest to present them as if they did. What transfers is the shape of the finding: retrieval over a real corpus reduces the failure a long way and does not remove it, and the residual failures are precisely the ones that come wearing a citation.
Does retrieval-augmented generation stop it?
Retrieval helps, and the help is real. The US Treasury's survey of AI in financial services records the argument as practitioners make it: "One respondent stated that retrieval augmented generation provides more transparency as to the basis of the responses and therefore enables the user to better assess the credibility of the output" (Treasury, December 2024). Note what is being claimed — transparency, so the user can assess. Not correctness.
The preregistered evaluation tested systems built on exactly that architecture and still found a failure rate between 17% and 33%. Its authors also explain why the vendors’ claims had gone untested for so long: the systems are closed, so nobody outside could check them systematically. Retrieval changes the failure mode rather than removing it. Instead of a fabricated source you get a real one, fetched correctly, attached to a sentence that drifted from it somewhere between retrieval and prose.
For a credit agreement there is a second, structural reason not to treat retrieval as the end of the problem. Retrieval ranks passages by similarity to the question, and the three versions of an amended clause are all highly similar to it. A retriever with no concept of an effective date can return the superseded one, and every downstream step will then do good work on the wrong text — which is why getting the clause is only half of a loan answer.
Four ways a cited loan-document answer goes wrong
A cited answer can still be wrong, and distinguishing the four ways is what makes a test design possible. The same failure on a bank's own credit policy is worked through in the credit policy citations guide; on a loan file the extra failure is the superseded clause. Each fails differently, each is caught by a different check, and only the first is what most people picture when they hear "hallucination".
One tidy answer and one checked answer, on the same question
Tidy, and wrong in two places: "No prepayment premium applies (§2.11(f))." The section is real and is the right one — but the version quoted is the one Amendment No. 2 put in place, under which the window had already closed for a different reason.
Checked, and shorter for it: "No prepayment fee. The 1.00% premium applies only to a Repricing Transaction, and a prepayment from cash on hand and sale proceeds is not one," citing §2.11(f) as amended by Amendment No. 3 §4, effective 22 Jul 2026.
Tidy, and wrong in two places: "Three business days’ notice is required (§2.11(a))." The clause says business days of a particular kind; the sentence drops the qualifier and the reader counts on the wrong calendar.
Checked, and shorter for it: "Notice by 12:00 noon New York time on Tuesday 27 October — three U.S. Government Securities Business Days before the prepayment date," with the counting shown.
Tidy, and wrong in two places: "Breakage of approximately $18,000 will be payable." No clause says that; the figure is an estimate the prose produced because the sentence wanted a number.
Checked, and shorter for it: "Each lender may certify its own funding loss under §3.05" — the sentence the read-back rewrote, because the agreement gives the certification to the lenders.
Tidy, and wrong in two places: Three sentences, three citations, all three references real and openable. Nothing about it looks wrong.
Checked, and shorter for it: Three sentences, each openable at the executed page, one of them visibly less satisfying than the question wanted.
Behind that comparison are four failure types worth naming separately. A fabricated source — a clause or case that does not exist — is now rare in retrieval systems and trivially caught by opening the link. A misgrounded citation is a real source that does not support the claim; this is the one the published evaluations count and the one a reader almost never catches. A superseded source is a real clause that was correct until an amendment replaced it, and is unique to documents with a history. A silent gap is a question the documents do not answer, filled from general knowledge of how such agreements usually work, with the nearest clause cited for colour.
Four tests to run on your own loan documents
A day of structured testing on files you already know beats any vendor-supplied benchmark, because you can mark the answers yourself. The four tests below map one to one onto the four failure types.
The four tests, what each catches, and how to score it
- The clause-support test
Take twenty questions your desk has already answered. For each sentence of the tool’s answer, open the clause it names and mark it supported, partly supported or not supported. Score the percentage of sentences fully supported, and read every partly as a failure. This catches fabricated and misgrounded citations, and it is the only one of the four that most demonstrations will not survive unassisted.
- The as-of test
Pick a facility with at least two amendments to the same section. Ask the identical question at a date before the latest amendment and at today’s date. The two answers must differ, each must name the version in force on its date, and neither may quote the other’s text. This catches the superseded source, and no general benchmark tests it.
- The silence test
Ask five questions the documents genuinely do not answer — a fee the agreement does not provide for, a consent it does not require. A pass is an explicit statement that the documents do not address it, plus the nearest clause. Anything that reads as a confident general answer is a failure, however well cited.
- The seeded-error test
Put an unexecuted draft amendment and a superseded conformed copy into the file, clearly labelled as such in the metadata but not in the body text. Ask questions whose answers those documents would change. A pass is that neither is ever cited. This tests the document custody rule, not the language model.
Two practical notes. Score the answers before you look at who produced them, and have a second person re-mark a sample of the first test — the clause-support judgement is a judgement, and two markers will disagree on a few. And keep every marked answer: the set becomes your regression suite the next time a version changes.
What a citation check actually removes
Inside the product the equivalent of the clause-support test runs on every answer before anybody sees it. Each sentence is taken with the clause it names and tested against it; what the clause does not support is rewritten or removed; and the checker may not add a source the draft did not already use, so it cannot rescue a doubtful sentence by finding new support for it.
In the sample run that step produced a visible result: seven sentences carrying nine citations went in, nine citations were confirmed, and one sentence was rewritten — the draft had attached a figure to the lenders' funding loss, and under the agreement each lender certifies its own, never an estimate. The rewritten sentence is less satisfying than the one it replaced. That is what the check is for.
The read-back step, and what it changed
The design target the desk configures is that every sentence of an answer carries its clause and page, and that a sentence which cannot be tied to one is removed rather than shown. That is a design goal, not a measurement: it describes what the system is built to do, and the only evidence about what it does on your files is the test you run on them.
The test the date adds: as-of accuracy
The as-of test deserves more than a line, because it is the one failure type that the legal research literature does not cover and that a loan book runs into constantly. Case law accumulates; a 2019 decision is still a 2019 decision. A credit agreement is overwritten. The same section number can carry four different texts over a facility's life, and three of them are wrong on any given day.
Run the test concretely. On the sample facility, ask what a repricing would cost on 1 March 2025 and then on 30 October 2026. The first answer must come from Amendment No. 2 — a 1.00% premium on a repricing before 14 August 2025 — and must note that Amendment No. 3 was not signed until 22 July 2026. The second must come from Amendment No. 3, with a window running to 22 January 2027. A system that gives the same answer twice has no concept of an effective date; one that gives the second answer to the first question is confidently describing a loan that did not exist yet.
There is a maintenance half of the same test. When a new executed amendment arrives, answers already saved against the old text are stale, and nothing in the question changed to signal it. On the sample desk, saved answers that cited a clause the new amendment touches are re-checked and marked, which is what makes a servicing note from last year safe to reuse.
Saved answers, and the one that was re-checked when the clause moved
What to measure, and what a passing score looks like
Supervisors do not publish a threshold for this, and no professional body has set one. What follows is our suggested bar for a loan servicing desk, offered so that a team has somewhere to start arguing from rather than as a standard anybody has adopted.
A scorecard for a cited-answer trial, and the bar we would set
| Measure | How to read it | A bar worth arguing about |
|---|---|---|
| Clause-supported sentences | Of all sentences in the sampled answers, the share fully supported by the clause they name. | Nothing below 100% should be shipped unreviewed; the useful question is how far below, and what the exceptions have in common. |
| As-of accuracy | Of the paired questions, the share where both answers name the version in force on their own date. | Every pair. A single miss here is a structural fault, not a tuning problem. |
| Silence handled | Of the questions the documents do not answer, the share answered with an explicit statement to that effect. | Every one. A desk cannot use a tool that invents the easy cases. |
| Seeded documents cited | How often a draft or superseded copy appears as a source. | Never. If it happens once, the custody rule is not implemented. |
| Minutes to a checked answer | From question to an answer the officer is willing to send, including their review. | Compare with your own current time on the same twenty questions, measured the same way. |
| Differences found and confirmed | Where a term reconciliation runs alongside, how many differences were real. | Count the confirmations, not the flags: a flag that the supervisor dismisses cost time. |
One warning about the first row. A system that always refuses is trivially perfect on clause support, so the measure only means something beside a coverage figure: of the twenty questions, how many produced a usable answer at all. Report the pair together or the number flatters.
Who is responsible for an answer that is wrong?
The person who sent it. That is the position every supervisory document on this subject converges on, and it does not move because a tool drafted the text. The Government Accountability Office's review of AI in financial services notes that "federal financial regulators primarily oversee AI using existing laws, regulations, guidance, and risk-based examinations" — meaning the obligations that already apply to the answer apply to it however it was produced — while also recording that "AI also poses risks, including potentially biased lending decisions, data quality issues, privacy concerns, and new cybersecurity threats" (GAO-25-107197, May 2025).
In the design that follows from this, the drafting steps never send anything: only the servicing officer sends a reply, and it leaves from their own mailbox. That is not a technical limitation dressed up as a principle. It is the point at which a named person has read the answer, opened the clauses they wanted to see, and accepted it — and it is also what makes the record meaningful afterwards, because the clauses behind each paragraph are saved with the reply. The desk's seats and the approvals between them set out where that accountability sits.
The authors’ own conclusion in the preregistered study is the right note to end on: their evidence should inform the responsibilities of professionals in supervising and verifying outputs. Supervising and verifying are activities, with a cost and a method. A tool that makes them cheap is doing something useful; a tool that makes them feel unnecessary is doing something dangerous.
Questions people ask
What is a misgrounded citation?
A reference to a real source that does not support the claim it is attached to. The clause exists, the link opens, the page is right — and the sentence beside it says something the clause does not. It is the dominant failure in retrieval-based systems and the hardest for a reader to catch, because everything visible about it looks correct.
How do you check an AI citation?
Open it and read the clause, not the summary of the clause. Then ask the narrow question: does this text, by itself, support this sentence? Not whether the sentence is plausible, and not whether the clause is about the right subject. On a credit agreement, add a second question — is this the version in force on the date the answer is about?
How many answers do I need to test to learn anything?
Twenty closed questions you have already answered is enough to see the shape of the failures, which is what the first pass is for. It is not enough to estimate a rate precisely, and you should not quote a percentage from twenty samples as though it were one. If the first twenty are clean, the useful next step is harder questions rather than more of the same.
Is a tool that refuses more often a better tool?
Only up to a point, and only if it refuses for the right reason. A refusal on a question the documents genuinely do not answer is a correct answer. A refusal on a question the documents do answer is a failure that costs the desk the same half hour it was trying to save. Measure both, and never report support accuracy without the share of questions answered at all.
Should answers be shown with confidence scores?
A number between zero and one is a weak substitute for the thing a reader actually needs, which is the clause. On a document question the ground truth is in the file and can be opened in one click, so a design that shows the source beats one that estimates its own reliability. Confidence scores are most useful where no checkable source exists, which is the opposite of this case.
Does a human review step solve the problem?
It solves it only if the review is possible. Asking someone to approve a page of prose with no sources is a signature, not a review. Asking them to check seven sentences, each of which opens the clause behind it at its executed page, is a review that takes a few minutes and can be evidenced afterwards. The difference is what the answer carries, not how carefully the reviewer is told to read.