Loan document automation: proving it on deals you have already funded

Loan document automation is software that reads the documents in a credit file, pulls the terms out of them and checks those terms against something it already knows — an approval, a policy, a programme rule — so that a person is handed a conclusion with its evidence rather than a stack to read. In indirect auto lending that means the deal jacket: the signed contract, the buyer's order, the stipulation documents, the add-on forms, the odometer disclosure and the title application. The claim is easy to make and almost impossible to judge from a demonstration, because a demonstration is run on documents chosen by the person running it. The only honest test is a replay on deals your own desk has already decided, and the number that matters in that test is not accuracy. It is the false clear: a deal the machine called clean that your desk would have pended. Indirect Auto Contract Funding is built to be measured that way.

The figures on this page come from a fictional sample book inside the solution — a credit union, twelve dealerships and a weekend of contracts. They are sample data, not a customer's measured result, and they are here to show what a replay produces rather than to stand as a benchmark.

What loan document automation is, and what it is not

It is worth separating three capabilities that are usually sold as one. The first is extraction: reading a scanned or electronic document and returning the values on it with a reference to where each came from. The second is comparison: holding those values against a different document or a stored decision and reporting the differences. The third is judgement: deciding what a difference means and what should happen next. The first two are mechanical and testable. The third is where the money and the regulatory weight both sit.

What it is not is a replacement for the credit decision. The approval was made before the jacket existed, on an application; the jacket is evidence about whether the thing that got signed is the thing that was approved. Software that reads the jacket is doing verification, not underwriting — which is also why it can be measured so cleanly. Verification has a right answer that somebody already wrote down.

Why a demonstration proves nothing

A vendor demonstration is a performance on curated material. Even an honest one — clean scans, one state's contract form, the exception the product handles best — tells you almost nothing about the fifth percentile of your own post. The documents that break a reader are not the interesting ones: they are a phone photograph of a pay stub, a nine-page lease addendum stapled behind a contract, a dealership that prints two deals on one PDF, a form a state revised last quarter.

There is also a structural problem with watching a demonstration. You are shown deals where the software found something. You are not shown the deals where it found nothing and there was something to find, because nobody in the room knows which those were. That asymmetry is the whole reason the test has to be run against deals whose answer is already known.

The replay test: re-run closed loans you have already funded

The supervisory texts already describe this kind of testing, because it is what quality control is. The Comptroller's handbook: "Effective QC should begin with fundamental transaction testing and review and should be performed by an independent group, where a prescribed sample of applications of new or existing loans is reviewed for adherence to bank policies and procedures and to ensure that exceptions are properly identified and reported" (OCC, Installment Lending, Comptroller's Handbook). A replay is that sentence with the sample re-read by software instead of by the quality control team, and with the team's own previous answers as the standard.

Two design choices make the difference between a replay that teaches you something and one that flatters everybody. Run it blind: if the system can see the pend reason, it is not being tested, it is being graded on a sheet with the answers on it. And include the pends, in proportion. A sample of funded deals contains almost no exceptions by construction — that is why they funded — so it measures nothing but the ability to say yes.

Size follows from how rare the thing you are counting is. If roughly a quarter of jackets carry at least one exception, a few hundred deals gives you several dozen exception deals to adjudicate, which is enough to see patterns and not enough to make the rarer ones statistically solid. For the tail — a product one dealership sells, a form one state uses — you are sampling for presence rather than frequency: does it handle it at all. In the sample book, 73 per cent of deals across thirty days were clean first time and 61 per cent on the weekend shown, so a 400-deal replay would be carrying roughly 110 to 155 exception jackets.

The two errors, and why only one of them is expensive

A verification system can be wrong in two directions, and the instinct to average them into one accuracy figure destroys the only information in the test. A false pend is a deal the machine sent back that your desk would have funded. It costs a dealership an afternoon, a relationship manager an awkward call, and the desk the few minutes it takes a person to overrule it. A false clear is a deal the machine passed that your desk would have stopped. It costs you a funded loan with a defect in it — an unperfected lien, an uninsured car, a price your programme never authorised — and you find out when a payment stops or an examiner pulls the file.

Tolerating some false pends is how you buy a near-zero false clear rate, and that trade is usually worth making explicitly rather than by accident. A desk that overrules twelve unnecessary pends a week and has never funded a defective jacket is in a far better position than one with a tidier headline number and no idea which way its errors fall.

How many closed deals a replay needs

Enough for every kind of deal to appear at least a handful of times, which in practice is driven by your mix rather than by a formula. Count the distinct shapes you fund: new and used, each state's contract form, each e-contracting channel, leases if you buy them, the product families that carry caps, the dealerships whose packages are notoriously thin. Then make sure each shape has enough deals in the sample to be judged at all. A replay of 500 deals that are all from three dealerships in one state has tested one shape 500 times.

The same arithmetic applies to the exceptions themselves. If you want to know whether the system catches a missing lienholder, your sample needs deals where the lienholder was missing. In the sample book those pend reasons are unevenly distributed, and at one dealership they look like this.

That distribution is also a shopping list for the replay. The two reasons at the bottom are the ones a thin sample will contain two or three of, and they are the ones worth adjudicating most carefully, because a missed add-on cap or a missed mileage mismatch is a false clear with a cost attached. What a mismatched mileage does downstream is set out in the odometer guide.

A deal Indirect Auto Contract Funding called clean, with its evidence

Reading the disagreements: the machine was right more often than you think

The most common surprise in a replay is not the error rate. It is how many disagreements turn out to be cases where the desk was wrong, or where the rule itself does not say. A pay stub dated thirty-one days before submission is inside or outside a thirty-day window depending on whether the window runs to submission or to completion, and most programmes have never written that down. Two analysts will answer it differently and both will be confident.

Those cases are the valuable output. The OCC handbook makes the same point about exception data generally: "Another benefit of exception tracking and analysis is the ability to drill down to the sources of those exceptions. By evaluating exceptions and subsequent performance by product, loan grade/credit quality, underwriter, manager, and channel (including individual dealer and branch), management can target corrective action to the source of the problem" (OCC, Installment Lending). A replay is the first time most programmes see their own rules evaluated that way, and the ambiguities it surfaces have to be resolved as policy before any score means anything.

So adjudicate into four buckets, not two: machine wrong, desk wrong, rule ambiguous, and document genuinely unreadable. Then re-score with the ambiguities decided. The honest acceptance decision is made on the second score, and the first one is kept, because the gap between them is a measure of how much of your programme lived in people's heads.

What stays with a person whatever the score says

A good replay result is an argument for automating verification. It is not an argument for automating the decision, and the supervisory texts are unambiguous about where the decision lives: NCUA's indirect lending guidance requires quality control processes that verify "the completeness, accuracy and validity of loan deals" while leaving approval authority inside the credit union (NCUA, Letter to Credit Unions 10-CU-15). What that means for a desk, rule by rule, is worked through in the due diligence guide.

Three classes are worth keeping with a person even in a programme with an excellent score. Anything that moves a deal off its credit approval, because that is a new credit decision. Any waiver of a condition, for the same reason. And the release of money, which in the sample programme is signed by an analyst on every deal and by a supervisor as well above a threshold. None of those are reading tasks; they are authority tasks, and a replay measures reading.

A disagreement, as Indirect Auto Contract Funding presents it

Keeping the score after go-live

A replay is a purchase decision. What keeps the decision true is the same measurement running afterwards, on a sample, forever. The OCC handbook puts continuous monitoring in the same breath as transactional testing: "The credit review function for larger banks, or banks with complex installment lending activities, should also include transactional testing of accounts for underwriting and collections. A 'best in class' credit review would also conduct continuous monitoring" (OCC, Installment Lending).

In practice that is three standing numbers. The override rate: how often an analyst disagrees with the recommendation, by rule, which is the early warning that a rule has drifted from the programme. A sampled false clear rate: a small number of funded deals re-read by a person each month, chosen at random. And clean first time by dealership, which is the operational number that tells you whether the pends you send are landing — 73 per cent across the sample book's thirty days, with a spread from 61 to 85 per cent between dealerships.

The last of those is also the one that pays for the exercise, because a documentation problem traced to a source is a problem that can be ended rather than caught. The handbook puts the end of that road bluntly: if "a particular dealer consistently fails to deliver contracts that conform to the bank's underwriting requirements, and the loans subsequently perform poorly, it may warrant terminating the bank's relationship with that dealer". Most of the time it warrants a conversation instead, and having the numbers is what makes the conversation possible. What a buyer should ask about the step the origination system does not cover is set out in the guide on where the origination system stops. To run a replay on your own closed deals, start from Indirect Auto Contract Funding.

Questions people ask

How accurate is AI at reading loan documents?

The question has no answer in the abstract, because accuracy depends on your document mix, your scan quality and the rules being applied. The answerable version is: on a sample of your own closed deals, how often did it disagree with your desk, and which way did the disagreements fall? Ask every vendor for that measurement on your material rather than a published accuracy figure on theirs, and insist that the result is reported as two separate error rates rather than one average.

How do you test document automation before you buy it?

Replay deals you have already closed. Give the system the same jacket and the same credit approval your desk had, with no sight of what happened next, and compare its verdict with the one your desk recorded. Include the pends in proportion, cover every state form and channel you fund, and have someone with authority adjudicate every disagreement into machine-wrong, desk-wrong, rule-ambiguous or unreadable.

What is a false clear, and why does it matter more than accuracy?

A false clear is a jacket the system passed that your desk would have stopped — a missing lienholder, an unsigned form, a price above your cap. It matters more than overall accuracy because nobody finds it at the time: an unnecessary pend is discovered within hours by an annoyed dealership, while a defective funded loan is discovered by a loss or an examiner. The two errors have completely different costs and should never be averaged into one number.

How many past deals do you need for a pilot?

Enough that every shape of deal appears often enough to judge, and enough exception jackets to see patterns — a few hundred is a reasonable starting point for most indirect programmes. Drive the number from your mix rather than a rule of thumb: each state's contract form, each submission channel, each product family and your thinnest dealerships all need to be present. A large sample drawn from three dealerships in one state has tested one thing many times.

What does a human still have to check after automation?

Anything that is an authority decision rather than a reading decision. A term that falls outside the credit approval is a new credit decision and belongs to the analyst who made the first one. A waiver of a condition is the same. And the release of money is signed by a person — in this solution, by the funding analyst on every deal, and by a supervisor as well above the programme's second-release threshold.

What should the acceptance threshold be?

Set by you, in writing, before the replay produces a number — because every threshold becomes negotiable once a result exists. State a ceiling on false clears measured on exception deals specifically, a separate ceiling on false pends measured against what an unnecessary re-sign costs you, a requirement that each deal shape passes on its own rather than on the average, and an agreed consequence if the test fails.