HMDA data validation means proving that every field on the loan application register (LAR) matches the loan file behind it, because that is how examiners test it: they pull a sample of files and count differences field by field. Botminds' HMDA and 1071 Data Validation hands the first pass to AI agents, which read each loan file, find the page every reportable field comes from, apply the FFIEC tolerances and show people only the fields that disagree. Before a bank relies on any tool like that, model risk and internal audit need a way to validate the AI model behind it. The answer is to measure it like a control, with five numbers: source coverage, break precision, break recall, tolerance discipline and refusal honesty. Each has a numerator a second person can recount.
The stakes are not abstract. Examiners use HMDA data in fair lending and CRA examinations, Regulation C is among the most cited consumer rules in bank exams, and a validation tool that misses errors fails quietly: the register looks clean until the examiner's sample says otherwise.
Why HMDA validation has to be measured, not assumed
38.2%
of all consumer-law violations Federal Reserve examiners cited at state member banks in 2024 were under Regulation C §1003.4(a)
253 of the violations cited in 223 exams
16%
of sampled files still carried errors in one bank’s 2017 register after it had been corrected and resubmitted
21 errors in 13 of 81 files, CFPB review
174,000+
data entries one lender corrected across 35 fields after a CFPB review of 159 files found 51 errors
nearly 20% of its applications affected
Is an HMDA validation agent a model under the 2026 model risk guidance?
Not cleanly, and that is the first thing to write down. On 17 April 2026 the Federal Reserve, FDIC and OCC replaced SR 11-7 with new interagency guidance (SR 26-2). It defines a model as "a complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates", and excludes "deterministic rule-based processes" and simple arithmetic. A footnote then sets generative and agentic AI aside: they "are not within the scope of this guidance", though a bank's own risk management "should guide the determination of appropriate governance and controls" for tools the guidance does not cover (SR 26-2 attachment). The letter says it is most relevant to organizations above $30 billion in assets.
An HMDA validation agent is therefore three things at once, and each part answers to a different discipline:
- The reading (finding the adverse action notice in a 212-page file and taking the date off it) is agentic AI. It is outside the model guidance and has to be governed by outcome testing the bank designs itself.
- The comparison (is 03/14 within three calendar days of 03/21?) is a deterministic rule. It is software, and it is validated the way software is: against boundary cases with known answers.
- The odds (the chance an examiner's 79-file sample orders a field resubmitted) are a statistical estimate. That part does fit the definition of a model, and its assumptions deserve a challenge of their own; how examiners test HMDA data sets out the sampling rule the estimate rests on.
If a vendor supplies the tool, the 2023 interagency guidance on third-party relationships applies as well. And the vendor section of SR 26-2, while written for models, gives the right instruction for the AI part too: "sound practice involves conducting ongoing monitoring and outcome analysis to assess whether vendor models are accurate, remain fit for purpose, and continue to be reliable." The rest of this page is that outcome analysis, made specific to HMDA.
The five measures of an HMDA validation agent
Five ratios cover what can go wrong: the agent may not look at a field, may raise a break that is not an error, may miss an error that is there, may apply a tolerance wrongly, or may invent a value when the file has none. Each needs its own count, because a tool can be excellent on one and dangerous on another.
The definitions follow the examiner's. The FFIEC HMDA Examiner Transaction Testing Guidelines tell examiners to "verify the accuracy of the data in the entries in the HMDA LAR sample(s) against the corresponding loan files", to let the institution explain differences, and to count the rest as errors. An agent's job is to do the same comparison earlier and on every entry, so its measures borrow the same vocabulary: field, file, difference, tolerance, error.
Five measures for an agent that checks HMDA data
- Source coverage
Applicable fields with a cited source page ÷ applicable fields, per entry and per field. A field with no page was not checked against the file, whatever the result column says.
- Break precision
Breaks a person corrected, or confirmed as a real difference with the file ÷ breaks raised (excluding differences closed as within tolerance). Low precision burns reviewer days and trains people to wave breaks through.
- Break recall
Known errors the agent broke ÷ known errors present, by field and by error type. Known means seeded by you or found by an examiner. This is the one that protects the bank.
- Tolerance discipline
Boundary cases classified exactly as paragraph 9 of the FFIEC guidelines says ÷ boundary cases. Two failure directions: counting a 3-day date difference as an error, and excusing a 4-day one. Target: 100%, because the rule is deterministic.
- Refusal honesty
Fields whose source was removed that the agent marked “no source” ÷ fields whose source was removed. The complement is the guess rate, and it must be zero.
Two of these are easy to get wrong. Precision is not "breaks the data manager agreed with". When a reviewer keeps the register value because a later document supports it, the break was false. When she keeps it because the document itself is wrong and a corrected copy has been requested, the break was true: the register and the file really disagreed, and an examiner would have asked the same question. A tool that records only "resolved" cannot tell these apart, so ask how the reason is captured before you trust a precision figure.
Coverage is the measure most often skipped, and it decides what the others mean. Take a sample lender's third-quarter register as it runs through HMDA and 1071 Data Validation: 1,846 entries, about 86 applicable fields each. On one denied purchase application, the loan file reader opened 31 documents and 212 pages and cited a source page for 86 of the 86 fields that apply. A recall test on that entry means something because nothing was skipped. On a tool that compares the LAR with the loan origination system, coverage of the file is zero by design, and no recall figure can fix that. The key fields mapped to their pages lists what "a source page" should mean for each field.
Coverage you can audit: the run on one application
The line to check first is the second one. Coverage (86 of 86) has to be true before any match count below it means anything.
What a true break, a false break and a missed error look like in HMDA data
Every check an agent makes falls into one of four cells, and two more outcomes sit beside them. Auditors know the matrix; what makes it useful here is filling it with HMDA cases, because the error types are not equally likely or equally costly. A missed error on a demographic field has no tolerance to hide behind, and a false break on rate spread wastes the most expensive reviewer time.
Four outcomes of a field check, with HMDA examples
The two outside cells are where an agent differs most from a rules engine. A rules engine has nothing to say about a missing page; it compares what it is given. An agent that reads documents can always produce something, and a model under pressure to fill a field will fill it. The loan file reader in HMDA and 1071 Data Validation is built to mark a field "no source" instead, and the refusal-honesty measure exists to prove that it does, on files where you removed the page on purpose.
Differences that are not adequately explained should be identified as errors.
FFIEC HMDA Examiner Transaction Testing Guidelines, paragraph 3
Seeded-error testing: plant known HMDA errors and count what comes back
A seeded-error test is the most direct way to measure recall: take loan files whose register entries are known to be right, change some values on purpose, run the agent, and count how many of the planted errors come back as breaks. It answers the question production data never can, because in production you do not know which errors were missed.
The seeds should copy how HMDA data actually goes wrong, not random noise. The common causes are well known to any data manager: an action taken date carrying the date the file was moved to Suspended rather than the decision; net income instead of gross, or a bonus the underwriter excluded; the adjustable-rate APOR table used for a fixed loan, or the application date instead of the lock date; demographic information collected online but never keyed; property value taken from the purchase contract instead of the appraisal; a census tract from the prior vintage; a cash-out refinancing reported as a refinancing.
Then add two families of seeds that test the other measures. Boundary seeds sit on the tolerance line: an application date exactly 3 calendar days off (must not count) and 4 days off (must); a loan amount $1,000 off and $1,001 off; an action taken date 2 days off that crosses 31 December, which the guidelines say must still count because it reports data for the wrong calendar year. Removal seeds take the source page out of the file altogether, so the only right answer is "no source".
A seeded-error test that measures recall, tolerance discipline and refusal at once
A person decides
Pick clean filesEntries already corrected and signed, from at least two channels and every action type; the list is kept from the agent.
A person decides
Plant three kinds of seedRealistic errors by field and cause; boundary cases on each tolerance; source pages removed.
Never: tells the vendor which entries carry seeds
Agent
Run as in productionThe same procedure version, tolerances and schedule the bank will use; no special settings for the test.
Agent
Report every outcomeBreaks, matches, within-tolerance notes and no-source marks, each with the page it relied on.
A person decides
Score and signRecall by field and cause, boundary accuracy, guess rate; every miss read and explained; the scorecard signed by the second line.
How many seeds are enough? Fewer than most teams plant. If an agent catches every one of n seeded errors of a type, the 95% upper bound on its miss rate for that type is about 3/n, the statistician's rule of three. Catch all 30 seeded income errors and the true miss rate could still be 10%. Catch all 300 and it is bounded near 1%. So plant by error type, not in one pool, and put the most seeds where the field has no tolerance: demographics, rate spread and denial reasons.
Back-testing on a register an examiner already sampled
Back-testing is the second source of known errors, and the stronger one, because someone outside the bank chose the files and judged them. Every examination leaves a set of labelled entries: the files in the sample, the fields reviewed, and the differences counted as errors after the bank had its chance to explain them. Run the agent on those entries exactly as they were first filed and two numbers fall out: the share of the examiner's errors it breaks (recall against an independent labeller), and the breaks it raises that the examiner did not count (to be adjudicated, not assumed false, since examiners may review only designated fields).
A resubmission is an even larger label set. When the CFPB found 51 errors in an initial review of 159 files in Freedom Mortgage's 2020 submission, the company corrected 35 data fields on resubmission, "errors in over 174,000 data entries affecting nearly 20 percent" of its applications (CFPB, October 2023). A bank with a past resubmission has its own version of that file: the original register, the corrected one, and the difference between them. Running the agent on the original and counting how many of the corrected values it would have broken is about as realistic as recall testing gets.
The human baseline: how accurate is the manual HMDA scrub it replaces?
Nobody publishes recall for a manual HMDA scrub, and that is the point to raise with the first line. An agent is usually asked to prove itself against perfection, while the process it replaces has never been measured at all. The public record shows how far a checked register can be from clean. Washington Federal's internal audit of its 2016 register found errors in 40 of 100 files; the CFPB then found 58 errors in 27 of 84 files in the initial 2017 register, and 21 errors in 13 of 81 files in the resubmitted one, a register that had already been corrected (CFPB consent order, 2020).
The fair comparison is to give the same seeded files to the people who scrub today, blind, under their normal time pressure, and score them on the same five measures. The result is rarely flattering to either side, and both numbers belong in the model risk file. Three ways to validate HMDA data tests a sample scrub, a rules engine and a full field-to-file check on the same application.
What the manual scrub and the agent can each be measured on
Manual sample scrub: The files in the sample, often a few hundred a quarter; the rest unchecked
Agent on every field: Every applicable field on every entry, each with the page it was read from
Manual sample scrub: Measurable only by seeding files into the reviewers’ queue
Agent on every field: Measurable by seeds, exam back-tests and past resubmissions
Manual sample scrub: Rarely an issue; a person flags only what they are sure of
Agent on every field: Must be tracked, from the reason given each time a register value is kept
Manual sample scrub: Applied from memory or a checklist, varies by reviewer
Agent on every field: Applied by rule; testable on boundary seeds to the day and the dollar
Manual sample scrub: Noticed, usually; noted inconsistently
Agent on every field: Must be marked “no source”; testable by removal seeds
Manual sample scrub: Worksheets and email
Agent on every field: Every break with its page, rule and decision, kept apart from the agent’s finding
What the after-check exam odds can and cannot tell you
Resubmission odds are a poor measure of a validation tool, because they collapse once recall passes about 80% and say nothing about the difference between 85% and 99%. That is worth understanding before anyone presents an odds chart to a committee as evidence of accuracy.
The FFIEC design for a register of 501 to 100,000 entries reviews 35 files first, extends to 79 when a field shows 2 errors, and orders that field corrected and resubmitted at 4 errors in the 79 (FFIEC guidelines). Model a field whose error rate as first entered is 5%, and let a validation tool remove a share of those errors before filing. The chance of a resubmission order on that field falls fast:
Above 80% recall, the exam odds stop distinguishing a good checker from an excellent one
Two consequences for audit. First, an "after" figure on any vendor's odds chart is only as good as the recall behind it. The solution's own estimate on its sample register (37 in 100 that one 79-file sample orders a field resubmitted as first entered, under 1 in 100 after the check) treats the remaining open breaks as the only errors left, which assumes the agents found every error. That assumption is exactly what seeded tests and back-tests exist to check. Second, the threshold is not the objective. The guidelines let examiners order a resubmission below the threshold when errors "will likely make analysis of the HMDA data unreliable", and require prospective changes to policies and audit processes for errors "capable of repetition" even when they fall within tolerance.
An odds card states its assumption; the recall test supplies the evidence
Read the green bars as “still open among the errors the check found”. Recall measured on your own files is what turns them into “still open among the errors there are”.
Tolerance discipline and rule versions: the deterministic part should be perfect
The tolerances are the one part of HMDA validation with a written right answer, so the standard there is 100%, and the control that matters is who can change them. The FFIEC lists four: 3 calendar days on the application date, $1,000 on the loan amount, 3 calendar days on the action taken date provided the difference does not move the data into the wrong calendar year, and rounding to the nearest thousand on income relied on. Everything else, including rate spread, ethnicity, race, sex and denial reasons, is exact. And the guidelines count an applicant's ethnicity or race as one data field group, so errors in two of its six or eight FIG fields on one entry are one error, not two.
A validation tool should not let anyone widen those numbers. In HMDA and 1071 Data Validation the tolerances can only be tightened, every change applies from the next run and is kept as a new rule version, and the bank's own procedure rules (a denial reported on the adverse action notice date, income taken from the underwriting worksheet) sit beside them with their version and date. For an auditor, that means a break from March can be replayed against the rules in force in March.
Tolerances that can be tightened, never loosened
Test it: try to raise a tolerance above the FFIEC value, then ask which rule version the next run used and when it took effect.
Evidence an examiner and internal audit can read
A measurable agent leaves two trails, kept apart: what the agent found, and what a person decided. That separation is what lets audit compute precision and recall after the fact, on production data, without trusting anyone's summary. If the agent's finding is overwritten by the human decision, the false breaks disappear and so does the measure.
For each break, the evidence should carry the register value, the file value, the difference, the rule and its version, the source page, the decision (correct, or keep with a reason), the person who signed and when, and a second reviewer on any change to ethnicity, race or sex. The keep decision is the one to watch. Each time a reviewer keeps a register value against the file, the reason classifies the break for the precision count, and the compliance testing manager sees it at sign-off.
A kept value needs a reason, and the reason feeds the precision count
What an examiner will see, in the end, is the corrections list: every change to the register this year with its before, after, evidence page and signer, exported for the examiner. That list does not prove the agent is accurate. The seeded tests, back-tests and boundary results do, and they belong in the model risk file with the date the acceptance bar was set. Inside the six agents shows which agent each measure tests and what each is forbidden to do.
Ongoing monitoring: what to re-measure each quarter
The agent should be re-tested whenever its inputs change, and checked lightly every quarter even when nothing seems to. SR 26-2 frames ongoing monitoring as asking whether a model performs "as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions". For HMDA that list is concrete: a new document template from the loan origination system, a new channel or acquired branch, a new FFIEC Filing Instructions Guide, a procedure change, or a change to the model behind the agent.
- Every quarter, from production: source coverage by field and document type; precision from the keep reasons; the share of breaks closed as within tolerance; the no-source rate by branch (a rising rate usually means a missing document, not a weaker agent).
- Every quarter, by test: a small batch of fresh seeds mixed into the live run, including boundary and removal seeds, scored blind.
- After every exam: the back-test on the examiner's sample, with every disagreement explained.
- On any change to procedure, templates, the Filing Instructions Guide or the underlying model: the full seeded suite again, before the next quarter-end run.
One more measure belongs here because no field-to-file check can see it. When the source document is itself false, the register and the file agree, and every measure on this page passes. The CFPB found that loan officers at Bank of America recorded that applicants declined to give race and ethnicity without asking them; after the bank stopped monitoring "information not provided" rates in 2016, those rates rose from 6% to 9% within a year and reached 17% by early 2020, and the order carried a $12 million penalty (CFPB consent order, 2023). Distribution monitoring (not-provided rates, visual-observation rates and income patterns by loan officer) catches what matching cannot. Keep it, and keep it outside the tool you are testing.
Measured this way, an agent stops being a black box and becomes a control with known error rates, tested the same way each quarter. The design these measures are written against, with every field drawn to its page and every change signed by a person, is on the HMDA and 1071 Data Validation page; going live covers when the first seeded run fits in.
Questions people ask
Does an AI tool that checks HMDA data need model validation?
The 2026 interagency guidance excludes generative and agentic AI and deterministic rules from its definition of a model, but tells banks to set governance for tools it does not cover. In practice that means outcome testing (seeded errors, back-tests, boundary cases) for the reading, software testing for the tolerance rules, and a model review of any resubmission-odds estimate.
Can we rely on the vendor's own accuracy figures?
Use them as a starting point, not as evidence. A vendor's test files are not your loan files, templates or channels, and SR 26-2's vendor section expects the bank's own ongoing monitoring and outcome analysis. Ask for the denominators and the list of misses, then run your own seeds.
Does the same framework work for Section 1071 data?
Yes, with different seeds. The 1071 practice register has no published examiner tolerances yet, so tolerance discipline tests the bank's own written rules, and the critical seeds test whether an application belongs on the register at all: for example a stated revenue under $1 million where the tax return shows more. Section 1071 under the 2026 rule covers the practice register.
What recall should we require before going live?
Set the bar before the test, per error type, with a minimum number of seeds behind it, so the result cannot be argued into a pass. Fields with no tolerance (demographics, rate spread, denial reasons) deserve the strictest bar, and the guess rate on removal seeds should be zero without exception.
Should we keep a manual sample once the AI check is live?
Yes, a small one, drawn from records the agents passed with no break. It is the only production measure of what the check misses, since a missed error by definition raises nothing. Statistics sets its limit: if a random sample of 100 clean records turns up no error, the miss rate is below about 3% per record with 95% confidence, so the sample bounds the misses rather than proving there are none.