The best way to evaluate fair lending software is a back-test, and a fair lending monitoring pilot should be one: take a past period whose pricing exceptions your team has already reviewed, run the monitor over every exception in it, and compare its results with your own findings on terms agreed before anyone sees the output. If it reproduces what you found, explains what you cleared and finds planted gaps at the size you planted them, you have evidence to rely on it. If it cannot, you have learned that cheaply. Botminds' Fair Lending Exception Monitoring is set up from your pricing and override policy, your rate sheets and a quarter of past exceptions, so a back-test is the natural first step. Below are the pass marks, the plan, the data and one consideration many pilots skip: what happens when a back-test finds something real.
A pass card for the back-test, agreed before the run
| Test | How it is measured | Passes when | Fails when |
|---|---|---|---|
| Every exception captured | Prices below the rate sheet of the lock day, against your exception report | Every exception on your report is found, and any extra ones are real sub-threshold discounts | Exceptions on your report are missing, or extras cannot be traced to a loan |
| Reasons classed correctly | A sample of reasons read by your analysts, compared with the agents' classes | Agreement meets the level you set, and every disagreement is explained | Unsupported reasons are classed supported |
| Peers you would accept | Analysts review the closest match for a sample of exceptions | Analysts accept the matches as like-for-like on your pricing factors | Matches cross products, rate sheets or markets |
| Known findings reproduced | Last period's flagged branches, officers or products | Each is found, with the same loans | A known finding is missed |
| Clearances explained | Areas your review cleared | Any new flag there is traced to loans and a reason | Flags no one can trace |
| Planted gaps found | Concessions for chosen loans shifted by a known amount | Each planted gap is flagged at about its planted size | A planted gap above your tolerance goes unflagged |
How to evaluate fair lending software
Start from where your fair lending risk sits, then ask each product to prove itself on that risk with your data. "Fair lending software" covers tools that do quite different jobs, and a product that is strong at one can be silent on another. The interagency procedures spread the risk across underwriting, pricing, steering, redlining and marketing (FFIEC, 2009); a lender whose exposure is mostly in loan officer pricing discretion needs something different from one whose exposure is in where it lends.
What different kinds of fair lending software are for
| Kind of tool | Built to answer | Evaluate it on |
|---|---|---|
| HMDA and geographic analysis | How do our applications, denials and prices compare by group and by area with peers? | Your own register against the public file; peer selection; maps by tract |
| Statistical testing of underwriting and pricing | Is there an average gap after credit controls? | Specification, controls, how it handles small groups and estimated race |
| Credit model fairness testing | Does a scoring or underwriting model produce disparate outcomes, and are there less discriminatory alternatives? | Access to the model's inputs and outputs; the tests it runs; how it documents them |
| Exception monitoring | Were the discretionary decisions, concessions, waivers and overrides, made alike for like borrowers, and are their reasons on file? | Capture from the rate sheet, reasons read against evidence, peers, known findings and planted gaps |
Whichever row you are buying, three questions sort products quickly. Does every number trace to loans, so a finding can be defended file by file? Are the settings, peers, thresholds and borrower-group method yours, written down and versioned, or the vendor's defaults? And will the vendor run it on a past period you have already reviewed, with pass marks you set? The rest of this page is about that last question, because it is the one that turns a claim into evidence.
How to test fair lending software before buying: back-test, don't demo
Run it on a past period you have reviewed, because that answers the question an examiner, internal audit and your own model risk team will ask: how do you know it works on your loans? A demonstration on sample data shows what the screens do. A back-test shows what the system does with your rate sheets, your reason fields, your branch structure and your known findings, and it does so against a benchmark you already trust, the review your team completed.
The responsibility does not move to a vendor either way. As a Federal Reserve examiner put it, "the institution remains responsible for ensuring that the output from vendor-provided systems and services meets regulatory requirements" (Consumer Compliance Outlook, 2019). A back-test is the institution meeting that responsibility before relying on the output rather than after.
Build, buy or keep the spreadsheet?
The choice is about coverage and the cost of keeping it up, not only about price. Most programmes start with an exception report, a spreadsheet and a matched-pair sample read by an analyst; some build statistical testing in-house; some buy. Each can satisfy an examiner if it is sound and documented. They differ in how much of the book they see and in who maintains them.
A spreadsheet and a sample, or every exception compared
Spreadsheet and a sample: The exception report, plus whichever files the sample draws.
Software that compares every exception: Every price below the rate sheet, flagged or not.
Spreadsheet and a sample: Chased after quarter end; evidence often gone.
Software that compares every exception: Asked for the day the exception is granted, read against the policy.
Spreadsheet and a sample: Usually missed: a 40-file sample rarely holds more than one or two loans from a cluster of 11.
Software that compares every exception: Tested whenever the group reaches the minimum size you set.
Spreadsheet and a sample: About 22 working days in our estimate for a mid-size mortgage lender.
Software that compares every exception: About 4.5 days in the same estimate, most of the saving in chasing reasons.
Spreadsheet and a sample: Possible with statistical staff; the bank then owns the code, its validation and its upkeep.
Software that compares every exception: The bank still owns the settings, the decisions and the output; the vendor maintains the system.
How long does a fair lending software pilot take?
About a month for most mortgage lenders. A back-test on one past year, followed by a short run beside the live process, fits in about a month for most mortgage lenders. The weeks below are an example plan, not a commitment; the length depends mostly on how quickly rate sheets and loan file documents can be extracted.
An example back-test and parallel run, week by week
- Data and settingsweek 1
- Back-test run and reviewweek 2
- Planted gaps, findings comparedweek 3
- Live exceptions, beside the teamweek 4
Week 1 is mostly about settings, and they deserve care. The peer definition (product, score band, LTV band, market, lock window), the tolerance band, the smallest group tested and the significance level are all choices. They should be the bank's documented policy, approved before the back-test runs, because thresholds chosen after seeing results can be tuned to find nothing. The Justice Department's Evolve order went further and required "methods, thresholds and controls approved in advance" (consent order, ¶13). The second line's view of these choices is in the model risk questions.
The choices a back-test should fix before it runs
The data a back-test needs
Most of it already exists; the work is in pulling it for one period and in finding the loan documents that support reasons. Regulation B requires lenders to keep records used in evaluating an application for 25 months, so a period from last year should still be complete (12 CFR 1002.12).
What to gather for a back-test on one past period
- Pricing and override policy, as in force then
The reason categories, the evidence each requires and the approval tiers. The agents read reasons against these rules.
- Rate sheets for every lock day
So every price below the sheet is found, including discounts that never reached the exception report.
- Loan data for every closed loan
Product, amount, score, LTV, DTI, lock date, branch, officer, approver, and every concession, waiver, credit and override.
- HMDA demographic fields
Reported race, ethnicity and sex where given; the rest is estimated for the compliance team only.
- Loan file documents behind reasons
Competing Loan Estimates, employer letters, deposit verifications, so supported reasons can be told from written ones.
- Your own review of that period
Workpapers, findings, clearances and the sample you read. Without it there is nothing to compare against.
Planting gaps: building the one test that can clearly fail
Reproducing known findings shows the monitor sees what your team saw. Planted gaps show whether it sees what nobody has seen yet, and they are the test most likely to fail, which is what makes them worth running. The method is simple. Before the run, someone outside the pilot team chooses groups of loans in the back-test data and lowers their concessions by a known amount, then records the change in a sealed file.
The plant should be designed to probe the settings, not just to be found. Put one gap well above the tolerance band in a group larger than the minimum cluster size: it must be flagged. Put one just below the band: it should not be flagged, and if it is, the band is not doing what the policy says. Put one large gap in a group smaller than the minimum: it tests what the bank has decided to accept as a blind spot. Spread the planted loans across an officer rather than a branch in one case, since the 2023 findings turned on officer discretion. And keep the borrower group of the planted loans out of the choice, as the peer matcher would.
When the run finishes, open the sealed file and compare. Each planted gap should be flagged at about the size planted, with the planted loans and no others. A planted gap that is missed is a finding about the system or the settings, and the pass card above says what happens next. Afterwards, the planted changes are removed so they cannot reach any report.
What to do if the back-test finds a real gap
Plan for it before the run, because a back-test over a full year of exceptions may well find something a sample did not. The CFPB's 2023 findings included lenders whose discounts below a threshold were never flagged and never monitored (Supervisory Highlights, Issue 30); a test that compares every price with the rate sheet will see those for the first time.
Two rules shape the response. Regulation B's self-test privilege requires "appropriate corrective action" when a self-test "shows that it is more likely than not that a violation occurred", and it does not extend to "loan and application files or other business records related to credit transactions, and information derived from such files and records, even if the information has been aggregated, summarized, or reorganized" (12 CFR 1002.15). A back-test built from loan files should therefore be run on the assumption that its results can be read by an examiner, with counsel involved in how it is scoped and recorded. The response to a finding should be the one the bank would make in live monitoring: the Fair Lending Officer reviews the loans, decides what the gap means and, where it is unexplained, corrects it for every borrower affected.
There is an upside to set against the risk. The interagency procedures let examiners "streamline the examination" where an institution's self-evaluation meets their requirements, and size their own samples to "the strength and thoroughness of the compliance programs" (FFIEC, 2009). A back-test that found a gap and fixed it is evidence of exactly that strength.
The back-test and the live run test different things
Back-test on a past period: Does it find what we found, and what we missed?
Live run beside the team: Does it work on exceptions as they are granted?
Back-test on a past period: The completed review, its findings and clearances.
Live run beside the team: The team's handling of the same exceptions that week.
Back-test on a past period: As written at the time, often missing; tests the reading.
Live run beside the team: Asked for the day the exception is granted; tests the clock.
Back-test on a past period: The pass card: capture, classes, peers, known findings, planted gaps.
Live run beside the team: Time to a written reason; reminders; flags the team agrees with.
Back-test on a past period: Relying on a monitor never checked against your own work.
Live run beside the team: Discovering in production that officers or documents do not arrive as planned.
The decision at the end of the month
The decision rule should be as plain as the pass card. If every test passes, go live, with the quarterly report as the first deliverable and the reason clock running from the first day. If a test fails for a reason the bank can fix, a setting, a missing rate sheet, a reason category the policy never defined, fix it and re-run that test. If a test fails because the system cannot do what the test requires, stop. A pilot that cannot end in "no" is a sales process, not a test.
The first deliverable after go-live: a report an examiner can test
A back-test also gives the go-live its baseline. The share of exceptions with no reason, the number of clusters and their size in the back-test period are the numbers the first live quarter should improve on. On the sample bank, the quarter before reasons were asked for at the time ended with 61 of 549 exceptions unexplained; once they were asked for on the day, the median reason arrived in 1.4 days, against a target of three. How that quarter ran is told in the case study, and why reasons at the time matter is set out in reasons written at the time. The Fair Lending Exception Monitoring page offers a run on a quarter of your own exceptions.
Questions people ask
What is the best fair lending software?
The one that passes a back-test on your own loans for the risk you actually carry. No regulator endorses a product, and the institution stays responsible for whatever it buys. A tool strong at HMDA mapping may say nothing about loan officer discretion, and the reverse.
What is a back-test of a fair lending monitor?
Running the monitor over a past period your team has already reviewed, then comparing its results with your findings on a pass card agreed beforehand. It should reproduce known findings, explain cleared areas and detect gaps planted at a known size.
Does fair lending software replace the Fair Lending Officer?
No. Software can capture, read, compare and test; deciding whether a gap is discrimination, what is owed and what changes stays with named people. In the sample bank, nothing is paid or changed until the Fair Lending Officer signs and the Chief Compliance Officer co-signs.
Should we back-test a quarter or a whole year?
A year, if the records exist. Small clusters at one branch need enough loans to test, and a year also covers rate sheet changes and seasonal volume. A single quarter is a fair start if it is the quarter your team reviewed most thoroughly.
Do we have to share back-test results with examiners?
That is a question for counsel. Regulation B's self-test privilege is narrow and does not cover information derived from loan files, so plan as if the results may be read, and act on anything they find.