The way to choose AI legal research software is to test it on your own work. To test an AI legal research tool, use questions your team has already answered and counsel has already signed. Ask them again, compare the new answer with the old one state by state and citation by citation, and have counsel read every disagreement. Your own memos are a better test than a vendor's demonstration: they cover the products you sell, counsel has already accepted them, and their dates let you see whether the tool picks up changes made since. Botminds' State Lending Law Research is offered on these terms, run on questions your team answered last month and against your own fee policy. The method below works for any tool.
What to bring to the pilot
- Twenty to thirty signed answers
Past fifty-state or multistate answers counsel reviewed, with their dates, covering your main products and topics.
- The loan facts each one assumed
Product, secured or not, amount, term and payment, so the tool answers the same question, not a near one.
- Where you are licensed
The jurisdictions in scope, so state counts match.
- Your fee and collections policies
The clauses each answer fed into, so the policy check can be tested too.
- A month of live questions
Logged with who asked and how long they took, for the side-by-side weeks.
- Two hours of counsel a week
Counsel decides who was right in each disagreement; nobody else can.
- The measures, written down first
What counts as a match, what would end the pilot, and who decides, agreed before the first question is asked.
What is the best AI legal research software for lending compliance?
The best one for a lender is the one that answers a compliance team's questions, which are not the questions most AI legal research software is built around. The products the term usually brings to mind grew out of case-law research for litigators: find the cases, summarise the holdings, draft the brief. A compliance team at a multistate lender asks something else. It needs the current statute, regulation or regulator guidance for every state, answered for its own product, dated, cited to the passage and compared with its own policy.
Those needs change what "best" means. A tool can be excellent at case law and weak at statutes that change by amendment and index. It can answer one state well and summarise the rest. It can cite well and never say when the text is unclear. Judge it on the work you will give it, not on a ranking. Botminds' State Lending Law Research is built for one band of that work, state lending-law questions about a lender's own products and policies, and should be judged on that band alone.
What should AI legal research software show you in a demo?
Ask for the five things below, on your questions, in the meeting. A vendor's prepared walkthrough shows the tool at its best; these requests show how it behaves at the edges where compliance errors happen.
Five demo requests that separate AI legal research software
| Ask it to | A good answer | A warning sign |
|---|---|---|
| Open the passage for any state you pick | The operative words highlighted in the section, with its version and date | A section number, or a summary, with no text behind it |
| Answer "as of" an earlier date | The text in force then, and what changed since | Only today's answer, or no way to tell which version it read |
| Accept a false premise ("since our state caps late fees at $15…") | The premise restated and corrected before any answer | An answer that builds on the mistake |
| Answer a question its library cannot | "No passage found" and a stop | A confident answer from general knowledge |
| Decide whether a state rule binds a national bank | A flag with the competing text, routed to counsel | A flat yes or no |
How to evaluate AI legal research tools: start from your own signed answers
A test set should be realistic, already judged and representative of your products, and signed answers are all three. A vendor's sample questions are chosen to show the tool well. Generic legal benchmarks test questions your team never asks. Your own signed answers already have a judgment attached: counsel signed each one, on a known date, for a known product.
The responsibility for checking is yours in any case. The US Treasury says firms should check AI use cases "for compliance with existing laws and regulations before deployment" (Treasury, December 2024). A Federal Reserve examiner put it more directly: "the institution remains responsible for ensuring that the output from vendor-provided systems and services meets regulatory requirements" (Consumer Compliance Outlook, 2019). The method has a precedent. The first public test of commercial legal research tools was preregistered, its questions and scoring fixed before any tool was run, and it found the vendors' claims about accuracy "overstated" (Magesh et al., 2024). A buyer should expect to find errors and design the pilot, in the same way, to count them.
A four-week pilot of State Lending Law Research
- Week 1 · replay past answers20–30 questions
- Week 2 · read disagreements with counselevery mismatch
- Weeks 3–4 · run beside the teamlive questions
Week one: re-ask the questions, exactly as they were asked
Each past question should be re-asked with the same loan facts and jurisdictions it had the first time, and also "as of today". Most tools, including this one, can also answer as of an earlier date, because sections are kept by version. Asking as of the memo's date tests whether the tool reads the law the way counsel did. Asking as of today tests whether it finds what changed since. Run both, because they answer different questions.
Then compare state by state. For each jurisdiction, record whether the cap or answer matches, whether the cited section is the same, and whether the tool flagged the state as unclear. Keep the comparison mechanical. Judging who is right comes in week two.
Week two: some disagreements will be your old answer going stale
Every disagreement falls into one of four kinds, and only one of them counts against the tool. The tool may be wrong. The memo may have been right when signed and gone stale since. The memo may have been wrong from the start. Or the text may support both readings, which makes it a question for counsel rather than an error. Expect stale memos in particular. In 2025 the CFPB withdrew guidance documents and rescinded its rule on state enforcement (90 FR 20565), and states amend fee rules on their own schedules, so a memo from 2024 may cite text that has since changed.
How each disagreement is settled
Agent
Re-askThe past question is answered again with the same loan facts and jurisdictions, as of the memo's date and as of today.
Agent
CompareEach state's cap, citation and flag is compared with the signed memo, and every mismatch is listed with both passages.
Never: mark its own disagreement resolved
A person decides
The analyst sorts the mismatchesInto four kinds: tool wrong, memo stale, memo wrong, or text that supports both readings.
A person decides
Counsel rulesCounsel confirms each classification. Stale and wrong memos go into the library as corrected answers.
Stale memos found this way are a result in themselves. If the pilot shows that several signed answers no longer match the current text, that is evidence about the team's existing library, whatever is decided about the tool. Treat them as a finding and fix them. How often do state lending laws change covers why it happens.
The measures to agree before the first question is asked
A measure is only useful if the tool could fail it, so set the thresholds before seeing any results. The measures below are the ones a compliance team can count without a statistician. Your own thresholds go in the last column. Our suggestion for the first row is that it allows no exceptions.
The pass-or-stop card for the pilot
| Measure | How it is counted | Pass, if agreed in advance |
|---|---|---|
| Citations that hold | Every cited passage exists in the current or dated version and says what the answer says. | All of them. One invented or misquoted passage stops the pilot until explained. |
| Caps that match on clear states | States where counsel agreed the text was clear: does the tool's cap on the same payment match? | Your threshold, set before week 1. |
| Flags where judgment was needed | States counsel had treated as judgment calls: did the tool flag them rather than answer flatly? | Your threshold. A flat answer here counts as a miss even if it happens to match. |
| Changes found | Rules that changed since each memo's date: did the tool find them? | Your threshold, checked against the changes counsel confirms. |
| Policy breaches found | States where your fee policy exceeds the cap, against counsel's own list. | Your threshold, with any missed state explained. |
| Hours, side by side | Analyst and counsel time per question in weeks 3 and 4, against your logged baseline. | A reduction you would act on, written as hours. |
The third measure is the one most pilots leave out. Matching counsel's answer on a disputed question is not a success if the tool should have flagged the question as disputed. When the statute is unclear explains why flagged states are the measure that matters most.
Weeks three and four: run beside the team on live questions
Replaying past questions tests accuracy. Running beside the team tests whether the tool changes how the work is done. For two weeks, live questions go to the analyst as usual and to the tool in parallel. The analyst logs hours both ways, counsel logs review time, and product records when it received the answer. Use the reviewed answers, not the tool's raw ones, for real decisions during the pilot.
The saved answers a pilot builds, each with its review and status
Deciding, and what changes after the pilot
The decision rule should be written in week zero: go ahead if every measure passed, stop if the citation measure failed, and otherwise discuss each miss with counsel. After a go-ahead, going live means widening scope: more products, the topics the team watches, the policies linked and the change window set. It does not mean dropping review, which stays in place.
The pilot and going live, side by side
Four-week pilot: 20–30 past answers, then two weeks of live ones in parallel.
After the decision: Every state-law question the team receives.
Four-week pilot: One or two products; the fee policy those answers fed.
After the decision: Each product's reference loan in settings; every policy that state answers feed.
Four-week pilot: Rules on every disagreement.
After the decision: Reviews every saved answer before it leaves compliance and signs every exception.
Four-week pilot: Tested against changes counsel confirms since each memo.
After the decision: Every cited section compared nightly; owners told when an answer goes stale.
Four-week pilot: Agreed in advance; pass or stop.
After the decision: The same measures, re-run each quarter on a sample of saved answers.
Model-risk teams will ask how the tool is governed. The April 2026 interagency model risk guidance that replaced SR 11-7 states that generative and agentic AI "are not within the scope of this guidance", and that it "does not set forth enforceable standards" (Federal Reserve SR 26-2). That leaves each institution to set its own controls. The pilot's measures, re-run each quarter, give it a record of how the tool has performed. The State Lending Law Research page shows the controls the measures test, and What a fifty-state survey costs turns the side-by-side hours into a return.
Questions people ask
How many questions does a legal research pilot need?
Enough to cover your main products and topics, which for most lenders is twenty to thirty multistate questions. Each covers up to fifty-one jurisdictions, so the number of state-level comparisons runs into the hundreds or more.
What if the tool disagrees with counsel?
Counsel decides. The disagreement is classified as tool wrong, memo stale, memo wrong or a genuine ambiguity, and only the first counts against the tool. The other three are findings about the existing answers.
Is AI reliable enough for legal research at a lender?
Reliable enough to read and calculate, if every answer is checked against its passage; not reliable enough to decide. NIST warns of automation bias, "excessive deference to automated systems", which "can exacerbate" the risk of confabulated answers (NIST AI 600-1). A pilot that counts disagreements is the guard against it.
How much does AI legal research software save?
Measure it in your own hours. The solution's own estimate, from its sample data rather than a customer's measurement, is about seven and a half analyst hours by hand for one fifty-state question and about forty minutes with the solution, before counsel's review. The side-by-side weeks of the pilot replace that estimate with your figures.
Can a pilot run without borrower data?
Yes. State-law questions, signed memos and fee policies contain no borrower records, and the loan each question assumes is a reference loan, such as $5,000 over 36 months, not a customer's account.