The fairest way to evaluate credit review software is a pilot that starts with reviews the bank has already finished. Take about 25 completed annual reviews from the last year or two, load the documents the analysts had at the time, and let the software produce its own comparison, covenant results and proposed grade for each. Because the bank knows what its analysts concluded, what loan review later said and what each borrower did next, the pilot can be scored rather than admired. The criteria are written before the replay starts, and at least one of them must be able to fail. Botminds' Annual Credit Review is built to be tested this way: every figure in its output links to the page it came from, so each difference from the bank's own review can be traced to a document, a definition or a judgment.
An annual review pilot in six steps, with the decisions people keep
A person decides
Write the criteriaCredit and loan review agree the measures and the pass marks, in writing, before any file is loaded.
A person decides
Choose 25 completed reviewsA risk-weighted sample, including loans later downgraded and loans renewed without paydown.
Agent
ReplayThe agents collect, spread, re-test covenants, check accounts, collateral and guarantors, and propose a grade for each, using only the documents on file at the time.
Never: Sees the bank's grade or the later outcome.
A person decides
Compare blindA reviewer who did not do the original review scores each difference: agent right, analyst right, or judgment.
Agent
Run beside the teamDrafts live reviews coming due while analysts also work them, so hours and flags can be compared on new files.
A person decides
DecideThe chief credit officer reads the scorecard against the criteria written in step 1.
What is credit review software, and which kind are you buying?
"Credit review software" covers at least three different products, and a bank should know which job it is buying for before it runs any test. Loan review software serves the independent function: it plans the sample, holds the reviewer's workpapers, records findings and tracks management's responses to them. Annual review software serves the line: it gathers the year's documents, spreads and re-tests the relationship, and drafts the review the analyst signs. Portfolio monitoring tools sit beside both, watching grades, concentrations and early-warning signals across the book.
The jobs answer to different rules. Loan review must be performed by people independent of the loans they test, and the 2020 interagency guidance expects its results to reach the board with "management's response to substantive criticisms or recommendations". The annual review is the line's own work, owned by the analyst and approver. A tool that blurs the two, letting the same workflow both write the grade and certify it, gives a bank less than either. Annual Credit Review is the second kind: it does the line's annual review, and it gives loan review a better-evidenced file to sample. The pilot below is built for that kind of tool, though most of it applies to any of them.
How do you evaluate credit review software? Replay completed reviews
By replaying reviews whose answers are already known. For a live review, nobody can say whether a proposed grade is right until the borrower's next year plays out. For a review completed a year ago, the bank knows the grade its analyst chose, whether loan review or an examiner later changed it, and whether the borrower then breached, paid down, upgraded or failed. That is the same logic examiners use: "To determine whether a bank's risk ratings are accurate, examiners will review a sample of loans and compare internal bank ratings with those assigned by OCC staff" (OCC, Rating Credit Risk). A pilot applies it to the software.
It also puts the bank in the position the regulators expect. The interagency third-party guidance is clear that using an outside provider "does not diminish or remove banking organizations' responsibilities to ensure that activities are performed in a safe and sound manner" (88 FR 37920). A pilot whose criteria the bank wrote, scored by the bank's own reviewers on the bank's own files, is evidence the bank can show an examiner. A vendor's demonstration is not.
A replay can be scored because the bank already knows the answers: what it decided, and what the borrower did next.
Choosing the 25 reviews: a sample that can embarrass the software
The sample should be built to find failures, not to show off. The OCC's 2026 sampling list for examiners is a good model because it is aimed at the loans where grading goes wrong: "Lowest-rated pass credits (e.g., watch-list loans)", "Recently upgraded loans", "Loans with high numbers of modifications, renewals, or extensions", "Loans to borrowers with significant overdrafts" and "Special mention and classified loans" (OCC, Lending and Loan Portfolio Risk Management).
A workable mix of 25: eight relationships that were later downgraded, to see whether the replay flags the move earlier than the bank did; five renewed without principal reduction; five stable credits where nothing moved, to see whether the software stays quiet; four with a covenant breach or waiver; and three complex relationships with several entities and guarantors. The exact numbers matter less than including each kind. A sample of only clean credits proves nothing; a sample of only disasters proves nothing about everyday reviews.
What data do you need for a credit review software pilot?
The files an annual review pilot needs
- Credit policy and review template
Review frequency rules, early-warning thresholds, who approves what, and the bank's annual review layout.
- Grading model and scale
The factors, weights and definitions, and the mapping of each grade to pass, special mention and the classified grades.
- For each of the 25: the prior approval and the documents on file at the time
Last approval memo, the year's statements and returns, compliance certificates, borrowing bases and guarantor statements, as the analyst had them.
- Credit agreements and every amendment
The covenant definitions are the agreement's, not the certificate's; amendments change them.
- Twelve months of account history
Payments, overdrafts and month-end line usage from the core system for each relationship.
- Collateral files
Appraisals or evaluations, insurance certificates and UCC filings with their dates.
- The answers, held back
The grade the bank chose, any later regrade by loan review or an examiner, and what the borrower did in the following year. These stay with the scorer, not the software.
What should a credit review software pilot test? Criteria that can fail
Each criterion needs a measure, a pass mark and a way to fail. A criterion that cannot fail is not a test. The marks below are examples to argue over; the bank should set its own, and record them before the replay.
Example pilot criteria for annual review software, each one able to fail
| What is measured | How | Example pass mark |
|---|---|---|
| Every figure traceable | Share of numbers in the comparison and draft that link to a source page | All of them; any untraceable figure is a fail for that review |
| Covenant results | Recomputed by the agreement's definitions and amendments, compared with the bank's own recomputation | Same pass or breach on every test; every difference from the certificate explained |
| Reported figures untouched | Spread figures compared with the statements | No reported figure altered; adjustments on their own lines |
| Early warning | For loans later downgraded, did the replay flag a moved line the original review missed? | Set by the bank; a replay that flags none fails |
| Quiet on stable credits | For loans where nothing moved, lines flagged that a reviewer judges noise | Few enough that a stable review still clears in minutes |
| Proposed grade | Against the best-informed grade (the bank's, or loan review's if it changed it), within one grade and at the pass / special mention / classified boundary | Set by the bank, with every boundary miss explained |
| Direction of misses | Of grade differences, the share where the proposal was less conservative | Not mostly on the generous side |
| Analyst time | Hours to a signed review on live files, against the bank's timed baseline | Set by the bank from its own baseline |
Two of these deserve attention because they cut both ways. "Early warning" is the reason to buy annual review software and the easiest thing to fake with hindsight, which is why the replay must use only the documents on file at the time. "Direction of misses" protects against a tool that agrees with analysts by being lenient: the OCC warns that "it can be relatively easy for loan officers and line managers to rate credits a step, or more, above what they deserve", and a proposal that leans the same way adds nothing. Special mention vs substandard explains the agreement measures in more detail.
Reading the results review by review
The scorer reads each replayed review the way the analyst will read live ones: last year's approved column, this year's column, every changed line and its source. A difference from the bank's original review falls into one of three categories, and the categories matter more than the count. The agent was right and the original review missed something; the analyst was right and the agent misread a document or a definition; or both are defensible and it is a judgment call. The first category is the value of the tool, the second is its error rate, and the third is ordinary credit work.
What the scorer reads for each replayed review
The leverage line is the kind of difference a replay should surface: the certificate says 2.88x, the agreement's definition gives 3.03x.
The grade comparison is read last, after the lines, because a grade difference usually traces back to one or two lines where the agent and the analyst read the evidence differently. The scorecard makes that visible: it shows each factor's score last year and now, the definition the proposal rests on and what would move it.
The proposed grade the pilot compares with the bank's own
What questions should you ask a credit review software vendor?
The questions that matter are the ones a pilot cannot answer: what happens to the bank's data, what changes without the bank's say, and how the bank gets out. The interagency third-party guidance is specific on the last point. Contracts should "protect the ability of the banking organization to change third parties when appropriate without undue restrictions, limitations, or cost", provide "for the timely return or destruction of the banking organization's data", and a bank "should also consider whether the contract includes burdensome upfront or termination fees" (88 FR 37920).
Beyond the contract, six questions separate tools quickly. Where does each figure in the draft come from, and can a reviewer open the page in one step? Are covenant definitions read from the agreement and its amendments, or typed in at set-up? What changes in the tool's behaviour without the bank changing a setting, and how is the bank told? Does anyone outside the bank see its borrowers' documents, and is any of that data used to train anything? Who can change the grading model and thresholds, and is every change versioned? And what does loan review see: the final review only, or every challenge, answer and override along the way?
Whether the tool counts as a model for model risk purposes is a question for the bank, not the vendor, and it turns on what the tool does. The revised interagency model risk guidance of April 2026 leaves generative and agentic AI outside its scope and excludes simple arithmetic and deterministic rules from its definition of a model, so a spreading and covenant engine and a grade-proposing scorecard may land in different places. Is financial spreading software a model? works through the same question for spreading.
Running beside the team, then deciding
A replay proves the reading; it does not prove the workflow. The second stage runs the solution on live reviews coming due while the analysts work the same reviews their usual way, so that hours, flags and grades can be compared on files nobody has seen before. Analysts keep signing their own reviews throughout; the pilot's output is advice until the bank decides otherwise. The plan below is an example; its length is the bank's choice.
An example pilot plan, in weeks
- Criteria, sample and set-upWk 1
- Replay and blind comparisonWks 2–3
- Live reviews beside the teamWks 4–5
- Score and decideWk 6
What a pilot settles, and what going live adds
The pilot: 25 completed reviews and a handful coming due.
Going live: Every relationship in the book, drafted ahead of its due date.
The pilot: Analysts sign their own reviews; the pilot output is advice.
Going live: Analysts sign the solution's drafted review after confirming or challenging each moved line.
The pilot: Traceability, covenant results, early warning, grade agreement and direction, hours.
Going live: The same measures, monthly, plus overdue reviews and grade migration.
The pilot: A blind scorer from credit or loan review.
Going live: Loan review, sampling signed reviews as it does today, with the evidence trail.
The decision is then a reading of the scorecard against the criteria, not a vote on a demonstration. If the replay flagged the moved lines on the loans the bank later downgraded, matched the bank's covenant recomputations, never altered a reported figure and did not lean generous on grades, and the live weeks met the bank's own hours baseline, the case is made in the bank's own numbers. If it failed a criterion, the reviews that failed say why. How long a commercial loan annual review takes shows how to turn the hours result into a return; the Annual Credit Review page shows the solution end to end.
Questions people ask
How many reviews does an annual review pilot need?
Enough to include each kind of loan where grading goes wrong. Twenty-five is a practical number for a replay; fewer leaves out whole categories, and many more mostly adds time.
Should loan review be involved?
Yes, as the scorer. Loan review already compares grades with an independent view, and its sign-off on the pilot's method makes the result credible. Annual credit review vs loan review explains why the two functions stay separate after go-live.
How long should a credit review software pilot take?
Long enough to replay the sample and to run live reviews beside the team for a few weeks; the example plan above is six weeks. If a pilot runs past a quarter without a decision, check first whether the criteria were written down.
How do you back-test risk grades?
Take grades assigned a year or more ago and compare them with what happened next: later downgrades, defaults, payoffs and loan review's regrades. For a pilot, the software's proposed grade on the old documents is set beside the bank's grade and the outcome, and every difference is classified as the tool right, the analyst right, or a judgment call.
What if the software disagrees with our analysts?
Then the pilot is working. Each disagreement goes into one of three categories, and only the ones where the agent misread the evidence count against it.