A credit risk rating model is whatever a bank uses to put a grade on a commercial loan and keep that grade right afterwards, and there are only four of them: the lender's judgement guided by a written scorecard, a statistical model estimated from default data, an independent review of a sample of files, and re-scoring every loan against the bank's own scorecard and its own look-alike borrowers. The reason the choice is confusing is that they are not really alternatives — three of the four sample, and the fourth does not decide anything. Botminds' Risk Rating Validation is the fourth, and this guide is written to let a chief credit officer judge where it fits beside the other three, what a pilot needs, and what it cannot do. The honest way to choose between any of them is the same: run last year's independently reviewed loans through the approach and count what it would have found before your reviewers did.
Examples come from a working demonstration on sample data: Kestrel Valley Bank, a fictional commercial lender with 2,305 rated loans and $13.67B of commitments. Regulatory material is quoted and linked where it appears.
What a credit risk rating model has to do
Three things, and no supervisor prescribes how. The OCC is unusually direct about this: it endorses no particular rating system, and asks only that every one of them measure both the obligor's ability and willingness to repay and the support that structure and collateral provide (Rating Credit Risk). That is the first job — measure the right thing. The second is reproducibility: the Basel framework requires that "written rating definitions must be clear and detailed enough to allow third parties to understand the assignment of ratings, such as internal audit or an equally independent function and supervisors, to replicate rating assignments and evaluate the appropriateness of the grade/pool assignments" (Basel Framework, CRE36).
The third job is the one the approaches differ on. A grade has to stay right between the day it is set and the day the loan matures, across different lenders, different regions and a changing economy. Measuring that is not a rating activity at all — it is a measurement of the rating activity, and it is where almost every bank's arrangement is thinnest.
Four approaches, one borrower: the same loan through each
Take one borrower from the demonstration. Ridgeback Freight Lines, an $18.5M commitment — a $12.0M term loan and a $6.5M revolver — graded 4 by its lender in Denver. The bank's own scorecard scores it 5.6, a grade 6. Its three closest matched peers, graded by other lenders in other regions, carry 7, 6 and 5, a median of 6. Its earnings are down 18%, its coverage sits below the bank's 1.25x guide, the guarantor statement on file is from June 2024, and the two-notch override that produced the grade 4 carries the reason code "none".
The same borrower through four approaches: what each one sees
What it would see on this loan: The scorecard says 6. The lender knows the sponsor, the history since 2014 and the contract pipeline, and overrides to 4.
What it would miss, and what it costs: Misses that three comparable borrowers elsewhere in the bank are graded worse by other people. Costs nothing extra; it is already how the grade was set.
What it would see on this loan: Produces a probability of default from the ratios. On these numbers it would land near the scorecard rather than near the grade.
What it would miss, and what it costs: Misses the override and the stale guarantor statement, because neither is an input. Costs a development and validation programme, and needs enough defaults to fit — which a $13.67B commercial book does not generate quickly.
What it would see on this loan: If this loan is in the sample, everything above plus a reviewer's own view of the credit, defended in writing.
What it would miss, and what it costs: Misses it entirely if it is not in the sample — 401 of 2,305 rated loans a quarter, about one in six. Costs reviewer time, and finds a drifting cell at the speed of the sample.
What it would see on this loan: The scorecard gap, the unexplained override, the stale statement, the three matched peers and their median, and the write-up scoring 34 of 100 — on every loan, not this one.
What it would miss, and what it costs: Decides nothing. It produces a proposal a credit officer must confirm, and it finds only what the bank has written down somewhere.
Read down the left column and the four approaches look like a progression. Read down the right and they look like layers, because each one misses a different thing. A bank running only the first has no second opinion at all. A bank running the first and third — which is most banks — has a second opinion on one loan in six. The fourth does not replace the third; it changes what the third spends its time on, by arriving with the comparisons already made.
The judgemental scorecard, and what it is good at
It is good at everything that is not written down. A lender knows that the borrower's largest customer renewed, that the founder's son now runs operations, that the guarantor sold a building. None of that is in a ratio, and a scale that cannot absorb it will be overridden until it is ignored. The Comptroller's Handbook expects a rating system to address the obligor's willingness as well as ability to repay, and willingness is a judgement.
What it is bad at is consistency across people. The same written definition, applied by two experienced lenders in two cities, routinely produces two grades — not because either is careless, but because the definition leaves room and the two have calibrated themselves on different books. That is what the demonstration's map measures: nine industries by five regions, 2,305 loans in 45 cells, four cells leaning more than half a notch from their matched peers and the widest leaning 1.2 notches.
The second weakness is the override. An override is a legitimate and necessary feature of a judgemental scale — the framework requires banks to articulate when officers may override, how far and by whom, and to track the overrides and their performance. An override without a reason recorded is a different thing: a year later nobody can say whether it was judgement or pressure. In the drifting cell of the demonstration, nineteen of twenty-two overrides carry no reason at all.
The statistical model, and the low-default problem it runs into
A fitted probability-of-default model is the most rigorous of the four and the least available to a mid-size commercial lender, for one reason: it needs defaults to learn from, and a well-run commercial book does not produce many. A book of 2,305 rated loans at a through-the-cycle rate near 1% yields on the order of twenty defaults a year, spread across nine industries. That is not enough to fit industry-specific behaviour, and industry behaviour is where the differences live.
The regulator's position accommodates this. For rating systems not tied to statistical probabilities, the OCC's expectation is only that credits with more severe ratings show higher defaults and losses — a rank-ordering test, not a calibration test. A bank that cannot build a statistical model is not thereby out of compliance; it simply has a lower claim to make and a different way of supporting it. What that test looks like in practice, and how many years it needs, is a page of its own.
There is also an adoption fact worth knowing before anyone assumes the market has moved past judgement. Reviewing the Federal Reserve's survey panel of banks lending to farmers between 1997 and 2002, Walraven and Barry found that adoption of risk rating held roughly flat over the period, and that a little under half the banks on the panel either used no rating system or reported one rating for every loan (FEDS working paper 2003-53). Two decades on, rating systems are near-universal, but the spread of sophistication behind them remains very wide.
Sampled review, and the arithmetic of what a sample cannot see
Credit risk review is the control the interagency guidance is written around, and the guidance is explicit about what a well-designed scope buys: "establishment of an appropriate review scope also helps ensure that the sample of loans selected for review, or portfolio segments selected for review, is representative of the portfolio as a whole and provides reasonable assurance that any credit quality deterioration or unfavorable trends are identified" (Interagency Guidance on Credit Risk Review Systems).
Representative of the portfolio as a whole is the operative phrase, and it is also the limitation. A sample designed to be representative of a book puts few observations in any one cell of it. The arithmetic below uses the sample bank's own fractions.
What a representative sample sees inside one industry in one region
The guidance also settles a question buyers ask early: the work does not have to be done in-house. "Additionally, some or all of the credit risk review function may be performed by a qualified third party." A tool that prepares the comparison is a lighter version of the same idea — the judgement stays inside the bank, and only the preparation moves.
Re-scoring every loan: coverage instead of judgement
The fourth approach trades depth for coverage deliberately. It does not form a view of a credit. It re-applies the bank's approved scorecard to every rated loan, puts every borrower beside like-for-like borrowers graded by other lenders, reads the write-up behind each grade against five fixed tests, and reports the gaps. On the sample bank's book that is 2,305 loans a night; on the flagged cell it is six agents finishing 38 loans, 61 grade changes, 142 matched peers and 38 write-ups in 94 seconds.
What changes is not the quality of any single judgement but the unit of analysis. A reviewer looking at seven files in a cell sees seven opinions. A map of 45 cells with a lean measured in each shows where to send the reviewer, and the reviewer arrives with the comparison already made. In the demonstration, that is the difference between five sampled downgrades in a quarter and a finding that names its cause: loans under $20M graded by the lender alone, nineteen of twenty-two overrides unexplained, and a factor missing from the scorecard for one industry.
Where a quarter's calibration days go, by hand and with the work done nightly
| The task | By hand, on a handful of loans | With the work done nightly |
|---|---|---|
| Pulling grades, overrides and reason codes | 3 days | 0.5 days |
| Re-spreading look-alike borrowers in other regions | 6 days | 0.5 days |
| Re-scoring on the bank's scorecard | 3 days | 0.5 days |
| Reading the write-ups behind each grade | 4 days | 1 day |
| Writing the calibration note for committee | 3 days | 1 day |
| Regional confirmation and sign-off | 1 day | 1 day |
| Total | 20 days | 4.5 days |
Nine industries by five regions: where Risk Rating Validation sends the reviewer first
The limitation is equally plain and should be stated by any vendor before it is discovered. Re-scoring finds disagreements between things the bank has written down. If the reason a grade is what it is was never written, the finding will be that the grade is unsupported — true, useful, and not the same as wrong. Two of the fourteen loans flagged in the demonstration are kept at their existing grades precisely because their write-ups do justify them.
What you bring: the five inputs that decide whether a pilot is worth running
Nothing here is unusual, and the state the data is in matters more than its volume. A bank that cannot export grade changes with the reason attached will get a thinner answer than one that can, and will learn something useful from the fact.
The five inputs, the form each has to be in, and who owns it
- Your loan system: grades, grade changes, overrides and reason codes
One row per grade change with who set it, when, the scorecard grade at the time and the reason. Owned by credit administration. If reason codes are mostly blank, that is the first finding, not a blocker.
- Financial spreads: borrower and guarantor statements on your ratio definitions
Enough history to show a trend, with statement dates. Owned by the spreading team. Missing statements are marked rather than estimated.
- Your approved risk rating scorecard, with weights and grade guides
The approved version, not a draft, and the version number. Owned by credit policy. This is what makes the re-score yours rather than a vendor's view.
- Credit memos: approvals, renewals, annual reviews and covenant reviews
The documents that set or affirmed each grade, in whatever form they are stored. Owned by credit administration. Scoring them needs the text, not a summary.
- Loan review and examination results, by region
Independent grade changes with dates and regions. Owned by credit risk review. This is the truth file a back-run is scored against.
How do you pilot it on loans you have already reviewed?
By running it backwards. Take the loans your own reviewers regraded last year, feed the approach the data as it stood before each regrade, and count four outcomes: the regrades it would have proposed earlier, the ones it agrees with at the same time, the ones it missed entirely, and the ones it would have flagged where the reviewer found nothing. That fourth number is the one vendors omit and the one that decides whether the output is usable, because a tool that flags everything has found nothing.
A back-run pilot, drawn to scale in working days
- Extracts agreed and delivered5 days
- Your scorecard and tolerances configured2 days
- Truth file assembled from last year's reviewer changes2 days
- Re-score and match over the closed period3 days
- Four outcomes counted against the truth file4 days
- Reviewers read a sample and disagree in writing5 days
- Thresholds tuned, then re-run3 days
Score the back-run on four counts and publish all four. The temptation is to report only the first — the regrades found earlier than the reviewers found them — and that number alone is unfalsifiable. Reported as a share of a whole, it becomes an argument.
The four outcomes of a back-run, and why all four belong in the result
Every reviewer regrade in the back-run period
- Would have been proposed earlier · 34%
- Agreed at the same time · 41%
- Missed — the reason was not in any file · 15%
- Reviewer found nothing where it flagged · 10%
Your scorecard or theirs?
Yours, and the answer should be verifiable rather than asserted. The re-scorer in this solution applies the bank's approved scorecard version factor by factor with its own weights and grade guides, and never changes either; the version in use is a setting, and in the demonstration it is v4.2, approved in January 2026, with a v4.3 draft that adds spot-rate exposure for truckload carriers going to validation before it is used.
The reason this matters beyond vendor independence is that a tool scoring against its own internal scale produces findings a bank cannot act on. A proposal to move a loan from grade 4 to grade 6 means something to a credit committee because grade 6 is defined in the bank's own policy. A proposal to move it from "medium" to "elevated" is a conversation about the vendor's taxonomy. Matching tolerances work the same way: coverage within 0.10x, leverage within 0.5x, revenue within 25% are settings in the demonstration precisely so that a credit policy committee can argue about them and then own them.
What to ask about configuration before a pilot, and what a good answer sounds like
| Question | A good answer | A warning sign |
|---|---|---|
| Whose scorecard does it apply? | Ours, by version, with the weights and grade guides we approved, and the version named on every run. | A proprietary score the vendor describes as equivalent to ours. |
| Who sets the matching tolerances? | We do, as settings, versioned and on the audit trail; the vendor supplies defaults. | Tolerances described as learned or adaptive, with no number we can state in a policy. |
| Can it write to our loan system? | No. It reads, proposes, and a person confirms. | Yes, with a toggle — which means the control is a setting rather than an architecture. |
| What happens when our officer disagrees? | The officer's grade stands, and the disagreement is kept and counted. | The disagreement is discarded, so the system can never be measured. |
| How do we know which version produced last quarter's finding? | Scorecard and threshold changes are versioned, and every step is on the audit trail with who, what and when. | Current settings only, with no history. |
What third-party risk management adds to the decision
A process, and a useful piece of proportionality. The interagency guidance on third-party relationships "outlines the third-party risk management life cycle and identifies risk management principles applicable to each stage of the life cycle", and "clarifies that not all third-party relationships present the same level of risk or criticality to a bank's operations" (OCC Bulletin 2023-17).
That second clarification is worth using. A read-only tool that prepares an analysis for a credit officer to confirm, with no write path to the system of record and no customer-facing function, is not the same risk as a core platform or an underwriting decision engine, and the diligence should be scaled accordingly. It is also not nothing: the data leaving the bank includes borrower financials and credit write-ups, which is a real confidentiality question whatever the criticality rating says. Scope the diligence to the actual data flow and the actual decision rights, and both conversations get shorter.
One more external datum belongs in a board paper on this subject, because it sets the scale of the grading problem outside your own book. In the 2025 Shared National Credit review, "U.S. banks hold 45 percent of all SNC commitments. However, they only hold 22 percent of non-pass loans, down slightly from the prior year" (2025 Shared National Credit report). Grading disagreements on shared credits are visible across institutions in a way they are not inside a single book — which is a reminder that the grades a bank sets are read by people other than its own committee.
Where does that leave the choice? Layer, do not substitute. Keep the lender's judgement and the written scorecard; keep independent review, because somebody has to form a view of a credit; add a statistical model if and when the defaults exist to fit one; and instrument the whole thing so that drift is a measurement rather than a discovery. To see the fourth layer on one book — the map, the comparisons, the write-up scores and the quarterly note — start from Risk Rating Validation; for what it costs to get a grade wrong in the meantime, read what one notch is worth.
Questions people ask
What is a credit risk rating model?
Whatever a bank uses to assign a commercial loan its grade and keep that grade accurate: in practice a written scale, a scorecard of weighted factors, an override path, and some arrangement for checking the result. The OCC does not advocate any particular system, but expects every one of them to address the obligor's ability and willingness to repay and the support given by structure and collateral. Whether a given scorecard counts as a "model" for model risk purposes is a separate and narrower question.
Should a bank use a statistical model or a judgemental scorecard?
Both, where the data allows, and a judgemental scorecard where it does not. A statistical model needs defaults to fit, and a commercial book of a few billion dollars produces on the order of twenty a year — too few to fit industry-specific behaviour. A judgemental scale with written definitions and a disciplined override is a legitimate answer; the regulator's test for it is lower but firm, that worse grades show higher defaults and losses.
What data do you need to start validating risk ratings?
Five things: grade changes with their overrides and reason codes, financial spreads on your own ratio definitions with statement dates, the approved scorecard with weights and grade guides, the credit memos that set or affirmed the grades, and your own loan review results by region. The fifth is the one that converts a demonstration into a measurement, because it is the truth file a back-run can be scored against.
How do you pilot a credit risk rating model?
Backwards, on loans your own reviewers already regraded. Feed the approach the data as it stood before each change and count four outcomes: proposed earlier, agreed at the same time, missed, and flagged where the reviewer found nothing. Publish all four with the denominator. A pilot that reports only the regrades found early has reported a numerator.
Does it need to use our own scorecard?
It should, and you should be able to see which version each run used. A tool scoring against a proprietary scale produces proposals a credit committee cannot act on, because the committee's authority, pricing and allowance all key off the bank's own grades. In this solution the approved version is a setting, applied factor by factor with the bank's own weights and grade guides, and never altered by the tool.
Will it write to our loan system?
No — and that is a design choice worth insisting on from any vendor. The agents read the loan system and never write to it; they propose regrades in a quarterly note, a regional credit officer confirms or changes each one, the Chief Credit Officer signs, and nothing moves until the committee approves. The worst outcome of a wrong recommendation is then a recommendation somebody declines.