Credit rating scorecard: back-test it on issuers you already rated

A credit rating scorecard is the quantitative core of a rating methodology: a fixed set of factors such as scale, leverage, coverage and liquidity, each scored against thresholds for every rating grade and combined with set weights into a single score that maps to an implied rating. Back-testing a scorecard means replaying it on issuers you have already rated, with the information you had at the time, and comparing its answers with the ratings your committee signed and with what later happened to those issuers. It is the one test that shows whether the weights rank risk the way the methodology claims, and EU money market fund rules require it. Botminds' Internal Credit Rating applies your scorecard only under approved, versioned methodologies and keeps every past file with its inputs, so a replay is a query rather than a project.

The examples use sample data on fictional firms: Quillfield Capital's Corporate Methodology v4.2 for transportation issuers, and a proposed v4.3 replayed across 61 transportation names. The regulatory texts are quoted from the source and linked.

What is a credit rating scorecard?

It is the part of a methodology that can be computed. Each factor has a measure, a weight and a table of thresholds that place the measure in a rating band; the weighted average of the factor scores maps to an implied rating, and the analyst then notches for what the scorecard cannot see. Rating agencies publish their methodologies; the SEC's staff notes it is "standard practice for NRSROs to disclose their methodologies for determining credit ratings on their websites" (2025 Staff Report). A buy-side firm writes its own, usually one per sector, and the weights are its view of what drives default in that sector.

Two things in that table are judgments disguised as arithmetic. Leverage carries a quarter of the weight, so a 0.5x error in the debt figure can move the implied rating; and business profile is scored from market position, which is exactly where an analyst might count a risk twice. The sample methodology forbids that: customer concentration may be an adjustment or part of business profile, not both.

What is a model-implied rating, and why it must come first

A model-implied rating is the rating the scorecard produces before any analyst judgment: the starting point every notch is measured from. It matters because without it a qualitative adjustment has nothing to adjust. The SEC's examiners classed as a material regulatory deficiency one rating agency's practice of applying a qualitative adjustment without first producing the model-implied rating its own methodology called for (SEC Office of Credit Ratings, 2025 Staff Report).

For back-testing, the implied rating is the thing being tested. If the analyst's notches are always in one direction, the scorecard is mis-weighted and the analysts are correcting it by hand, which is a methodology problem hidden as judgment. In the sample, Orrin's scorecard gives BB, the analyst notches once for a 38% customer concentration, and leaves an owned-fleet notch unapplied because the liquidity score already reflects it: the file shows each move and the page behind it.

What do the rules require for back-testing a rating methodology?

The clearest rule is European. An EU money market fund's methodologies "shall be subject to validation ... based on historical experience and empirical evidence, including back testing" (Regulation (EU) 2017/1131, Article 19(3)), and the Commission's delegated regulation says what validation means. The methodology passes only if, among other criteria, "past internal credit quality assessments produced using the internal credit quality assessment methodologies have been properly reviewed ... to determine whether the credit quality assessment methodologies are a suitable indicator of credit quality" (Delegated Regulation (EU) 2018/990, Article 3(1)(d)).

The same article adds three requirements a buy-side firm should borrow even if it runs no money market fund. Validation must "assess the sensitivity of the methodologies to changes in any of their underlying credit quality assumptions and criteria" (Article 3(2)). Firms need processes to ensure "any anomalies or deficiencies highlighted by the back testing ... are identified and appropriately addressed" (Article 3(3)). And the methodology must "make it possible to compare past internal credit quality assessments" (Article 3(4)(c)): if you cannot replay an old file, you cannot back-test.

The internal credit quality assessment methodologies ... shall make it possible to compare past internal credit quality assessments.

Delegated Regulation (EU) 2018/990, Article 3(4)(c)

Rating agencies work to the same standard. EU-registered agencies must "use rating methodologies that are rigorous, systematic, continuous and subject to validation based on historical experience, including back-testing" (Regulation (EC) 1060/2009, Article 8(3)). The SEC's internal control rule lists controls to ensure "quantitative models incorporated into in-use credit rating methodologies are periodically reviewed and back-tested" (17 CFR 240.17g-8(d)(1)(vi)). Banks have the Basel equivalent, comparing realised default rates with each grade's estimate every year; back-testing a bank's own grades against default rates covers that version.

How to back-test a credit rating model on issuers you already rated

Replay, compare, then explain. The replay must be point in time: the scorecard sees only the statements, ratings and events available on the day each file was signed, or the test flatters itself with hindsight. The comparison has three targets, and each answers a different question.

Read the measures together. A scorecard that agrees with nearly every signed rating but ranks later downgrades badly is reproducing your analysts' habits, not predicting risk. One that ranks outcomes well but agrees with only half the signed ratings suggests the analysts are overriding a better tool, and the override reasons deserve a look.

How do you test a scorecard against defaults when there are almost none?

Use more outcomes than defaults, and borrow the agencies' history. An investment-grade book can go years without a default, so a back-test that counts only defaults proves nothing. Agency downgrades, distressed exchanges, your own later downgrades and spread widening are all observable outcomes, and the agencies' rating histories are public: every registered agency must disclose free on its website the ratings it assigned and "any subsequent upgrade or downgrade of the credit rating (including a downgrade to, or assignment of, default), and a withdrawal of the credit rating" (17 CFR 240.17g-7(b)).

Bank supervisors faced the same problem and accept a second-best target. For a shadow rating model, one built to explain agency ratings rather than observed defaults, the ECB expects a firm to show "how the model performs on the application population in terms of predicting defaults or, if that is not possible (when there are not enough internal default data), at least in terms of predicting the target ratings" (ECB guide to internal models, July 2025, paragraph 224). Predicting agency ratings is a weaker test than predicting defaults, but it is a test.

How close should a scorecard get to the agencies? A classic Federal Reserve Bank of New York study is a useful benchmark. Regressing the average of Moody's and S&P sovereign ratings on eight public variables "explains more than 90 percent of the sample variation and yields a residual standard error of about 1.2 rating notches" (Cantor and Packer, FRBNY Economic Policy Review, October 1996). A good corporate scorecard should explain most of the agencies' ranking, too; the residual of a notch or so is where the judgment lives, and where a two-notch split from the agencies needs its written reason.

Replaying a methodology change before anyone approves it

A methodology change is a back-test in reverse: run the proposed version over the book you hold today and see who moves. The EU rules treat a revision as serious. When methodologies, models or key assumptions change, the manager must "review all affected internal credit quality assessments as soon as possible" (Article 19(4)(f)), and a revision counts as a material change "except if managers of MMFs can substantiate that the change is not material" (Delegated Regulation (EU) 2018/990, Article 8(5)).

Nine moves out of 61, none of more than one notch, is a result a committee can discuss name by name: is each move what the change intended? A replay that moved thirty names by two notches would be a different methodology, not a revision. In the solution a proposed version is re-run on the book first, validated by the model risk team and approved by the CRO, never by the person who proposed it.

Where the scorecard's inputs come from: the spread

A back-test is only as honest as the inputs it replays. If last year's leverage was keyed by hand from a PDF and the source page is lost, the replay inherits the error and nobody can find it. In the solution every spread value keeps its page, its confidence and the original wording beside the English, so a replay can show not only the score but where each number came from.

A spread whose every value can be traced, in any language

The scorecard as the committee sees it, beside the agencies

How often should a rating methodology be reviewed?

At least once a year, and whenever a back-test or a market change says so. EU money market fund managers must review their credit quality assessment methodologies "at least annually" and send the review to their competent authority (Article 19(4)(e)). EU rating agencies review "credit ratings and methodologies on an ongoing basis and at least annually" (Article 8(5)). The SEC's rule asks agencies to consider controls ensuring performance measures "are used to evaluate the performance of credit ratings as part of the review of in-use methodologies" (Rule 17g-8(d)(1)(ix)), which is a back-test by another name.

An annual back-test is also the cheapest way to prepare for the annual review a regulator may ask to see. In the sample, the Central Bank of Ireland's request for the annual review of the internal credit quality assessment procedure of a euro liquidity fund sits beside the SEC and NAIC requests, mapped to the files and methodology versions it covers. Running your methodology as versioned settings explains how the versions are kept.

Questions people ask

Is a credit rating scorecard the same as a credit score?

No. A consumer credit score is a statistical score of an individual's likelihood of repaying. A credit rating scorecard rates an issuer or a bond: it combines weighted financial and business factors into an implied rating on a letter scale, which an analyst then adjusts and a committee approves.

How do you validate an internal rating model?

Check that it is applied the same way to every issuer, that its inputs are reliable, that past assessments were reviewed against what happened, that its sensitivity to its assumptions is known, and that development and approval allowed real challenge. Those are close to the EU's criteria for money market fund methodologies in Delegated Regulation 2018/990, Article 3.

How many issuers do you need to back-test a scorecard?

Enough closed files per sector to see a pattern in the overrides and outcomes, which usually means several years of files. Where defaults are too few, test against agency downgrades and the agencies' ratings themselves, as the ECB accepts for shadow rating models when internal default data are lacking.

What does EU money market fund regulation require for credit assessment?

A prudent internal credit quality assessment procedure with systematic, continuous methodologies validated by back testing; a review of every assessment at least yearly and on any material change; documented rationale, histories and responsible persons kept for three complete annual accounting periods; and approval by senior management and the governing body (Regulation (EU) 2017/1131, Articles 19 to 23).

What should happen when a back-test fails?

Fix the methodology, then re-run the affected ratings. The EU rules require a methodology to be "promptly improved" when a review shows it does not ensure systematic assessment, and every affected assessment to be reviewed as soon as possible after a change.