Backtesting compliance rules: an example on twelve months of holdings

Backtesting compliance rules means running a coded investment restriction over historical holdings, day by day, and listing the days it would have flagged. A rule is right only if it fires on the days the clause would really have been breached and stays silent on the rest; a back-test is the cheapest way to see both, before the rule goes live. Run under each plausible reading of an ambiguous clause, it does something no amount of debate can: it shows how far apart the readings are in practice, and which holdings make the difference. Botminds' Compliance Rule Coding runs a 12-month back-test of every reading of every rule, with edge cases built from the clause's own words, and gives the result to the rule analyst who decides which reading the engine will run.

The example comes from the solution's fictional sample: a German pension fund's segregated global equity mandate managed by a fictional asset manager. Regulator findings are linked where they appear.

What is backtesting compliance rules? The idea, explained

A back-test treats a coded rule as a hypothesis about a clause and history as the evidence. For each past business day it takes the holdings, prices and classifications as they stood that day, computes the rule's numerator and denominator, and records whether the limit or the warning level was crossed. The output is a time series and a list of days; the useful output is the explanation of each day.

It answers questions a test on today's portfolio cannot. Would the rule have fired on a stress day, a month-end rebalance, a large redemption? Does it fire on days the portfolio manager would say were fine, which teaches people to ignore it? And when two readings of a clause are possible, how often do they disagree, and on what? A rule that looks right on one portfolio snapshot can be wrong on fifty days of the year.

A backtesting compliance rules example: one clause, two readings, 251 days

The clause is from a German side letter to an investment management agreement. It caps securities of "Emittenten mit Sitz in Schwellenländern", issuers with their seat in emerging markets, at 10% of the portfolio's market value. Read literally, "Sitz" is the registered seat: the issuer's domicile, its country of incorporation. But the side letter's first clause gives its terms the meaning in the agreement, and the agreement defines an Emerging Market Issuer by country of risk: principal place of business, primary listing and majority of revenue. Two agents read it two ways: the Coding agent literally, at confidence 0.71; the Reading agent through the definitions, at 0.93.

The new account had no history of its own, so the Tester agent replayed both readings on the strategy's representative portfolio for 251 business days, from 20 October 2025 to 16 October 2026. Under the literal reading the rule would have been over 10% on three days in March 2026. Under the defined term it peaked at 9.1%, on 12 March.

What the count of days shows, and what it does not

The count does not say which reading is right. That is a legal question, and on this clause the answer was in the documents: the side letter imports the agreement's definitions, so the defined term governs. What the count shows is the stake. Three days a year is not a rounding difference; under the literal rule the engine would have blocked trades and raised alerts in March on holdings the client's own definition says are not emerging-market issuers.

The holdings behind the difference are the most useful part of the result. Four holdings separate the readings. Two are incorporated in the Cayman Islands with China as their country of risk: emerging under the defined term, invisible to the literal one. Two run the other way: one domiciled in Mexico and one in South Africa, both with country of risk outside emerging markets. On the March days the Mexican-domiciled holding was large enough to push the literal count over the limit. A coder who sees those four lines understands the clause; one who sees only "3 days" does not.

The breach days, the holdings that explain them, and the analyst's decision

What data do you need to back-test investment restrictions?

You need the portfolio as it was each day, and the world as it was described each day. The second half is the one most back-tests get wrong. A rule that tests country of risk must use the country of risk recorded on each past date, not today's; a rule that tests an emerging-markets classification must use the benchmark's classification on that date, because markets move between categories. The sample rule's edge cases included exactly that: a market reclassified from emerging to developed mid-year, tested on a point-in-time classification.

Edge cases: the back-test's partner

A year of history tests the cases the portfolio happened to contain. Edge cases test the ones it did not. They are small, made-up portfolios built from the clause's own words: exactly at the limit, a hair above it, each exception, each defined term, each data gap. The sample rule passed eight of them, and each maps to a sentence in the documents.

Investment compliance testing framework: three layers, one rule

An investment compliance testing framework for coded rules has three layers, and each answers a different question. Edge cases ask whether the rule implements the clause as read. The back-test asks what the reading means on real holdings, and whether a different reading would behave differently. The dry-run in the engine's test environment asks whether the engine runs what the paper says, and how the new rules sit beside the live ones. On the sample mandate the dry-run compiled 40 engine rules and one manual control, found no breaches on the transition portfolio, and raised one warning: the emerging-markets rule at 8.7% against its 8.5% warning level.

The framework also needs a fourth, outside the rule: comparing the engine's alerts with someone else's. The AMF asked managers to reconcile their own alerts with those of their funds' depositaries, and listed regular formal reconciliations as good practice; where the two disagreed, managers traced it to configuration errors or to differences of interpretation (AMF SPOT summary, February 2025, sections 4.3.4 and 4.3.6). The AMF also stressed the good practice of a statistical analysis of trends in breaches, to catch problems in the quality of ratio configuration early (AMF news release, 19 February 2025).

How often should coded rules be re-tested?

Every time something they depend on changes, and in full at least once a year. Rules depend on more than their own text: an amendment to the agreement, a new exclusion list, a benchmark that moves a country from emerging to developed, a vendor that changes its country-of-risk method. Each can change what a rule does without anyone touching the rule.

On the sample firm, a benchmark provider's announcement that one market would move from emerging to developed touched 64 rules in 118 portfolios. The Tester agent re-ran all of them overnight: 61 results unchanged, 3 that change, one of them a fund that would fall below its 85% emerging-markets name policy on the effective date. The triggers that should re-test a rule are set out on their own page.

What does internal audit ask about coded rules?

Internal audit tests whether the coding control works, which means it asks for a sample of rules and, for each, the path from clause to production: the source text, the reading chosen and by whom, the checker's signature, the tests run and their results, and the load. It also asks the reverse: for a sample of live rules, can each be traced to a clause, or is it a rule nobody can explain?

Regulators ask the same of breach investigations, and the records are often missing. Testing samples of 15 to 20 overruns at each of five managers, the AMF found the evidence of how the control systems identified them "was not retained by AMCs 3 and 4 for the entire sample tested", and was incomplete at two others (AMF SPOT summary, section 4.3.6). In the United States, the SEC's examiners have observed advisers whose annual compliance reviews included testing but who did not keep the documentation of it (SEC risk alert, September 14, 2026), and the books-and-records rule requires advisers to keep records documenting their annual review (17 CFR 275.204-2(a)(17)(ii)).

Backtesting compliance rules as a PDF: what the evidence should contain

The back-test that convinces an auditor is not a chart; it is a record. For each rule, it should state the clause and both readings, the portfolio and period replayed, the data sources and their as-of dates, the days each reading crossed the limit or the warning, the holdings that drove each day, the edge cases with results, and who decided what. Whether it lands as a PDF, a CSV or both matters less than whether it can be shown to be unchanged since it was produced.

The solution assembles that record from its audit trail. One export for a rule or a mandate gathers the source clauses in the original language and English, both readings with confidence and citations, the analyst's decisions with reason codes, the checker's sign-offs, the edge cases and 12-month back-tests, the engine dry-run and load results, and the agent and model versions used, as a PDF report, a CSV event log and a JSON bundle, with every file listed in a SHA-256 manifest.

Each element of a rule cited, and the back-test beside it

One case shows what a back-test at the moment a limit is adopted would reveal. When the board of four collective investment trusts adopted a 10% single-issuer limit in May 2016, the trusts already held between 30% and 89% in one company at the end of the previous year, and every year-end and mid-year figure the SEC's order reports until late 2018 was above the limit (SEC order, Advisers Act Release 5416, December 10, 2019). A rule that fails on day one needs a remediation plan with dates, not an alert; what that case cost is set out elsewhere. To see both readings replayed on sample mandates, start from Compliance Rule Coding.

Questions people ask

How long a history should a compliance back-test use?

Long enough to include the portfolio's normal range of positions and at least one period of stress or rebalancing; twelve months of business days is the solution's default setting. Shorter windows miss seasonal effects such as month-end flows; much longer ones mix in strategies the portfolio no longer follows.

Can you back-test a rule for a brand-new mandate?

Yes, on a representative portfolio of the same strategy, which is what the sample mandate used because the account had no history. The result shows how the rule behaves on that strategy's typical holdings; it should be re-run on the account's own holdings once a few months exist.

What if both readings of a clause give the same back-test result?

Then the difference does not matter on the holdings you have, but it may on the ones you buy next. Record the decision anyway, and keep an edge case that separates the readings, so the rule is tested on the case even if history never produced it.

Is a back-test required by regulation?

No regulator prescribes back-testing of coded investment rules. The obligation is broader: procedures that can be shown to prevent breaches, and a yearly test that they work. A back-test is one of the most direct pieces of evidence that a coded rule does what its clause says.

Who should decide between two back-tested readings?

The rule analyst, with a reason, and where the clause is a client's, often with the client's confirmation. In the sample, the analyst chose the defined term with the reason code "Defined term governs", and the agents drafted a one-line confirmation to the client in German and English.