AI for risk management is only as useful as the errors it can be shown to catch, and the way to show it is a seeded test: take a period whose answers you know, plant errors you chose, run the tool without telling it where they are, and count. In a fund's liquidity risk management program under SEC Rule 22e-4, where every holding is classified into one of four liquidity buckets each month and a wrong bucket can hide a breach of the 15% illiquid limit (17 CFR 270.22e-4), forty planted errors covering the ten ways classifications actually go wrong are enough to tell a second line of defence whether to rely on the tool, and where not to. The work is hard because a classification that is wrong usually looks reasonable. Botminds' Liquidity Risk Management Program is an agentic solution whose Checker agent re-tests every proposal and leaves each catch on record, which is what makes this test countable.
The design below is written for a risk, compliance or internal audit team that has to sign off before go-live. The error types come from the rule, the SEC's 2024 guidance, the staff FAQ and the first liquidity rule case the SEC brought. The fund group and its numbers are fictional; the pass marks are our suggestions, not measured results.
The 40 planted errors: ten ways a liquidity bucket goes wrong
- Currency behind capital controls left highly liquid (5)
Tests the 2024 reading that cash means US dollars, and that a currency which cannot reach dollars in seven calendar days, and what sells into it, is illiquid.
- Local-settlement bond classed by days to sell, not days to dollars (5)
Tests the gap between moderately and less liquid: sold in six days, cash in twelve.
- Restricted private placement or lock-up called less liquid (5)
Tests whether transfer terms in the documents override a market-data bucket.
- Halted or suspended security left in its old bucket (4)
Tests whether an exchange or trustee notice reaches the classification.
- Stale bucket after an intra-month trigger (5)
Tests the 2024 guidance on reviewing classifications intra-month when markets move.
- Same security, two buckets, no reason (4)
Allowed with a reason under the staff FAQ; planted here without one.
- Position enlarged, classified at the old trade size (4)
Tests the rule’s trade-size test and the 2024 note on enlarged positions.
- Day-count or market-holiday error (3)
Business against calendar days, and a long foreign closure left out.
- Highly liquid collateral pledged against less liquid derivatives, not identified (3)
Tests the pledged-asset rule behind the minimum.
- Limit arithmetic wrong (2)
A liability counted in the illiquid share, or the share taken on gross rather than net assets.
Why does a clean test run prove nothing?
Because a month with no visible errors is what both a good tool and a blind one produce. If an AI tool reviews last month's buckets and raises nothing, you have learned that it agrees with last month's process, which is the thing you were trying to check. Agreement with the incumbent is a measure of consistency, not of accuracy. The only way to know whether the tool catches an error is to know where errors are, and the only way to know that is to put them there.
Liquidity classification makes this sharper than most risk work. A wrong bucket rarely breaks anything visible: the month closes, N-PORT is filed, nobody redeems. The error surfaces only when the holding has to be sold, or when an examiner reads the file. The dismissed Pinnacle case shows the shape. In a settled order, whose findings the trustee neither admitted nor denied, the SEC found that a restricted private placement worth about 23% of net assets was classified "less liquid" while counsel and auditors said otherwise (Release IA-6303). A month-end process would have run cleanly every month.
How do you test an AI classifier before go-live?
Use a month you have already closed, plant the errors in a copy, and keep the answer key with someone outside the team running the tool. Five steps are enough. First, pick a past month-end with a full set of approved buckets, overrides and source documents, ideally one with a market event in it. Second, choose the errors, forty here, and where each goes: which fund, which holding, which field. Third, plant them in the copy, changing only what a real error would change. Fourth, run the agents on the copy exactly as they would run live, with the same settings. Fifth, score every catch, miss and false alarm against the key.
The SEC staff already expect testing of automated classification. A fund may rely on an automated framework for asset-class exceptions without justifying each security, but "would conduct periodic testing of the framework and processes" (SEC staff FAQ, Q17). The staff's answer is about exceptions; a seeded test is the most direct way to do that testing for any classifier, and its results are a work paper the program can hand to the board.
A seeded test takes about three weeks, most of it preparing the answer key
- Pick the month, copy the data2 d
- Design and plant 40 errors; seal the key5 d
- Blind run1 d
- Score catches, misses, false alarms3 d
- Read the misses; fix; rerun4 d
What errors should a seeded test include?
The errors that examiners and the SEC's own guidance say happen, in proportion to the harm they can do. The ten types above each trace to a text. The currency errors come from the September 2024 guidance, which reported funds "classifying any currency as a highly liquid investment, regardless of the amount of time it would take to convert that currency to U.S. dollars", and said that where a currency cannot be converted into dollars within seven calendar days "because of currency controls or otherwise", the currency, and investments sold for it, should be illiquid (89 FR 73764). The stale-bucket errors come from the same release, which found funds "not prepared to review classifications intra-month".
The trade-size errors test the rule's requirement to consider "sizes that the fund would reasonably anticipate trading" (Rule 22e-4(b)(1)(ii)(B)) and the 2024 note that a fund which "substantially increases the size of its position" may need to reclassify it. The cross-fund errors are planted without a reason because the staff allow different buckets for one security across funds, but the program has to know why (Q5). The collateral errors test the rule's requirement to identify highly liquid assets pledged against derivatives classified below highly liquid, which feeds the minimum. Two errors in the limit arithmetic test "illiquid investments that are assets": a liability counted in the share is a quiet way to misstate it.
What forty planted errors can and cannot prove
40
planted errors across ten types, in one copied month
Test design
40/40
a perfect score shows only that the true catch rate is at least 92.8%, at 95% confidence
83%
the lower bound if 38 of 40 are caught (95% confidence)
59
errors needed, all caught, to show a catch rate above 95%
Running the test blind
Blind means the tool and its operators cannot tell planted from real. Plant errors the way they arise: change the bucket and the reason field together, not a value nobody would type; leave the source documents as they are, so a planted "less liquid" restricted share still sits beside a lock-up that says otherwise; for the stale-bucket type, add the market event to the data and leave the buckets unchanged. Mix in real holdings that look odd but are right, such as a bond that is less liquid in one fund and moderately liquid in another for a stated size reason, so false alarms can be counted too.
Run the agents with the program's own settings. In the sample settings of Botminds' Liquidity Risk Management Program that means a 10% trade size, value impact by asset class, a 90% confidence threshold for auto-confirming an unchanged bucket, and intra-month triggers on a spread move above 150 basis points or dealer quotes down by more than half. A test run on looser or tighter settings than production tests a different tool.
What Liquidity Risk Management Program's Checker agent raised in one sample cycle
Scoring: catches, misses, false alarms and late catches
Score every planted error and every challenge, and give each a time. A catch counts only if the tool flags the right holding for the right reason; a challenge on the correct holding for an unrelated reason is a lucky hit, and it will not repeat.
How to score a seeded liquidity bucket test
| Outcome | What it means | Why it matters |
|---|---|---|
| Catch | Planted error flagged, right holding, right reason | The tool finds this type; check it finds every instance |
| Miss | Planted error not flagged, or flagged for the wrong reason | The type the program cannot yet rely on; read every one |
| False alarm | A correct holding challenged | Cost: analyst time. Too many and people stop reading challenges |
| Late catch | Caught, but after the date a real breach would have needed reporting | For limit-critical types, a late catch is a miss |
| Wrong fix | Caught, but the proposed bucket is still wrong | The tool sees the problem but not the answer; keep a person on that type |
What catch rate is good enough?
It depends on what the error would hide. For the five types that can move a fund across the 15% limit or its minimum, currency controls, restricted holdings, suspensions, pledged collateral and limit arithmetic, the pass mark should be every planted error caught, on time. For the types that misplace a holding between the middle buckets, a high but imperfect rate can be acceptable if the misses are understood and a person reviews that type by hand.
Suggested pass marks by error type: all-or-nothing where a limit is at stake
Be honest about what the numbers mean. With forty planted errors, a perfect score shows, at 95% confidence, that the true catch rate is at least about 93%; to show it is above 95% you would need fifty-nine planted errors and no misses. That is why the test is per type and per miss, not a single percentage for a slide. A tool that catches 38 of 40 has not failed if the two misses are day-count errors it now handles; it has failed if one was the restricted private placement.
A seeded test is not there to produce a percentage. It is there to find the one type of error the program cannot yet rely on, before a redemption or an examiner finds it.
Reading the misses
Every miss has a cause, and there are only a few. The data was missing: the tool never saw the trustee's notice or the lock-up. The rule was missing: nothing in the settings told it a closure longer than seven days matters. The threshold was wrong: the confidence or trigger level let a weak proposal through. Or the reasoning was wrong: it had the facts and drew the wrong bucket. The first three are fixed in the program and rerun; the fourth means that error type stays with a person until a later test shows otherwise.
Read false alarms the same way. The SEC staff recognise that "more sensitive exception methodologies may result in numerous false positives" (Q16); a checker that challenges a third of all holdings will be ignored within a month. The useful measure is false alarms per hundred holdings reviewed, set against the analyst time available. In the solution's sample month, the Checker agent raised 46 challenges on 60,418 holdings, fewer than one in a thousand; in a seeded test you can see how many of those are noise.
Keeping the test alive after go-live
Run it again whenever the tool, its settings or the market changes, and at least once a year for the annual report to the board. The rule asks the administrator for a written report that "assesses its adequacy and effectiveness of implementation" (Rule 22e-4(b)(2)(iii)); a seeded test with its catches and misses is evidence of effectiveness that a list of events is not. It also gives the board a direct answer to the question the SEC's examiners are now asking about firms' use of "super intelligence (SI)": how the technologies "influence outcomes" and whether firms still meet their existing, technology-neutral obligations (SEC, FY2027 examination priorities).
Between tests, real sales are a running check. In Liquidity Risk Management Program the Checker agent back-tests each month's actual sales from the order system against the bucket each holding carried; in the sample September data, 158 of 164 sales fell within their bucket. A back-test cannot find the restricted holding nobody tried to sell, which is why the seeded test is still needed. How the agents divide the program's work, and what stays with people, is in AI tools for mutual funds' liquidity programs; the history behind the restricted-holding type is in SEC v. Pinnacle Advisors.
An intra-month review in Liquidity Risk Management Program: trigger, re-test, change
Questions people ask
How is AI used in risk management at fund managers?
Mostly for gathering and checking: pulling market and position data, computing measures such as days to liquidate, reading documents and drafting reports. The SEC's fiscal 2027 priorities describe registrants using generative "super intelligence (SI)" tools for research, drafting and "compliance monitoring and surveillance", and say examiners will look at how those tools influence outcomes.
How many errors should a seeded test include?
Enough to cover every error type at least twice and the limit-critical types several times; forty across ten types is a practical size for one month. To certify a catch rate above 95% at 95% confidence you would need at least fifty-nine planted errors and no misses, so most programs test per type instead of chasing one percentage.
Who should plant the errors?
Someone independent of the team operating the tool: second-line risk, model validation or internal audit. They keep the answer key sealed until the run is scored. The SEC's 2021 risk alert already expected oversight of third parties that supply liquidity classifications; the same independence applies to an internal tool.
Does a seeded test replace back-testing against real sales?
No. Back-testing shows whether holdings that were sold behaved like their buckets; it says nothing about holdings that were never sold, which is where restricted and suspended securities sit. A seeded test covers those. The two together give the board evidence on both.
Are liquidity buckets the same as days-to-liquidate bands?
Not quite. Rule 22e-4's four buckets combine days with value impact and, for the last two, settlement: a holding sold in six days but settled in twelve is less liquid, not moderately liquid. The SEC proposed in 2022 to fold the less liquid bucket into illiquid, but did not adopt the change, so all four remain.