SystemInvestment Operations & Middle Office

Evaluate · testing before trusting

AI for settlement exceptions: test it with seeded breaks first

Before relying on AI for settlement exceptions, plant exceptions with known answers in a real month of your own queue and count what the agents catch, mislabel, miss and escalate. No regulator publishes a pass mark, so set yours first. This guide gives the seed list, the scorecard and the arithmetic, and shows how Failed Trade Management is tested this way.

Updated · 12 min read

See one case, screen by screen →
Watch with transcript →

The way to test AI for settlement exceptions is a seeded-error test: take a past month of your own settlement exceptions, plant a set of exceptions whose correct cause and correct action you have written down in advance, such as a stale standing settlement instruction, a quantity booked in units instead of face amount, a counterparty that never instructed, a forgotten hold, a look-alike email asking to change an account, and run the agents over the whole queue. Then count four things for each planted item: caught and correctly labelled, caught but mislabelled, missed, and escalated to a person. A test that cannot fail is not a test, and nobody outside the firm will set the bar: no regulator or independent study publishes an accuracy figure for settlement exception handling, so the pass mark has to be agreed before the pilot starts. Botminds' Failed Trade Management is built to be tested this way, and its first-time setup ends with a run on the firm's own past fails.

The causes to seed come from what market participants told ESMA and from the European Central Bank's settlement rules, linked where they appear. Examples of the agents come from the solution's fictional sample firm, Quillfield Capital; the scorecard is a template, not a result.

A seeded-error scorecard: the bar each planted exception must clear

Seeded exceptionPass looks likeA fail looks likeTolerance

Stale SSI: counterparty changed account by email

Pass looks likeCaught and routed to the SSI team

A fail looks likeMissed, or fixed by editing the SSI

ToleranceZero failures

Look-alike sender asking for a new account

Pass looks likeFlagged as possible fraud; nothing changed

A fail looks likeAny action on the request

ToleranceZero failures

Quantity in units instead of face amount

Pass looks likeCause named as quantity type

A fail looks likeLabelled as lack of securities

ToleranceAgreed pass mark

Counterparty never instructed

Pass looks likeChaser drafted to the right contact

A fail looks likeRe-instruction drafted on our side

ToleranceAgreed pass mark

Own instruction left on hold

Pass looks likeHold owner named; release proposed

A fail looks likeChaser sent to the counterparty

ToleranceAgreed pass mark

Partial offer that disagrees with the instruction

Pass looks likeEscalated with both readings

A fail looks likePartial accepted in the draft without a flag

ToleranceZero failures

Message that cannot be linked

Pass looks likeWaits for a person with candidates

A fail looks likeLinked to the wrong trade

ToleranceZero failures

Our template. For each seeded type, record how many were planted, caught and correctly labelled, caught but mislabelled, missed and escalated. 'Strict' rows allow no failures at all in the test; 'firm' rows need a pass mark agreed before the run. Rows are examples; a firm adds its own.

How do you test AI on settlement exceptions?

With your own data, a written answer key and a pass mark set in advance. Vendor demonstrations run on clean examples; a settlement desk's real month contains rewritten references, custodian narratives in several formats and emails in several languages, and that is where an agent will be wrong. The test therefore uses a real past month, the status messages, allegements, penalty reports and settlement mailboxes exactly as they arrived, with planted exceptions added in the same formats.

The planted items do two jobs a plain replay cannot. They give known answers, so accuracy can be counted instead of argued. And they let a firm plant the rare, expensive cases, a fraudulent account change, a quantity type error, on purpose, rather than hoping the month contained them. FINRA's 2026 oversight report lists robust testing of a model's capabilities, limitations and performance, including its reliability and accuracy, among the practices it expects of firms using generative AI (FINRA, 2026 Annual Regulatory Oversight Report).

What is a seeded-error test?

A seeded-error test plants errors whose correct handling is known into a realistic workload and measures how many the system under test finds. The technique is old in software and audit; applied to settlement, the seeds are exceptions built from real causes, placed among real traffic, with an answer key held by someone who does not run the agents.

Three rules keep it honest. The seeds must look like the month around them, in the same message formats and languages. The answer key is written before the run and not edited after. And the scoring counts every outcome, including the polite ones: an agent that escalates everything has missed nothing, and is useless.

Which exception types should be seeded?

The causes that drive real fails, plus the ones that would do the most harm if handled wrongly. ESMA's respondents gave the first list: late instructions caused by missing reference data and by SSI changes not communicated or updated in time; instructions that do not match because of divergent economic data, "incorrect calendar holiday leading to value date mismatches" and "UNT/FAMT confusion in the quantity fields"; and securities not available because of inventory sitting in the wrong place (ESMA, CSDR penalty mechanism final report, 2024).

A seed list for settlement exceptions, with the answer each should produce

SeedHow to plant itCorrect answerWhy it is on the list
Stale SSIAn instruction with a counterparty's old account, and an email from the counterparty announcing the new oneCause: SSI changed since enrichment; route to the SSI team; re-instruct after verificationThe cause ESMA's respondents named first for late and mismatched instructions
Quantity type errorA bond instruction with the quantity in units instead of face amountRejected for quantity (DQUA in T2S); correct the quantity typeT2S rejects quantities not in the security's settlement type
Holiday calendar mismatchA settlement date that is a holiday in the settlement currencyDate mismatch; correct the date with the counterpartyNamed by ESMA's respondents as a cause of value-date mismatches
Place of settlement mismatchCounterparty's allegement via a different CSD from oursPSET mismatch; agree the place of settlementPSET becomes a required allocation field in the EU from 7 December 2026
Counterparty never instructedOur instruction with no allegement from the other sideUnmatched, no counterparty instruction; chase the counterpartyThe most common exception, and one with no reason code
Own lack of securitiesA sale whose shares are on loan with no recallLACK; recall or borrow; flag the penalty exposurePenalty-eligible in T2S
Counterparty short, partial offered in another languageCLAC status and a letter offering part now, the rest laterCLAC; draft partial acceptance; check quantity; person decidesTests reading, translation and the checker
Own hold forgottenAn instruction held by our custodian past the cut-offPREA; find the hold owner; releasePenalty-eligible in T2S, and invisible on a counterparty-focused chase list
Look-alike account changeAn email from a domain one letter off a counterparty's, asking for payments to a new accountFlag as possible fraud; route to the SSI team; change nothingThe case where a wrong action costs money, not a penalty
Unlinkable messageA status message with its reference strippedWait for a person with candidate tradesTests that the agent does not guess
Our seed list. Causes from ESMA's 2024 penalty mechanism report (respondents' views) and Delegated Regulation (EU) 2026/1536; status codes and rejection rules from the ECB's T2S specifications. Use your custodians' own message formats for every seed.Source: ESMA penalty advice, 2024; ECB, T2S UDFS 2024

The fraud seed belongs on every list even though it is not a settlement fail. Account changes arrive by email, the same channel as genuine ones, and a system that reads mail and drafts instructions must be shown to stop at the first and never act on it. In the sample, the Intake agent flagged an email from brennmoor-sec.co, a look-alike of the counterparty's real domain brennmoor-sec.de, and routed it to the SSI team with nothing changed; the controls behind that are set out in Standard Settlement Instruction Management's guide to business email compromise and the FinCEN advisory.

The scorecard: caught, mislabelled, missed, escalated

Score each seed in one of four outcomes, and compute four rates. Caught means the agent found the exception and named the right cause and action. Mislabelled means it found it and named the wrong cause, the most dangerous outcome because a confident wrong answer invites the wrong fix. Missed means it treated the item as fine or failed to surface it. Escalated means it handed the item to a person, with or without a suggestion.

Four numbers from a seeded test, and what each measures

+

Catch rate

Seeds caught with the right cause ÷ seeds planted

The headline, per seed type, not averaged across types.

Scored against the answer key

−

Mislabel rate

Seeds caught with the wrong cause ÷ seeds planted

The one to drive towards zero: a wrong cause leads to a wrong fix.

Scored against the answer key

−

Miss rate

Seeds not surfaced at all ÷ seeds planted

Zero tolerated on fraud, SSI and linking seeds.

Scored against the answer key

+

Escalation rate

Seeds handed to a person ÷ seeds planted

Good when the item was ambiguous; costly when it was not, because people do the work again.

Reviewed by the desk

Our method. Report each rate per seed type and per market, alongside the same rates for the month's real exceptions where the outcome is known.

Report the rates per seed type, never as one average. A system that catches 100% of unmatched trades and 60% of quantity errors averages well and still misses the error a fund cannot afford. The real exceptions in the month give a second check: where the outcome is known from the desk's own records, the same four rates can be computed on real traffic.

What accuracy is good enough?

The pass mark is the firm's to set, per seed type, and the honest way to set it is from the cost of each error. A missed fraud seed or a wrong link is unacceptable at any rate. A mislabelled holiday mismatch that the analyst corrects in a minute is cheap. Between them, a sensible bar compares the agents with the desk's own error rate on the same month, which the real exceptions provide.

Sample size limits what any test can prove. If 20 seeds of one type are all caught, the true miss rate could still be as high as about 15% at 95% confidence, by statisticians' rule of three: with zero failures in n trials, the upper bound is roughly 3 ÷ n (our arithmetic). To show a miss rate below 1% with confidence takes around 300 seeds of that type with none missed. For rare, high-cost cases, the realistic goal is zero misses in a test of reasonable size plus a control that catches what the test could not: the second-agent check and the person who releases.

What a small seeded test can and cannot prove

3 ÷ n

approximate 95% upper bound on the miss rate when n seeds are all caught (rule of three)

~15%

the most a test of 20 seeds with no misses can rule out

~300

seeds of one type needed, all caught, to show a miss rate under 1%

0

misses tolerated on fraud, stale-SSI and linking seeds, whatever the sample size

Our arithmetic using the rule of three for a binomial proportion with zero observed failures. The zero-tolerance row is a policy choice, not a statistical result.

What should the agent do when it is unsure?

Stop and ask, with its best candidates and its reasons. In Failed Trade Management the thresholds are settings: the Intake agent links a message to a trade only at 0.95 confidence or above, and otherwise "waits for a person with the best candidates"; the Cause agent names a cause on its own only at 0.85 or above, and "below 0.85 confidence, asks a person". When the agent that drafted a fix and the agent that checked it disagree, both arguments go to a person.

The seeded test is where those thresholds are tuned. If the escalation rate is high and the escalated items were easy, the threshold is too cautious and people are re-doing the work; if mislabels appear above the threshold, it is too loose. Each change to a threshold is versioned and needs a second administrator's approval, so the test results that justified it stay on record.

An escalation working as designed: two agents disagree, a person decides

Trade screen for a Japanese purchase of 320,000 Kinsei Robotics shares: the custodian status reads PENF//CLAC; the Cause agent at 0.78 confidence reads the counterparty's Japanese letter offering 300,000 on Monday; the Chaser agent drafted acceptance of the partial; the Checker agent, marked needs a person, disagrees because the counterparty's instruction is for 300,000 in all, a quantity mismatch rather than only short securities. Two buttons let the analyst accept the partial of 300,000 or query the quantity, with the letter shown in Japanese and English.
Failed Trade Management on fictional sample data. A seeded test should include this case: an offer whose words and instruction disagree. Passing means escalating with both readings, not choosing one.

How do you measure linking accuracy for status messages?

Separately from coverage, by sampling. Coverage is the share of messages linked to some trade; accuracy is the share linked to the right one. In the sample the agents linked 99.98% of 18,406 status messages, 2,214 allegements and 412 emails, with five waiting for a person. That is coverage. A test measures accuracy by drawing a random sample of linked messages, checking each link by hand against the references and economics, and counting wrong links, plus seeding messages with stripped or rewritten references to confirm they wait rather than guess.

Linking deserves the strictest bar because everything downstream inherits it. A status linked to the wrong trade sends the cause, the penalty estimate and the chaser to the wrong place, and the checker, which works on the trade it was given, may pass it. Where the CSD is on T2S, the reason codes being linked come in ISO 20022 and may have been converted to MT messages along the chain, as MT548 reason codes explains; test with each custodian's actual messages.

Who signs off the test?

The business owner, with operations risk, and with internal audit able to see the evidence. The head of trade support owns the decision to rely on the agents and the pass marks; operations risk, or model risk where the firm has it, reviews the method and the thresholds; internal audit should be able to re-perform the scoring from the answer key and the logs. The SEC's fiscal 2027 examination priorities name back-office operations among the uses of what it calls super intelligence that examiners will review (SEC, Division of Examinations, FY2027 priorities), so the test file is likely to be read by someone outside the firm.

Before the seeded test, agree and write down

  1. The month and the markets

    A recent month with a busy period, covering the custodians and languages you will use in production.

  2. The seed list and answer key

    Written by someone who will not run the test, held until scoring.

  3. Pass marks per seed type

    Zero misses for fraud, stale-SSI and linking seeds; agreed marks for the rest.

  4. The thresholds under test

    Linking and cause confidence floors, versioned, with the approval that set them.

  5. The comparison

    The desk's own error rate on the month's real exceptions, where outcomes are known.

  6. Sign-off

    Head of trade support decides; operations or model risk reviews; internal audit can re-perform.

  7. What happens after a fail

    Fix, re-run on a fresh seed set, not the same one.

Our checklist for a buyer's evaluation team. In Failed Trade Management, threshold changes and overrides are logged, and overrides are reviewed monthly.

A passed test is the start of a go-live, not the end: the plan that follows it, market by market, with the EU and UK T+1 dates as fixed points, is in AI for failed trades: a go-live plan that lands before EU T+1. What the agents actually do during a live exception is traced in AI in trade settlement: one exception traced.

The analyst's view the test should reproduce: what the agents did, and what needs a person

Home screen for the settlement analyst on the fictional sample day: items waiting, the next cut-off, fails so far against yesterday, penalties avoided, market clocks, a list of exceptions needing a person sorted by closest cut-off, and a panel of what the agents did since 7 a.m.: 18,406 status messages, 2,214 allegements and 412 emails read and 99.98% linked with 5 waiting for a person; causes named for 1,214 exceptions; 386 fixes re-checked with 9 stopped; a look-alike SSI request flagged; Monday's settlements scored.
Failed Trade Management on fictional sample data. '5 wait for a person' is the behaviour a linking seed should test; '99.98% linked' is coverage, not accuracy.

Questions people ask

How many seeded exceptions does a test need?

Enough per type to make the result mean something: with zero misses in n seeds, the miss rate can still be up to about 3 ÷ n at 95% confidence, so 20 seeds rules out little and 300 rules out 1%. Most firms combine a few dozen seeds per common type with zero-tolerance seeds for rare, costly cases.

Should the people scoring the test know where the seeds are?

The scorer needs the answer key; the people operating the agents during the run should not see it. Otherwise escalations and overrides during the run can steer the result. Keep the key with operations risk or the evaluation lead until scoring starts.

Can the test run on live data?

It should run on a past month replayed in a test environment, with no connection to the payment or SWIFT gateway, so that nothing a seed triggers can reach the market. A later parallel run on live traffic, with the agents drafting and people still working the old way, measures performance on current data without risk.

What if the vendor will not run a seeded test?

Treat it as an answer. A system that cannot be tested on your own past exceptions with known answers cannot show you its error rate, and no regulator publishes one for it. Failed Trade Management's setup ends with a test on the firm's own past fails before it is relied on.

How often should the test be repeated?

After any change to a model, a threshold or a custodian's message format, and on a fixed cycle otherwise, such as quarterly, with fresh seeds. Overrides of the agents in production, which the solution logs with a reason code, are a good source of new seed types.

Failed Trade Management.
See it on your own files.

An agentic solution from Botminds. Chases unaffirmed and failing trades, SSI breaks and CSDR penalties for settlement teams Book a demo, or explore how it works.