Portfolio monitoring tools are the systems an investment team uses to watch what it owns and be told when something needs attention: price and limit alerts, news and filing trackers, risk dashboards and, at the professional end, event monitoring that reads every notice on every holding and judges whether it matters. They are hard to compare in a demonstration, because a demonstration shows the events the vendor chose. The reliable test is a replay: take a closed period, such as last quarter, take today's holdings and theses, and run every filing, notice and story from that period through the tool, then count what it caught, when, at what severity and for whom. The same replay is how a firm should test any change to its own alert thresholds before it goes live. Botminds' Portfolio Alerts builds the replay in: no materiality rule changes until the agents have replayed the last 90 days and a person other than the proposer has approved it.
How firms test AI system changes today
48%
always run regression tests on high-materiality updates to vendor models or their guardrails
73%
always or usually have a defined rollback plan for a problem update
25%
always or usually let high-materiality vendor updates flow straight into production
~1 in 2
report ongoing monitoring among their controls on AI data risk
What are portfolio monitoring tools?
The phrase covers several kinds of software that watch different things. Price and limit alerts fire when a number crosses a line. News and filing trackers match articles and filings to names on a watchlist. Risk and compliance monitors check positions against limits and mandates, a separate discipline from news. And event monitoring reads the events themselves, each notice, filing, rating action or story on a holding, and decides whether it matters to the case for owning the stock.
This guide is about the last two kinds of tool in that list, news trackers and event monitors, because they are the ones a demonstration flatters most. Checking holdings against an investment policy's limits is a different job, covered in investment monitoring software. A price alert is easy to test: either it fired at the level or it did not. An event monitor makes judgments, and a judgment can only be tested against events whose outcome you already know.
A vendor's demonstration and a replay on your own quarter
The demonstration: Chosen by the vendor, usually famous ones.
The replay: Every filing, notice and story on your holdings over a closed period, including the dull ones.
The demonstration: A sample portfolio.
The replay: Your positions in every fund, and your analysts' own thesis pillars.
The demonstration: English, major wires.
The replay: The sources you license, in the languages your holdings publish in.
The demonstration: That the tool can find a famous event.
The replay: What it would have caught, missed, merged and routed, how fast, and how many alerts each analyst would have received.
The demonstration: An hour of your time.
The replay: Connecting your holdings and sources, and an analyst's time to review the misses; much of it is the work of going live anyway.
How do you test a portfolio monitoring tool?
By replaying a period whose answers you already have. Pick a closed window, 90 days is long enough to include an earnings season, and assemble the answer key before the tool runs: every regulatory filing and exchange notice on your holdings in that window, every day a holding moved by more than a threshold you choose, and every event your analysts actually acted on. Then run the tool over the same window, on the holdings and theses you have today, and compare.
Three design choices decide whether the test means anything. Write the answer key first, so nobody fits the result to the tool. Include crowded days on purpose: Hirshleifer, Lim and Teoh found that on days with many earnings announcements the immediate price response to each surprise is much weaker and the later drift stronger, so those are the days attention fails and a tool earns its keep (Journal of Finance, 2009). And replay at the speed the sources published, not as a batch, so that lead time can be measured.
What a replay should report, and how each measure is defined
| Measure | Definition | What a good result looks like |
|---|---|---|
| Caught | Events in the answer key the tool alerted on, at any severity | High on filings and notices; misses explained one by one |
| Missed | Answer-key events with no alert, especially those followed by a large price move | Few, and none on the events your analysts acted on |
| Merged | Items read against distinct events after merging | Repeats folded into one alert; no two events merged into one |
| Lead time | Alert time minus the time of the first source, by language | Minutes, measured from the home-market notice, not the English wire |
| Routed | Alerts that reached the covering analyst, against alerts that reached the wrong person or nobody | Every alert has an owner from the coverage map |
| Load | Alerts routed per analyst per week, and how many were repeats | A number the analysts agree they can answer |
| Dismissed | Alerts an analyst would mark not material, sampled by someone else | Low, and stable as thresholds change |
What is backtesting an alert rule?
It is running a proposed rule over past data to see what it would have done, before it goes live. Banking supervisors have the clearest public description of the principle. Their revised model risk guidance of April 2026, SR 26-2, calls it outcomes analysis: set what a model produced against what actually happened, judged by what the model was built to do, either inside ongoing monitoring or as a separate back-test or outlier review. It adds a point that fits alert rules exactly: where a model rests mostly on expert judgment, measuring its outcomes is how that judgment gets tested (Federal Reserve, SR 26-2 attachment). An alert threshold is exactly that kind of judgment.
Two caveats. SR 26-2 is guidance for banking organisations, not for asset managers, and it states that generative and agentic AI models are outside its scope. It is cited here for its description of good practice, which an investment firm is free to adopt. For a US investment adviser, the binding obligation is more general: Rule 206(4)-7 requires written compliance policies and a review, "no less frequently than annually, the adequacy of the policies and procedures ... and the effectiveness of their implementation" (17 CFR 275.206(4)-7). Where alerts support a firm's controls on material nonpublic information or its records, a replay is evidence for that review.
The replay in practice: one rule change, 90 days, 14 more alerts
In the fictional sample in Portfolio Alerts, the research operations lead, Talia Brooks, proposed version 15 of the firm's materiality rules on October 14. The change targeted guidance revisions by Japanese and Korean companies of more than 10% of operating profit, which she found had been scoring below the alert line. Before anyone could approve it, the agents replayed it over the previous 90 days, July 16 to October 13.
Replaying rule version 15 over 90 days: 14 more alerts, none dropped, 3 before a large move
The replay answers the three questions an approver needs. Would the change catch more of what matters? Yes: three of the 14 added alerts came before a move of 5% or more within a day. Would it lose anything? No alert would have been dropped. What would it cost the analysts? About 0.2 more alerts each per week. With that in front of her, the head of investment risk, Margaret Halloran, could approve with a reason code. The system would not let Talia approve her own proposal.
Approving a materiality rule change in Portfolio Alerts, with its 90-day replay attached
How do you set alert thresholds?
Set them where the analysts can answer every alert that crosses them, then move them only with a replay. A threshold is a trade between misses and load, and both have costs that research measures. Too low, and the alerts multiply: in Ancker and colleagues' study of clinicians, each extra reminder per visit cut the chance that any reminder was accepted by about 30% (BMC Medical Informatics and Decision Making, 2017; the study is read for analysts in alert fatigue: an analyst's 24 hours). Too high, and the misses are the events that matter most.
In Portfolio Alerts the threshold is two lines on a score of 0 to 100. Events at 60 or above are routed to a person; events below wait in the 06:00 digest, scored and logged, so a missed event can be found and its score inspected. A second line at 85 marks the scores at which a missed deadline can page the lead portfolio manager. The weights behind the score, event type 30, thesis fit 30, position 25 and surprise 15, change only in a new rule version, after a 90-day replay. Fixing the digest's contents in the record matters as much as the lines: it is what lets a replay say which events would have moved from the digest to a person.
What should a portfolio monitoring tool catch?
Every event that could change the case for owning a holding, from the first source, in its own language. In practice that means the home-market exchange notice or regulatory filing, not only the English wire; ownership filings, where a large holder or an insider is the filer and your holding is the subject; rating actions; and local press that reports something the company has not yet said, followed by the company's response. Form 8-K items, read as a holder lists the US events, and ad hoc disclosure by market the rest.
It should also catch its own failures. A source whose login has expired or whose feed has gone quiet produces no alerts, which looks exactly like a quiet day. In the sample, the analyst's summary from her agents noted that one licensed source's sign-in had expired overnight and that research operations was on it; a replay should include a day when a source was down, and show whether anyone noticed.
How do you measure missed alerts?
Against an answer key, not against the alerts themselves. A tool cannot report what it did not see, so the measure has to start outside it: the complete list of filings and notices on your holdings from the regulators' and exchanges' own systems, and the list of days each holding moved by more than your threshold. A missed alert is an entry on either list with no alert, or with an alert that reached nobody or arrived after the move.
Price moves are a useful but imperfect key. A large move with no news may be the market; a large move on news the tool did not cover is a miss. The sample's 5%-within-a-day key is a setting, not a standard; no regulator or independent study publishes one. Pick a level that matches your holdings' volatility, write it down before the replay, and keep it for every replay after, so this quarter's misses can be compared with last quarter's.
Who should approve a change to alert rules?
Someone other than the person who proposed it, with the replay in front of them. That is maker-checker, the oldest control in finance, applied to the alerting system itself. It matters more as more of the reading is done by AI: in the Bank of England and FCA's 2026 survey, no more than 9% of firms expected agents to change limits, thresholds and controls even counting a year of plans (Bank of England and FCA, 2026). Thresholds are a control, and the person who owns the tool should not be the only one who can move them.
In Portfolio Alerts, research operations proposes, the agents replay, and the head of investment risk approves or rejects with a reason code. Changes to an agent need a second approver too. First-time setup ends with a test alert replayed from the past; in the sample it passed end to end in 4.2 seconds, with the same score, owner and deadline as the original. The wider question of what agents may and may not decide is covered in AI stock alerts: what agents may decide.
A buyer's checklist for a replay
A replay is worth running even if you never buy anything, because it tells you what your current tools miss. These are the things to ask for, of any vendor including us.
Ten things a replay on your own quarter should show
- Your holdings and theses, not a sample
Positions in every fund as of today, and the pillars your analysts actually wrote.
- An answer key written first
Filings, notices and price moves listed before the tool runs.
- Every source in its own language
Lead time measured from the home-market notice.
- Events merged, with the merge rate
Items read against distinct events; no two events merged into one.
- Severity and owner for every alert
Who would have been told, by when, and who copied.
- The misses, one by one
With the reason: source not covered, issuer not matched, scored below the line.
- Load per analyst per week
Agreed with the analysts before go-live.
- A crowded week included
An earnings peak or a market shock, where attention fails.
- A threshold change replayed
What one proposed change would add and drop, and who approves it.
- The record of the replay kept
So next quarter's replay can be compared with this one.
Where Portfolio Alerts fits among portfolio monitoring tools
Portfolio Alerts is event monitoring for buy-side research: it reads notices, filings, rating actions and stories on every holding, scores each event against the covering analyst's thesis and the position in every fund, and routes it to an owner with a deadline. It works with the systems a firm already runs, taking intraday holdings and the restricted list from the order management system, theses and the coverage map from the research management system, and prices and consensus from the market data vendor. It is not a risk system, a pre-trade compliance engine or a trading tool, and it does not replace them. To replay your own last quarter through it, start from Portfolio Alerts.
Questions people ask
What is portfolio monitoring software?
Software that watches the holdings in a portfolio and tells someone when something needs attention. It ranges from price and limit alerts to news trackers and risk monitors. Event monitoring, the kind tested in this guide, reads each notice and filing on a holding and judges whether it matters to the investment case.
How long should a replay test cover?
Long enough to include an earnings season and at least one crowded week, which in practice means about a quarter. The sample in Portfolio Alerts replays 90 days for every rule change. Research on investor distraction shows that crowded announcement days are where reactions are weakest, so a replay that skips them flatters the tool.
Is backtesting an alert rule the same as backtesting a trading strategy?
The idea is the same, running a rule over past data, but the measures differ. A trading backtest measures returns; an alert replay measures what was caught, missed and merged, how fast, who was told and how much load it added. SR 26-2 calls the general practice outcomes analysis.
How often should alert thresholds be reviewed?
Whenever someone proposes to change them, with a replay, and at least as often as the firm reviews its compliance programme: Rule 206(4)-7 requires US advisers to review their policies and procedures no less than annually. Rising acknowledgment times or dismissal rates are a signal to review sooner.
Can a monitoring tool tell you what it missed?
Only partly. It can show what it read and scored below the line, which is why a digest kept on the record matters. Events it never saw, such as a source it did not cover, can only be found against an outside answer key: the regulators' and exchanges' own lists and the days your holdings moved.