Agentic AI proof of concept: re-make last month's factsheets

To run an agentic AI proof of concept for fund factsheets, give the agents a month of factsheets you have already released, the signed data and documents you used to make them, and your templates and rules, and let it re-make them; then compare its output with what you published, field by field. You know the right answer for every field, because you signed it. Each difference falls into one of four kinds: the tool got it wrong, the tool held what it could not prove, your rules differ from what was configured, or your released factsheet was wrong. The last kind is the most valuable finding a pilot can make, and it is the reason this design beats a vendor demonstration on sample data. Botminds' Factsheet Automation goes live this way: its agents first prove themselves on a cycle already released by hand, and each finding is confirmed by your compliance reviewer before go-live.

The design takes a month or less, needs no live publishing, and tests the tool on your share classes, your markets, your languages and your exceptions rather than on the ones that make a good demonstration.

How do you run an agentic AI proof of concept for factsheets?

Freeze a month, re-run it, and compare. Take a recent month whose factsheets you released and have not changed since. Snapshot everything that was true at that month-end: the signed returns file, administrator NAVs, holdings, the KIDs and prospectuses then in force, the registration list on that date, the clause library, the approved commentary and translations, and the templates. Give the tool exactly that, with read-only access, and let it build and check the month. Then compare every field of every factsheet with the released version, and classify each difference as above.

Freezing the inputs matters because factsheets are judged at the time they are used. US rules measure an advertisement's returns against its "date of use", which covers the whole time investors use it, not just the day it first appears (17 CFR 230.482(g), note; the production guide sets out both clocks). A pilot that feeds today's data into last month's factsheets will produce differences that are not errors at all; a pilot that uses the month-end snapshot will produce only real ones.

What data do we need for a pilot?

You need what your team used to make the factsheets that month, and the factsheets themselves. If you cannot assemble it, that is itself a finding: it means your released factsheets could not be reconstructed for an examiner either. US advisers must already keep the working papers that show how each performance figure in their communications was calculated (17 CFR 275.204-2(a)(16); the client reporting accuracy guide quotes the rule), so the performance part should exist.

Choosing the sample: which classes and markets to include

Include the hard cells, not just the easy ones. A pilot on a single English-language fund family tells you little. Choose a month and a scope that contain the situations where factsheets go wrong: a class newly registered in a market, a market with more than one language, a currency-hedged class, an ETF with a listing line, a fund whose KID changed during the month, a benchmark restated after sign-off, a corporate action in the top ten. ESMA's statement on AI asks firms to test AI tools for "accuracy, fairness, and reliability in various market scenarios" and to run "periodic stress tests to evaluate how these AI systems perform under extreme market conditions" (ESMA, 30 May 2024, paragraph 21); for factsheets, a volatile month with restatements and late corporate actions is the stress test.

Size follows from the cells. If the whole month is a few hundred factsheets, run all of them: the comparison is mechanical and a full run removes any argument about sampling. If it is thousands, run whole waves rather than a random scatter, so that each finding can be traced to a template, a market or a source.

How long does a factsheet pilot take?

About four weeks for one month's factsheets, most of it spent on data and rules rather than the run. The plan below is our suggestion, not a measured duration; your own access requests and approvals will set the real pace.

Two weeks of the four are about your rules, not the tool. Writing down the rules your team applies from memory (which legend goes on which market, when a KID change holds a factsheet, how a hedged class is benchmarked) is work a pilot forces and every option needs. It is also the work that makes the result comparable across vendors.

Who should run the pilot, and who should judge it?

The production lead should run it, and marketing compliance should judge it, with internal audit watching. The production lead owns the inputs and knows the month's exceptions; the performance analyst confirms which figures were signed and when; a linguist for each language in scope reviews any difference of meaning; the compliance reviewer classifies each difference and owns the decision on published errors. Internal audit's role is the one most often left out: an observer who did not configure the rules and does not want the tool to win keeps the classification honest, in the same way as in a seeded-error test.

Keep maker and checker apart in the pilot as you would in production. The person who wrote the rules into the tool should not be the only person who classifies the differences, because a rule gap and a tool error look alike to the author of the rule. The vendor's role is to explain each tool error and fix it, not to classify.

Pass criteria to agree before the pilot starts

Agree them in writing before the run, with compliance, so that the result cannot be argued afterwards. No regulator or independent study publishes a pass mark for factsheet tools; the criteria below are ours, and the thresholds are yours to set.

• Regulated fields. No unflagged tool error on returns, benchmark, costs, risk class, dates or legends. An error the tool held is a catch; an error it released is a failure.
• Explained differences. Every difference classified with a reason and both sources.
• Holds. A false-alarm rate your reviewers can live with, and the time to resolve a hold.
• Evidence. For each value, its source, as-of date, check result and approver, exportable.
• Limits. Shown, not described: the tool never edits a source, never writes legal text, never publishes without a signature, and the maker cannot release.

The last criterion matters as much as accuracy. FINRA expects firms that use such tools to have policies that "address technology governance, including model risk management" where the tools are part of supervision (Regulatory Notice 24-09), and ESMA expects firms to check, regularly and after the fact, any process in which AI helps deliver information to clients (ESMA, paragraph 16). The pilot is the first of those checks, run before a single factsheet goes out.

Stringent oversight should be adopted by firms when employing AI in this context.

ESMA, public statement on AI in retail investment services, 30 May 2024, paragraph 16

Reading the result: a tool error, or a mistake you already published?

Expect rule gaps and published errors among the differences, not only tool errors, and be ready for that. Released factsheets were checked on a sample; a pilot checks every field. Where the tool and the released factsheet disagree, open both sources before deciding. Suppose the KID in force at month-end shows ongoing charges of 1.72% and the released factsheet shows 1.65%: the tool is right, and the released factsheet was out of date.

A published error found by a pilot is a real error in a live marketing communication. Treat it under your error policy, not the pilot's. In the UK, the Consumer Duty expects a firm that identifies issues in its communications to "investigate the issue" and "correct any deficiencies" (PRIN 2A.5.10R(2)). In the EU, the information on a fund's features "should be kept up to date" (ESMA guidelines, paragraph 23). Whether to reissue depends on what the error was and who relied on it, which is a compliance judgement.

What Factsheet Automation's own dry run looks like

Factsheet Automation is deployed with this kind of run built in. In the solution's fictional sample, the agents first proved themselves on a cycle the sample firm had already released by hand: a dry run on 1,184 April factsheets turned up 37 findings, each confirmed by the compliance reviewer before go-live. After go-live, the sample firm's cycle report tracks the same measures a pilot starts: business day of release (3 in September against 8 to 10 before May), field checks run (50,266 in September, 41 on each of 1,226 factsheets), and the share of agent flags people confirmed (241 of 250 over three cycles). These are fictional figures from sample data, shown to illustrate the measures; your pilot produces your own.

During the run, every action by an agent or a person is written to an append-only audit trail with its reason and evidence, which is what lets a reviewer classify a difference in minutes rather than reconstruct it. The agents hold what they cannot prove rather than guess, so a pilot's "correct holds" are visible as exceptions with owners, not buried in the output. How each agent works on a live cycle is traced in the AI factsheet generator guide, and the cost case the pilot should feed is in fund data management solutions vs agents.

Factsheet Automation's audit trail: every action with its reason and evidence

A fix re-checked by the agents after a person approves it

From proof of concept to go-live: what to run next

An agentic AI project can pass its proof of concept and still stall before production, because the proof ran on clean, frozen data with no deadline and production has neither. A passed pilot proves the tool on one month; go-live should prove it on live months before anyone relies on it. A practical sequence is one or two cycles in parallel, with the manual process still releasing and the tool's output compared but held back; then a switch of the release to the tool, wave by wave, starting with the simplest range; and then the controls that continue indefinitely. Those are a risk-based release sample that always includes every first-time class and market, a seeded-error test each quarter, and four numbers tracked every month: business day of release, holds by reason, flags confirmed by people and reissues.

The parallel cycles also test what the frozen pilot cannot: live feeds that arrive late, restatements that land after sign-off, and registrations that change mid-month. They are where the tool's holds, and the people who resolve them, meet a real deadline for the first time, so time them as carefully as you count the differences.

Questions people ask

Can we pilot without connecting the tool to production systems?

Yes. A frozen month-end snapshot of files and documents is enough for a re-run, and it is safer: nothing can be published or written back. Connections to live systems belong to the go-live plan, after the pilot has passed.

What if the pilot finds errors in factsheets we already published?

That is a finding about your current process, not a reason to stop the pilot. Classify each one, take it to compliance under your error policy, and fix the cause; the Consumer Duty in the UK expects firms to investigate and correct deficiencies in communications they identify.

Is a free AI fact sheet generator good enough to pilot?

It can be piloted the same way, and the re-run will show the gap quickly: a general generator has no access to your signed returns, KIDs, registrations or approved clauses, so it cannot show a source for each value or hold what it cannot prove.

How many factsheets should a pilot cover?

Enough to include the hard cases: new registrations, several languages, hedged classes, ETFs, KID changes and a restatement. For most ranges a full month or a full wave is practical, because the comparison is mechanical.

What happens after a successful pilot?

Run one or two live cycles in parallel with the manual process, with the tool's output held back, then switch the release to the tool once its results match your criteria. Keep the seeded-error test running quarterly afterwards as an ongoing check.