Equity research automation: a pilot that replays last season

Equity research automation is software that takes the routine work of earnings season off a research team: catching each release, updating the analyst's model, checking the numbers and drafting the earnings note, with the analyst deciding and signing. The safest way to test it is to replay a season that has already happened. Last season's releases are public, the analysts' own model updates from that season are a ready-made answer key, and every disagreement between the agents and the analyst can be settled by opening the page. A replay can run in the quiet weeks between seasons, before any portfolio manager depends on the output. Botminds' Equity Research Automation is built to be piloted that way; this guide sets out a five-week plan, the data a pilot needs, what to measure, and how to switch on name by name before the first release of the next season.

Product examples come from the solution's fictional sample firm, Quillfield Capital. The pilot plan is our recommendation, not a regulatory requirement.

What is equity research automation?

Equity research automation is the use of software to do the repeatable parts of research coverage, mostly in earnings season, so analysts spend their hours on judgment. The repeatable parts are the ones with a right answer on a page: did the company report, what did it report, does it add up, how does it compare with the model. The judgment parts have no page to check against: whether the thesis still holds, whether the estimate should move, whether the rating should change.

In Equity Research Automation the repeatable parts belong to seven agents: a Release Watcher, an Extractor, a Tie-out Reviewer, a Variance agent, a Note Drafter, a Compliance Sentinel and a Coverage Planner. Each has written limits. The Extractor writes actuals only and never touches forecast cells; the Tie-out Reviewer never fixes a value by guessing; the Note Drafter never changes a rating or sends a note. A pilot tests both halves: that the agents do their work correctly, and that they stop where they should. The guide to the equity research process maps which task sits with whom.

Why the weeks between seasons are the window

A pilot should run when the team has time to look at its results, and the calendar fixes when that is. For calendar-year companies the last 10-Q deadline of a quarter falls 45 days after quarter-end, and the next quarter ends about six weeks later (Form 10-Q, General Instruction A.1). Releases for the new quarter then start a few weeks after that. The annual season is the exception: the 10-K deadline for the smallest filers is 90 days after year-end (Form 10-K, General Instruction A), which is March 31, the day the first quarter ends.

The practical reading: the best windows are late August to early October, mid-November to mid-January, and late May to early July. A pilot started in the second half of November 2026 can replay the third-quarter season that ended on November 16 and be live before the fourth-quarter releases arrive in January. A pilot that starts in February runs into the annual season and has no clean gap after it.

What data do you need for a pilot?

A replay needs what the agents would have had on each morning of last season, plus what the team actually did. Most of it already exists; the work is gathering it with the right dates attached.

Can you test AI on past earnings releases?

Yes, and it is the most informative test available, because the answers are known. Every figure in last season's releases can be checked against its page; every estimate change and thesis call the analyst made is on record; every company that recast its history did so in a document the agents should have found. A demonstration on a vendor's chosen company shows that the tool can work. A replay of your own season shows how it works on your names, your models and your quirks.

The profession's standard points the same way. CFA Institute Standard V(A) asks members to test the output of models and other preprogrammed analytical tools, and to do it before the tool becomes part of their methods (CFA Institute Standard V(A)). For a US adviser, a tool that drafts research also falls under the annual review of its compliance policies, which Rule 206(4)-7 requires at least once a year (17 CFR 275.206(4)-7). A scored replay is evidence for both.

The solution applies the same discipline to itself. Any change to an agent or a threshold is tested on a set of 1,240 past releases with known answers before the Director of Research approves it. In the sample, a proposed tightening of the tie tolerance on derived ratios came back with "2 more holds, no missed errors, no change to time to note".

What should a pilot measure?

A pilot should measure what the team will be judged on: whether every figure in a note is right, whether every wrong figure was stopped, and how long it took to get a signed note to portfolio managers. Speed matters only after accuracy, and accuracy has two sides: wrong values written, and wrong values caught.

Five measures cover most of it. Values: of every value the agents wrote, how many matched the page, how many differed from what the analyst keyed, and in each difference who was right. Holds: how many notes were held for a failed tie, and of those, how many were real errors in the release, recasts that needed confirming, or false alarms. Misses: how many errors reached a draft without a hold; the target is zero. Time: release to pick-up, to model updated, to draft, and the analyst's minutes from opening the draft to signing. Load: how many values went to the research associate because they were below the confidence line.

Holds are a result, not a failure

A pilot will produce holds, and a team should read them as the tool working. A hold means the agents found something they could not tie and stopped instead of guessing. Some holds will be the issuer's own errors or ambiguities; some will be recasts the model did not know about; some will be over-cautious, which a threshold change can fix.

The FinanceBench researchers made the same point about language models answering questions on filings. Models that refused to answer did less harm than models that answered wrongly.

"Models refusing to answer is arguably preferable to giving an incorrect answer as it creates less risk of error, and misplaced trust, by users."

Islam et al., FinanceBench, arXiv 2311.11944 (November 2023; not peer reviewed)

A replay is where the team decides how cautious it wants the agents to be. The trade-off is explicit in the settings: the tie tolerance, the confidence line below which a value goes to a person, and the variance alerts. Tighten them and there are more holds and fewer misses; loosen them and the reverse. The right setting is the one the Director of Research approves after seeing both counts.

Exceptions: everything held instead of guessed, with what the agent found and proposes

How long does a research automation pilot take?

Our plan takes five weeks, and the replay itself is the shortest part. The first week goes on connecting the inputs and mapping every covered name to its model and analyst; the replay runs in the second; the third is spent scoring it, which is where the analysts' time goes; the fourth tunes the settings and shadows any company reporting off-cycle; the fifth switches names on.

Off-cycle reporters are a gift to a pilot. The SEC's count of companies filing domestic annual reports in 2024 found that 8% ended their fiscal year in a month that was not a calendar quarter-end (Federal Register, May 7, 2026), so some covered companies report in the gaps between seasons. Those releases can run live, with analysts signing as usual, before the peak arrives. The timeline stretches when model versions are missing, when a coverage list includes many non-US names in several languages, or when the restricted list lives in a system that takes time to connect.

Going live with Equity Research Automation, name by name

Switch on by name, not by team, and have each analyst approve their own names after reviewing their replay. An analyst who has seen the agents work on their own models, found where they held and why, and checked a dozen values against the page will sign the first live note with confidence. One who has only seen a demonstration will re-check everything, and the hours will not come back.

Two rules make the switch safe. First, the sign-off chain does not change: the analyst signs as the maker, a different person releases as the checker, and the agents never send a note. Second, the first live season is still measured. The same five measures from the pilot, run weekly, tell the Director whether the live results match the replay. In the solution, the board and the audit trail make that a report rather than a project.

Evidence a pilot should not lean on

Three kinds of claim turn up in vendor material, and none should substitute for a replay. The first is a published result that no longer stands. The best-known working paper behind the 'language models beat analysts' headlines was withdrawn by its authors in February 2025, after a co-author could not replicate its analyses (arXiv 2407.17866, withdrawal notice). Financial statement analysis with large language models, tested sets out what remains.

The second is a general benchmark. FinanceBench is a useful warning, but its questions are about other companies' filings and its results describe 2023 models; it cannot say how a tool will do on your names. The third is a vendor's own count of minutes saved per model update. It may be true for the vendor's demonstration; the replay will tell you whether it is true for you. Supervisors are watching the same gap between claim and practice: the Bank of England and FCA found in 2026 that pre-deployment controls such as human approval and vendor due diligence were each reported by more than 90% of firms, while ongoing monitoring and periodic review were reported by only around half (Bank of England and FCA, October 26, 2026). A pilot that keeps measuring after go-live closes that gap.

We will run Equity Research Automation on last season's releases for your own covered names and your analysts' own models.

Questions people ask

How do you test equity research software?

Replay a season you have already covered. Feed the software last season's releases and your models as they stood the night before each one, then compare every value with the page and with what your analysts keyed. Count the errors it caught, the errors it missed and the time it took.

When should a research team go live?

In the weeks before a season starts, after the replay has been scored and each analyst has reviewed their own names. For calendar-year companies, the windows after the second- and third-quarter seasons each give about six weeks before the next quarter ends.

What is a golden set in research automation?

A fixed set of past releases with known, checked answers, used to test any change before it goes live. In the solution's sample, every change to an agent or threshold is run against 1,240 past releases and approved by the Director of Research with the result attached.

How many names should a pilot cover?

All of the names whose analysts take part, for at least one full season. A pilot on a handful of easy names hides the recasts, the foreign-language releases and the crowded mornings that a full season contains, which are where automation either earns its place or does not.

Do analysts have to change their models for a pilot?

They should not have to. The agents write reported actuals into each analyst's own Excel model as a new version and leave forecast cells untouched, so the pilot tests the tool against the models the team really uses.

What if the replay finds errors in our own past notes?

Some replays will, because the answer key is the analysts' own keying, and people make keying errors under time pressure. Each difference is settled by the page. Finding them is part of the value of the exercise, and a reason to keep the replay results on record.