Best execution analytics are the measurements and checks that turn a quarter's orders into evidence of execution quality: cost against benchmarks and expected cost, outliers beyond the policy's thresholds, the reasons behind them, and broker and venue comparisons. The way to test them before buying is not a demonstration on the vendor's data; it is a replay of a quarter your best execution committee has already reviewed and signed. Every number the tool produces can then be checked against a number your people produced, and every outlier and reason against what the committee decided. Botminds' Best Execution Review is built to be tested this way, and the pass marks below are the ones we would set for it, written down before the run so nobody grades the result after seeing it.
A pilot scorecard for best execution analytics, with pass marks set before the run
Population
What is testedOrder count reconciles between the order system and the cost analysis file, late bookings included
Pass markCounts tie, every difference named
Figures
What is testedEvery figure in last quarter's committee papers recomputed and tied to its source row
Pass markAll tie, or each difference explained by a file version
Outliers
What is testedEvery outlier in the signed papers found under the same policy; any new ones listed
Pass markNone missed; each new outlier traced to a rule
Exclusions
What is testedEvery order set aside carries a code that matches the policy
Pass markNo uncoded exclusions
Reasons
What is testedReasons found for the outliers the committee accepted, with sources
Pass markAgrees with the committee on a pre-agreed share; every disagreement explainable
Brokers
What is testedBroker rankings and trends against expected cost recomputed
Pass markSame ranking, or differences traced to data
What are best execution analytics, and what should a pilot prove?
Best execution analytics are everything between the raw orders and the committee's decisions: reconciliation, benchmarks, thresholds, outliers, reasons, comparisons and the figures that summarise them. The cost measurement at its core is transaction cost analysis, which has its own guide to what TCA measures and misses; best execution analytics is the wider job of turning those costs into a reviewed, signed answer to whether the duty was met. A pilot should prove two things. First, that the tool reproduces what your firm already knows, so its numbers can be trusted. Second, that it finds what your firm missed, so it is worth paying for.
The first half is often skipped because it is unglamorous, and it is the half regulators care about. FINRA's guidance on relying on someone else's best execution analysis is to keep "periodically reviewing both the methodology and the results of the review" (FINRA, December 2025). A pilot that never checked the tool against known answers has reviewed neither.
How do you test best execution analytics before buying?
Replay a signed quarter through the tool, with your own policy, and compare the result with the committee papers line by line. Set the comparison and the pass marks before the run; let compliance hold the answer key; and run the tool without telling it what the committee decided. If the tool uses AI agents to find reasons, make sure the people judging the result can see where each reason came from.
The point of using a signed quarter is that the answers already exist and were reached independently. A demonstration on a vendor's sample data shows the interface. A replay of your own quarter shows whether the tool would have changed anything your committee did, and whether it would have survived the questions an examiner asks: the SEC staff's first finding in 2018 was advisers who "could not demonstrate" their reviews (OCIE, July 2018).
Which quarter should you replay?
Choose the most recent complete quarter that the committee has signed, ideally one with something difficult in it: a volatile week, a broker the committee argued about, a provider re-statement, a new asset class. A quiet quarter will pass any tool. A quarter with known problems shows whether the tool finds them without being told.
Avoid a quarter whose data has since been reprocessed by the cost analysis provider unless you keep the file the committee actually saw; the comparison needs the same inputs. And make sure the policy version in force for that quarter is the one the tool applies, thresholds and exclusions included. A pilot run on this quarter's policy against last quarter's decisions measures the policy change, not the tool.
What data do you need for a best execution analytics pilot?
The pilot needs the quarter's orders and fills, the cost analysis results the committee saw, the policy, the signed papers and minutes, and read-only access to the places reasons are written. The table lists each source and why it matters.
Data for a best execution analytics pilot, and what each source is for
| Source | What the pilot takes from it | Why it matters |
|---|---|---|
| Order management system | Orders, portfolio manager instructions and allocations for the quarter | The population, and the instructions that explain many outliers |
| Execution management system | Fills, algorithm settings and trader notes | Trader notes are the most common source of a documented reason |
| Cost analysis provider | Results, pre-trade estimates and peer data, every file version | The figures the committee saw, and any re-statement since |
| Market data and newswire | Trading pauses, auctions, quotes, volumes and headlines | What a reason is tested against |
| RFQ platforms | Bond and FX quote logs and last-look records | Whether a trade was done at the best quote received |
| Message archive | Desk chat and email, searched in place | Reasons written outside the trading systems |
| Policy, papers and minutes | The policy version, the signed committee papers and the minutes | The answer key the pilot is scored against |
How do you compare the result with last quarter's committee papers?
Compare four things in order, because each depends on the one before: the population, the figures, the outliers and the reasons. If the order counts do not reconcile, nothing after them can be trusted; if the figures do not tie, the outliers will differ for the wrong reason.
The comparison of outliers needs care in both directions. An outlier in the signed papers that the tool missed is a failure, unless it traces to a policy rule the committee applied by hand. An outlier the tool found that the papers did not contain is either a gap in last quarter's review or a misapplied rule, and every one should be explained before the pilot is scored. The reasons come last: for each outlier the committee accepted, did the tool find a documented reason, from which source, and would that reason have been accepted?
How a quarter becomes outliers, every exclusion counted: the first page to compare in a pilot
What counts as a pass in a best execution pilot?
A pass is whatever you agreed before the run, and the agreement should be strict where the answer is arithmetic and honest where it is judgement. Population and figures should tie completely, or every difference should be explained by a file version or a late booking. Outliers should match the signed papers under the same policy, with every new one traced to a rule. Reasons are judgement: set an agreement rate with the committee's decisions in advance, and require that every disagreement can be explained from the sources.
A pilot that finds things your review missed has not failed; it has done its second job. In Best Execution Review's sample setup, the back-test on the quarter before go-live found seven outliers that the manually prepared second-quarter papers had missed. The sample's third-quarter run also caught four orders booked on the last day of the quarter after the cost analysis provider's cut-off, and one provider re-statement that would otherwise have left a figure in the papers 0.5 basis points too low.
What a replayed quarter can turn up: Best Execution Review on sample data
7
outliers found in the back-test that the manually prepared second-quarter papers had missed
4
late-booked orders missing from the provider's file, found and re-pulled
1
provider re-statement caught: an FX average cost of 2.9 bp corrected to 3.4 bp
1,284
figures in the committee papers, each tied to its cost analysis row
A two-week best execution analytics pilot, day by day
Two weeks of elapsed time is enough for a replay if access is arranged first; most of the time is spent comparing, not running. The plan below is ours, built around ten working days and the people who need to be in each step.
A ten-working-day pilot: most of the time goes on comparison, not on the run
- Read-only access; policy and pass marks agreed3 days
- Load and reconcile the quarter2 days
- Run1 day
- Compare with the signed papers3 days
- Readout to trading and compliance1 day
Put three people on it. The trading analyst who prepared the signed papers knows where every figure came from. A compliance reviewer holds the answer key and scores the comparison. The head of trading, or a deputy, judges the reasons where the tool and the committee disagree. Traders should not be asked anything during a replay: the point is to see what the tool finds without help.
Four ways a best execution pilot goes wrong
Most failed pilots fail in the set-up, not the run. The first is moving the thresholds after seeing the results: if the outlier list does not match, a vendor can tune the rules until it does, and the pilot then proves only that the tool can be tuned. Freeze the policy version before the run. The second is clean data. A file someone has tidied for the pilot hides exactly the late bookings and re-statements that a real quarter contains; use the files as they were.
The third is helping the tool. Asking traders for reasons during the replay, or pointing the vendor at the trades the committee argued about, turns a test into a rehearsal. The fourth is scoring reasons by count. A tool that "explains" 60 of 63 outliers has done well only if each explanation cites a source a reviewer would accept; ten reasons that say "market conditions" are worse than five honest requests to traders. Score the source and the fit, not the total.
How long does best execution analytics take to go live?
After a successful pilot, go-live is mostly the same work again on the current quarter: the connections already exist, the policy's thresholds have been tested, and the pass marks become the first quarter's controls. What usually lengthens it is approvals, not technology: compliance approving the thresholds, the committee agreeing the new papers, and, where the tool uses AI, approval of the test set that any later model change must pass. What AI should and should not do in a best execution review covers those controls.
A pilot also has a compliance use beyond the purchase. Rule 206(4)-7 requires an adviser to review its policies at least annually, including "the effectiveness of their implementation" (17 CFR 275.206(4)-7), and a documented replay of a quarter against the policy is direct evidence of exactly that.
A replay of last quarter answers the question the annual compliance review must ask, whether the procedure works, with orders whose answers are already known.
What to do with what the pilot finds
Treat the pilot's findings the way the committee treats any review: as decisions to minute. If the replay found outliers the signed papers missed, decide whether they need reasons now. If it found a rule the policy states but the team never applied, fix the policy or the practice. If it found that thresholds no longer separate ordinary trades from unusual ones, propose new ones for compliance to approve. The FCA's 2025 review noted that banks which regularly analysed outlier trades "demonstrated more actions to continuously evolve their arrangements" (FCA, December 2025); a pilot is a concentrated version of that.
FINRA's guidance gives the standard for acting on a difference: where execution quality differs materially, change the arrangement or "justify why it is not modifying" it (Regulatory Notice 15-46). The same applies to a pilot's findings about the review itself. And it is timely: the SEC's fiscal 2027 priorities put trading among the parts of an adviser's compliance program the staff will examine (SEC, FY2027; the SEC's best execution findings lists what earlier examinations found). Twelve questions for best execution monitoring software covers what to ask beyond the pilot. To arrange a replay of your own last quarter, start from Best Execution Review.
The outlier page of the replayed quarter, each trade with its limit and the source of its reason
Questions people ask
What is best execution analysis?
Best execution analysis is the assessment of whether orders were executed as well as they reasonably could have been: cost against a benchmark chosen for the order's instruction, against expected cost and against peers, with outliers explained and brokers compared. It is the core of the periodic review advisers and broker-dealers must run and document.
What are best execution analysis tools?
They range from transaction cost analysis, which measures cost, to surveillance tools that flag exceptions and review tools that document reasons and decisions. A firm usually needs the first and one of the others; a replay of a signed quarter shows which combination a product actually delivers.
Should a pilot use live data or a copy?
Read-only access to the live systems is better than a copy, because the pilot then also tests the connections go-live will use, and no new store of trader messages is created. Either way, use the cost analysis file version the committee saw.
Who should approve a best execution analytics pilot?
The head of trading sponsors it and compliance should approve the pass marks before the run, since compliance will rely on the tool's output in future reviews. Recording both approvals, with the scorecard, gives the firm evidence that the tool was tested before use.
What if the tool disagrees with the committee's past decisions?
Investigate each disagreement before scoring it. Some will be the tool's errors; some will show the committee accepted a weak reason or missed an outlier. Both are useful, and the second kind should go back to the committee as an item for its next meeting.