RFP tracking tools promise to tell an asset manager about institutional manager searches early, and the only fair way to compare them is a back-test on searches whose outcome you already know. Take the RFPs your firm could have entered over the past twelve months, give each tool only the documents that were public before each RFP, and score four things: how many searches it saw first, how many days ahead, whether its probabilities meant what they said, and why it missed the rest. A tool that cannot be run this way cannot be judged. Botminds' RFP Tracking builds the test into first-time setup: before it forecasts anything live, it re-reads the last twelve months of the watched plans' board documents against the RFPs that landed, and shows the results by search.
Figures from the solution on this page are from its fictional sample firm, Quillfield Capital, and are results on sample data, not a customer's measurement. The method is the point: every number below can be computed from your own files.
Four numbers a back-test should hand you, shown here on sample data
52 of 61
relevant RFPs seen before they were issued
96days
median lead time from first signal to RFP
71%
of searches scored 60–80% that actually opened
9
misses, each with a reason: 5 watch-list gaps, 3 with no public minutes, 1 portal behind a login
What RFP tracking tools do, and what none of them can see
RFP tracking tools fall into three kinds, and they differ in when they can first know about a search. Alert and listing services report RFPs once they are issued, or once a consultant announces a search. Document monitors watch plan websites and send the new agendas, packets and minutes they find. Forecasting tools read those documents, find the sentences that start a search, and predict whether and when an RFP will follow.
The first kind cannot show lead time before an RFP is issued, by construction: a list of issued RFPs starts at the RFP. That does not make it useless, since a missed RFP is worse than a late one, but it does mean a back-test will score it differently. The second and third kinds can see a search as early as the documents do, and no earlier.
What none of them can see is a search with no public trail. On the sample firm's books, three of the nine misses were searches a consultant ran for a plan that publishes no minutes, and one sat behind a procurement portal that required a login, which the solution's reading agent will not use. A tool that claims to see everything should be asked how.
How to back-test RFP tracking software on last year's searches
A back-test replays history: the tool sees each plan's documents as they stood on each day, makes its calls, and is scored against what happened next. The steps are simple; the discipline is in not letting the tool see the future.
A back-test of an RFP tracking tool in six steps
A person decides
List the RFPs that landedEvery RFP in the past 12 months that your shelf could have entered: asset class, vehicle, size and minimums within reach. Add the ones you lost and the ones you never saw.
A person decides
Freeze the watch listThe plans the tool will read, fixed before the test starts. A plan added afterwards to catch a known search does not count.
Agent
Replay the documents by dateThe tool reads each plan's agendas, packets, minutes and notices as of each posting date and records every signal and forecast with its date.
Never: reads a document dated after the day it is forecasting.
Agent
Match forecasts to outcomesPairs each predicted search with an actual RFP, or with none, and records the first date the search was seen.
A person decides
Score four thingsRecall, lead time, calibration and the reasons for every miss, on the frozen list.
A person decides
Read the misses one by oneSeparate the misses a tool could fix from the ones no tool could see.
Build the test set: the RFPs that landed, and the board papers before them
The test set has two halves: the outcomes and the evidence. The outcomes are the RFPs. Draw them from the procurement portals of the plans you watch, the announcements of the consultants you cover, and your own CRM's history of searches entered, lost and declined. Keep only the searches your shelf could have entered, or the test will reward a tool for finding mandates you would never pursue.
The evidence is every document each plan made public before its RFP. Public plans keep these: Virginia's open-meetings law, for example, makes minutes public records, drafts included, along with recordings and other records of open meetings (Va. Code § 2.2-3707). Date each document by when it was posted, not by the meeting it records. Approved minutes posted 33 days after a vote were not available on day 10, and a tool that is credited with seeing them on day 10 has been allowed to read the future.
Size matters too. Twelve months across the sample firm's 2,014 watched plans produced 61 relevant RFPs. With a set that small, each search is worth more than a point and a half of recall, so a difference of one or two searches between two tools is noise. A larger watch list, or two years of history, gives a firmer answer.
Score one: how many searches did the tool see first?
Recall is the share of relevant RFPs the tool flagged before they were issued: searches seen first, divided by relevant RFPs. The sample firm's figure, 52 of 61, is 85%. Count a search as seen only if the tool flagged that plan's search in that asset class; a general note that a plan "is reviewing its fixed income structure" seen by a person who then guessed the rest belongs to the person, not the tool.
Recall needs a partner, precision: of the searches the tool forecast with any confidence, how many became RFPs? A tool that flags every board agenda will see everything and help no one. Calibration, below, is the better way to look at the same question, because it asks whether a 70% forecast behaves like 70%.
Score two: how early, and by how much?
Lead time is the number of days from the first dated signal the tool cited to the RFP's release. Report the median, which a few very early searches cannot distort, and the spread. On the sample firm's 52 searches seen first, the median was 96 days, and the distribution shows why the median alone is not enough.
Days between first signal and RFP on 52 searches seen first
Fifteen of the 52 came less than 60 days ahead. For those, the warning is real but short: enough to clear contact with compliance and update a consultant, not enough to reposition a strategy. Lead time also depends on the plan. A plan whose minutes are approved at the next meeting and posted with the following packet gives a reader weeks less than one that posts drafts, and a back-test that reports lead time by plan shows which boards are worth watching most closely.
How do you test whether search forecasts are calibrated?
Group the forecasts by the probability the tool gave, then compare each group's probability with the share that actually opened. If searches scored at 60–80% open about 70% of the time, the scores mean what they say; if they open 40% of the time, every number the tool shows is too high. The method comes from weather forecasting: a widely used score for probability forecasts comes from Glenn Brier's 1950 paper in Monthly Weather Review, and it is the mean of (forecast − outcome)² across all forecasts, with each outcome 1 if the search opened and 0 if it did not. Lower is better, and a tool that always says 50% scores 0.25.
Calibration on 135 forecasts: how often searches at each score actually opened
| Score band | Searches that opened | What one search moves | Reading |
|---|---|---|---|
| 30–45% | 37% | — | In line with the band. |
| 45–60% | 53% | — | In line with the band. |
| 60–80% | 71% (24 of 34) | about 3 points | In line: scores can be read as stated. |
| 80% or more | 86% (19 of 22) | about 5 points | A little under the scores given; the sample firm lowered its top band by two points in version 2.4. |
Two cautions keep calibration honest. Bands with few forecasts swing: in the top band, one search more or less moves the opened share by about five points, so a gap of three points is not evidence of anything. And calibration must be scored on forecasts made before the outcome, at the dates they were made, which is why the back-test replays documents by posting date.
Calibration also tests the sizing, not only the timing. A forecast of a $736 million search, sized at a 4% target of an $18.4 billion fund, can be checked against the size in the RFP when it lands; a tool that cannot show the arithmetic behind its sizes cannot be scored on them. Research on plan sponsors adds a reason to watch the pattern behind hires as well as the count: in a sample of 3,400 plan sponsors, sponsors hired managers after strong excess returns that did not persist, and fired them for many reasons besides underperformance (Goyal and Wahal, Journal of Finance, 2008). A tool that predicts searches from a manager's returns alone will mistake a laggard for a search; the board's own words are the better evidence.
Score four: the misses, and why each was missed
The misses are the most useful part of a back-test, because they say which gaps are fixable. Sort each miss into one of three causes. A watch-list gap means the plan was not being read; that is a configuration choice, fixed by adding the plan. No public trail means the search was decided where no document was posted, such as a consultant-run search for a plan without public minutes; no reading tool can fix it, and coverage of that consultant is the only remedy. An access gap means the documents existed behind a login; that is a policy question for the firm, not a tool setting.
The sample firm's nine misses split five, three and one. Five were fixable by watching more plans, at the cost of more reading. The other four were not misses of reading at all. A back-test that reports only a recall figure hides that difference, and the difference is what decides whether a better tool, a longer watch list or more consultant coverage is the right next spend.
The back-test, kept running: lead time, calibration, open pipeline and product gaps
RFP alert services vs reading the minutes: what the back-test shows
Run the same test on an RFP alert service and a reading tool and they answer different questions. An alert service is scored on completeness: of the RFPs issued, how many did it report, and how fast after release? Its lead time before the RFP is zero or close to it, except where it reports a consultant's announcement of a coming search. A reading tool is scored on lead time and calibration, and on whether its citations hold up.
Many teams will want both: the alert service as a net for the searches with no public trail, the reading tool for the months before the RFP. The back-test tells you what each is worth on your own list of plans. It also tells you where a spreadsheet is enough. A team covering forty large plans in one language may find that a person reading their packets on a calendar beats any tool; an RFP tracking template in Excel, compared with alerts and agents on one search, sets out where that breaks.
What to ask an RFP tracking vendor before the trial
Ask for the back-test before the demo. A vendor that will run its tool on your watch list and last year's RFPs, under the rules above, is offering evidence; one that shows a dashboard of searches it found elsewhere is offering a story.
Eight questions for an RFP tracking vendor
- Will you replay our plans' documents by posting date for the last 12 months?
And show every forecast with the date it would have been made.
- Can every signal be opened at its page?
A cited page, in the original language, with the stored copy and the date it was read.
- Does a second reader check the source before a forecast exists?
Ask what happens when the quote is not on the cited page, and how often that occurred in the test.
- Will you show calibration by band, with counts?
Shares without counts cannot be judged; small bands swing.
- Will you list every miss with its cause?
Watch-list gap, no public trail or access gap.
- Which languages are read in the original?
And is the translation checked against the original before a figure is used?
- Does the tool ever log in, or contact anyone?
Public sources only is the safe answer.
- What does it write to our CRM, and when?
Nothing before a person approves is the defensible answer.
Where RFP Tracking fits, and where it does not
RFP Tracking fits public plans and other asset owners that publish board documents, in any language: its agents read agendas, packets, minutes, meeting video transcripts, procurement notices and consultant announcements of the plans on a firm's watch list, and every forecast carries its page, a second agent's re-read, a probability with its reasons and a calibration history. It suits teams whose problem is the reading: too many boards, too many pages, too many languages for the people available.
It fits less well where searches leave no public trail. Corporate pension plans governed by ERISA do not hold public board meetings. Their annual Form 5500 filings, which for large plans include a Schedule C naming service providers paid $5,000 or more, become public data the Labor Department describes as covering "approximately 800,000 retirement and welfare benefit plans" (DOL, Form 5500 datasets), but a filing is due by "the last day of the 7th calendar month after the end of the plan" year (2025 Form 5500 instructions): a list of who was hired, not a warning of a search. Consultant-run searches for plans that publish nothing are in the same position. For those, consultant coverage remains the only early signal.
The cleanest way to decide is the test on this page, run on your own plans. To see what a back-test looks like in the solution, start from RFP Tracking; the KPIs it reports afterwards are set out in why average RFP win rate is the wrong first KPI.
Questions people ask
What is the best RFP tracking software?
The one that scores best on your own plans in a back-test: recall on last year's relevant RFPs, median lead time, calibration by band and an explained list of misses. Lists of issued RFPs and tools that read board documents answer different questions, and many teams use one of each.
What is an RFP alert service?
A service that reports institutional RFPs and search announcements as they are issued, usually by email or a searchable list. It is a net for searches you might otherwise miss, but by design its lead time before an RFP is close to zero, because it starts from the RFP or the announcement.
How long should a back-test cover?
At least twelve months, and longer if the watch list is small. The sample firm's twelve months produced 61 relevant RFPs, where one search is worth more than a point and a half of recall; differences of one or two searches between tools are noise at that size.
What is a Brier score?
The mean squared difference between probability forecasts and what happened, with each outcome scored 1 or 0. It comes from Glenn Brier's 1950 paper on verifying weather forecasts. A perfect forecaster scores 0 and one who always says 50% scores 0.25, so it rewards both sharp and well-calibrated forecasts.
Can an RFP tracker find searches at corporate pension plans?
Rarely before the fact. Corporate plans do not meet in public, and their Form 5500 filings, due by the last day of the seventh month after the plan year before any extension, show service providers already hired. Consultant coverage is the early signal for those plans.
Can RFP tracking tools read documents in other languages?
Some can. Ask whether the tool reads the original or a machine translation, and whether figures are checked against the original. In the solution's sample, a Japanese plan's tender notice was held for a person because the asset class in the sentence and in the reference line did not agree.