How do you pilot AI deal screening before buying it is a question every head of underwriting should be asking a vendor before a demo, not after a contract. The honest answer does not start with a live deal; it starts with deals the fund has already decided, because only those have a signed answer to check the tool against. Botminds runs Private Credit Deal Screening this way by design: the product goes live only once it has been built from a fund's own mandate, add-back policy and screening template, and tested against a sample of the fund's own past memoranda and decisions. This guide walks through what that pilot actually looks like, week by week, and the pass bar a fund should set before it starts, not after the scores come in.
Why your own closed deals are the pilot's gold set
A pilot that only ever shows a vendor's tool live deals is testing plausibility, not accuracy, because nobody yet knows the right answer for a deal that has not been decided. A fund's own closed, passed and amended deals are different: each one carries a signed screen, a leverage number the fund actually relied on, and a reason for every add-back treatment an underwriter chose. Replaying the tool against that record is the only test where a disagreement has a known right answer to check it against.
It also tests the tool on the fund's own mandate, not a generic one. Private credit's own opacity makes this more important than it would be for a public-markets tool: the Financial Stability Oversight Council notes that "a lack of secondary market trading and limited available information on borrower fundamentals can make timely valuations challenging and result in limited transparency into the sector" (U.S. Department of the Treasury, FSOC 2025 Annual Report). A tool trained or tuned on public, liquid-market data has no particular reason to classify a private credit add-back the way your own underwriters do; only your own deals can show whether it does.
What to bring: mandate, add-back policy, screening template, twenty past memos and decisions
A pilot needs specific inputs, not a general sense of how the desk works. The fund's current mandate — leverage limits, add-back policy, excluded sectors, approval thresholds — has to be the actual document underwriters work from today, not last year's version. The screening template defines what a finished screen should contain. And the sample of past deals has to be large enough and varied enough to mean something. Twenty deals is a sensible floor: at around five add-backs a memo, as on the sample Project Lantern, that is about a hundred add-backs, and if none is falsely accepted the statistician's rule of three (Hanley and Lippman-Hand, JAMA, 1983) puts the 95% upper bound on the false-accept rate near 3%; thirty deals bring it near 2%. The deals should span several sponsors, sectors and outcomes, closed, passed and at least one amended, so the pilot is not graded only on the easy cases.
What to bring to a deal-screening pilot
- The fund’s current mandate
Leverage limits, add-back policy, excluded sectors and approval thresholds, as written today, not a prior version.
- The screening template
What a finished one-page screen has to contain, so the pilot is scored against a real deliverable.
- Twenty or more past deals, with their decisions
A mix of closed, passed and amended deals, across several sponsors and sectors, each with the add-back treatment an underwriter signed off on.
- The underwriters who signed those deals
They are the ones who can say, line by line, whether the tool’s replayed sort matches what they actually decided and why.
- A written pass bar, agreed before the replay starts
Set out below — agreeing it after the scores are in defeats the purpose of having one.
The replay week: what to compare, and how to read a disagreement
The replay week takes the twenty past deals and has the tool screen them again, without showing it the signed answer, then compares the two line by line: the fund's own adjusted EBITDA and leverage, each add-back's treatment, the mandate and portfolio tests, and the red flags raised. A match is not interesting on its own; what matters is what happens where the tool and the signed file disagree.
The pilot's replayed screen, next to the underwriter's own signed file
The tool's replayed screen: Sorted into accept, haircut or reject under the mandate’s written policy, cited to a page.
What the underwriter actually signed: The treatment the underwriter actually used, and any note on why a borderline add-back was decided the way it was.
The tool's replayed screen: Recomputed on the tool’s own EBITDA figure, against every credit and portfolio limit.
What the underwriter actually signed: The leverage number and test results the fund actually relied on when it closed, passed or amended the deal.
The tool's replayed screen: Whatever the tool’s reading of the memorandum and QoE surfaces.
What the underwriter actually signed: What the underwriter actually raised in diligence, whether or not it appears in the signed screen itself.
A disagreement is not automatically the tool's fault. On some deals the replay will show the underwriter missed an add-back issue the tool caught; on others it will show the opposite. Reading every disagreement with the original underwriter, rather than scoring matches and misses mechanically, is what turns the replay week into a real test of the tool and a useful check on the desk's own past practice at the same time.
The live week: running beside the desk, not instead of it
Once the replay week clears the fund's own bar, the next test is live memos, but run beside the existing desk process, not in place of it. For a defined window, one to two weeks is enough on most desks' volume, every new memorandum gets screened both the old way and through the tool, with the desk's existing sign-offs still the ones that count. The comparison is the same as the replay week's, except the right answer is not known yet; the point of the live week is to see whether the tool keeps performing once the deals are not ones it, or the people grading it, already know the outcome of.
Setting a pass bar that can actually fail
A pass bar agreed on after the scores are in is not a pass bar; it is a rationalization with a number attached. The bar has to be set, in writing, before the replay week starts, and it has to be specific enough that a real result could fail it. A workable bar for a deal-screening pilot sets a floor on the add-back match rate against the twenty-deal sample, a stricter ceiling on the false-accept rate specifically — because an add-back wrongly accepted inflates EBITDA and understates leverage on a live deal — and a near-100% requirement that every classification cite a page, since an uncited sort cannot be checked by the underwriter relying on it.
A pass/fail bar, written before the pilot starts
| What is measured | What the fund requires | Who sets it |
|---|---|---|
| Add-back match rate against the signed files | A floor, agreed with underwriting before replay starts | Head of underwriting |
| False-accept rate on EBITDA add-backs | A stricter ceiling than the match-rate floor implies on its own | Model risk or compliance |
| Page-cited rate on every classification | Close to 100%, so a sort can be checked, not only trusted | Model risk or compliance |
| Mandate and portfolio test agreement | Exact match with the signed file’s pass/fail result on every test | Head of underwriting |
Why "not yet fully tested" performance makes the pass bar matter more
The case for a strict, written bar rather than an informal "it looked good" is the same case regulators make about leveraged lending generally. The 2022 Shared National Credit review warned that "the performance of these leveraged loans has not been fully tested in a stressed economic environment that may contribute to higher default and lower recovery rates" (Board of Governors of the Federal Reserve System, OCC and FDIC, Shared National Credit Program review, February 2023). A classifier's own performance carries the same caveat: a replay week run during a calm credit cycle, on deals that mostly closed cleanly, says little about how the tool handles a sponsor under real pressure to dress up its numbers. The stricter the bar, and the more varied the twenty-deal sample, the less a fund is relying on conditions holding steady.
Who signs off on the pilot's result
A pilot's result is itself a decision, not a report that speaks for itself, and it needs the same kind of named sign-off the product asks of every underwriting decision it drafts. A model risk or compliance reviewer should be the one who states, against the written bar, whether the tool passed; the head of underwriting is the one who decides what scope, if any, it runs at live. That split matters for the same reason the interagency guidance on buying software from a third party gives: the OCC's own bulletin asks a bank to scale its review to "the bank's risk profile and complexity as well as the criticality of the activity supported by the third party" (Office of the Comptroller of the Currency, Bulletin 2023-17, June 2023), and underwriting is about as critical an activity as a fund has.
What happens after the scores come in
A person decides
Model risk or compliance reviewerReads the replay and live-week scores against the written pass bar, and states plainly whether the tool cleared it.
A person decides
Head of underwritingDecides whether to run the tool on live deals at full scope, at reduced scope, or not yet.
A person decides
VendorFixes what the pilot found before a second pilot runs, if the first one did not clear the bar.
NIST's AI Risk Management Framework states the same principle for any organization relying on an AI system, not only a bank: that "policies and procedures are in place to define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems" (National Institute of Standards and Technology, AI Risk Management Framework 1.0). A pilot with no named person accountable for reading its result is not oversight; it is a demo with a spreadsheet attached.
A thirty-day plan, week by week
Thirty days is a realistic span for a pilot that is thorough without stalling a live decision for a quarter. The structure below assumes the fund has already gathered the twenty past deals and agreed the written bar before day one.
Thirty days: replay, live, and the decision
- Replay week7 d
- Score and review disagreements3 d
- Live week, running beside the desk10 d
- Score the live week3 d
- Sign off and decide scope7 d
What the live week adds to the pipeline
Where the thirty days ends depends on what the scores show. A pilot that clears the written bar on both the replay and live weeks is a straightforward decision to go live, usually in stages rather than across the whole pipeline on day one. A pilot that clears the bar on replay but not on the live week points to a gap between how the tool performs on known deals and how it performs on new ones, which is worth another live window before expanding scope. And a pilot that does not clear the replay bar at all should not reach a live deal.
What a fund keeps from every pilot, pass or fail
Questions people ask
What data do you need to run a pilot?
The fund's current mandate and add-back policy, its screening template, and at least twenty past deals — a mix of closed, passed and amended — each with the add-back treatment and mandate test results an underwriter actually signed off on. Without a signed answer to compare against, a pilot cannot tell you whether the tool is right.
What is a pass/fail bar for an AI underwriting pilot?
A written threshold, agreed before the pilot starts, that states exactly what result counts as good enough: typically a floor on the add-back match rate against your own past deals, a stricter ceiling on the false-accept rate specifically, and a near-100% requirement that every classification cite a page. Agreeing the numbers after the scores come in defeats the purpose of having a bar at all.
How many past deals should a pilot replay?
Twenty is a reasonable floor, and more matters more than a round number: the sample should span several sponsors, sectors and outcomes, including at least one amended deal, so the pilot is not graded only on the cases that were easy to get right.
What should match between the AI's screen and your own?
The fund's own adjusted EBITDA and leverage figure, the treatment of every EBITDA add-back, the result of every mandate and portfolio test, and the red flags an underwriter actually raised in diligence. A disagreement on any of these is worth reading with the underwriter who signed the original file before deciding whose read was right.
How long should a private credit AI pilot run?
Thirty days is a realistic span: about a week to replay past deals, a few days to score and review disagreements with the underwriters who signed them, a week or two running live beside the existing desk process, and a week to score that live window and decide scope. A thinner sample of past deals or a stricter bar should extend the plan, not shorten it.
What happens if the pilot fails its pass bar?
The tool does not go live, at least not at the scope tested. A fund should expect a vendor to fix what the replay or live week found and run a second pilot, scoped to the specific gap — a sector, an add-back category, a mandate test — rather than treating a single failed pilot as proof the whole approach cannot work, or waiving the bar to get the deal flow it was meant to protect.