Private credit portfolio monitoring software is built to watch a loan book between closing and repayment — financials, covenants, interest, marks, and every change to a loan's terms — so that a pattern of concessions shows up before a loan goes on non-accrual. The pitch is usually some version of "AI reads the documents so your team doesn't have to." The question worth asking is not whether that claim is true in general. It is what the tool does with a fact it cannot source, who is allowed to act on a score it produces, and whether a loan's original covenant is tested alongside the one currently in force. Those three questions separate a tool a credit committee can rely on from one it has to double-check anyway. This guide answers them for Botminds' Private Credit Portfolio Monitoring as well as setting them for any vendor; the loans and screens shown from it are fictional sample data.
The National Institute of Standards and Technology frames the goal plainly for any AI system used in a decision that matters: its risk management framework "is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems" (NIST AI RMF). For a monitoring tool sitting between a loan file and a watchlist committee, trustworthiness has a narrow, checkable meaning: does it show its work, and does it know where its own authority ends.
What should private credit portfolio monitoring software actually do?
At minimum, it should read the documents a loan generates — the credit agreement and every amendment, waiver and consent; the administrative agent's interest notices; compliance certificates and financial statements; valuation marks; and sponsor or lender-call updates — and turn them into a loan history a person can act on, not a loan snapshot. The output that matters is not a single risk number. It is a timeline of concessions, a covenant tested on more than one definition, and a stress score whose drivers each point back to a document. A tool that cannot show which amendment moved a score from 74 to 81 is not meaningfully different from a spreadsheet someone filled in by hand, except that nobody can check its arithmetic.
Evaluation questions, grouped by what a bad answer looks like
- Does every number on the screen link to the document it came from?
A score, a headroom figure or a liquidity number with no link is an assertion, not an answer. Good: a click opens the credit agreement page or the certificate. Red flag: "our model is proprietary and highly accurate" with nothing to open.
- Does it quote the section, or just name the document?
Naming "the credit agreement" is not a citation. Good: "Section 7.1(a), as amended by Amendment No. 3." Red flag: a source list at the bottom of a report instead of a link on each figure.
- Can it change a risk rating, a mark or non-accrual status on its own?
Good: no — it proposes, and a named person decides. Red flag: a "confidence threshold" above which the tool updates the rating without a human step.
- Does it test a covenant on the definition at closing, not only the version currently in force?
A reset test is the one a vendor demo will show passing. Good: it recomputes on the original level and add-back cap too, and shows both. Red flag: one ratio, one pass/fail, no mention of the original definition.
- What happens when it cannot find a source for a fact?
Good: it says so and leaves the field blank or flagged for a person to fill in. Red flag: it fills the gap with an estimate and presents it the same way as a cited figure.
- How often does it re-read the loan file?
Good: on a schedule you set — nightly, on every new document, or both — stated plainly. Red flag: "continuously" with no description of what triggers a re-read.
- Does it separate interest in kind designed at closing from interest in kind added later?
The two mean different things for stress. Good: tracked apart, with only the added kind feeding a score. Red flag: one "PIK percentage" that blends both.
- Who can change the scoring weights, and is every change recorded?
Good: a named role, versioned, with the old and new weights both visible later. Red flag: a settings screen with no history.
- Can you see what changed when a reset test replaced the original one?
Good: the old limit and the new one, side by side, dated. Red flag: only today's limit is shown, with no way to see what it used to be.
- Does it draft a committee note, or write a conclusion?
Good: a draft a named reviewer edits and signs, every figure cited. Red flag: a finished-sounding memo with no visible author and no sign-off step.
- What does it do with a document it cannot parse?
Good: flags it for a person, with a reason. Red flag: silently skips it, with no record that a document went unread.
- Can you run it on last quarter's closed loans before it touches a live file?
Good: yes, and it will show you where its read of a closed quarter agreed or disagreed with what your team actually decided. Red flag: a vendor who only wants to start on live data.
- Does the valuation decision stay separate from the stress score?
Good: the tool sends evidence — mark against peers, liquidity, sponsor news — and the valuation team keeps the number. Red flag: a tool that proposes or adjusts a mark itself.
The first question: does it cite its source, every time?
Citation is the cheapest thing to check in a demo and the easiest thing for a vendor to skip. Ask to click on any number on the screen — a leverage ratio, a liquidity figure, a driver's points — and see what opens. The honest answer is the credit agreement, the certificate or the interest notice, at the section that produced the figure. An answer that opens "see appendix" or does not open anything is telling you the number was typed in by a person at some point upstream, or computed by a model nobody can trace back to a document. Six months later, when a committee member asks why a loan is rated the way it is, "the model said so" is not an answer a chief investment officer can take to an examiner.
A scoring rubric for the demo: what a good answer and a red flag look like
| Question | Good answer | Red flag |
|---|---|---|
| Does it cite a source? | Every figure opens the document and section it came from. | A number with no link, or a link to "see appendix." |
| Who decides? | The tool proposes; a named person signs before anything reaches a committee or a regulator. | A rating, mark or non-accrual status that updates on its own above a confidence threshold. |
| What is scored? | Interest in kind added by amendment counts; interest in kind designed at closing is tracked, not scored. | One blended PIK number that cannot be unpicked. |
| What happens on a gap? | A missing fact is flagged, not estimated and presented as fact. | A field is filled in without saying how. |
| How often does it look again? | A stated schedule — nightly, or on every new document. | "Continuously," with no description of the trigger. |
Does it decide, or does a person still decide?
This is the question that separates a monitoring tool from an underwriting one, and it is where a vendor's marketing language is least precise. "The AI flags the loans that need attention" sounds like the tool decides nothing, but the same sentence is also compatible with a tool that silently updates a rating and calls the update a flag. Ask directly: can the software change a risk rating, a valuation mark, or move a loan to non-accrual on its own, above some confidence threshold it sets itself? The answer you want is a flat no, followed by a description of what it does instead — propose, with a citation, to a named person who signs before anything moves.
In the working example behind this guide, the agents divide a quarterly review six ways and never cross that line: the stress scorer "never puts a loan on the watchlist or non-accrual," and the committee note writer "never sends anything without the portfolio manager's signature." What each of the six agents does, and the line it may not cross, sets out the full division of labor.
What documents does it need to read, and how often?
A monitoring tool is only as current as the documents it has actually read. Ask what it ingests — credit agreements and amendments, the administrative agent's interest and PIK notices, compliance certificates, valuation marks and memos, lender call notes, sponsor and news updates — and then ask the harder question: on what schedule does it re-read them. "Continuously" is not a schedule; it is a word that avoids one. A concrete answer names a trigger: new documents are read as they arrive, and every loan is rescored on a fixed cadence, so a certificate filed this morning is reflected in tomorrow's numbers, not whenever someone next opens the file.
Can it test a covenant on its original definition, not just the reset one?
This is the question most vendor demos are not built to answer well, because a demo usually shows a covenant passing its current test. A fund that has reset a covenant once already knows the current test is the easy one to pass; the harder, more honest question is what the loan looks like against the covenant written at closing, before any reset. A tool that only reports today's definition will show a loan performing right up until the quarter it goes on non-accrual. One that recomputes both will show the gap — a borrower at 6.57× on a reset 6.75× test and at 7.39× against the original 5.75×, failing for six straight quarters — long before that.
Why the score is 81: every driver cites the document behind it
What NAIC's own transparency concerns mean for a vendor demo
Insurance regulators, who hold private credit through their members' portfolios, have already written down what worries them about this asset class, and it reads like an evaluation checklist for monitoring software. The National Association of Insurance Commissioners notes that "some private credit investments, however, can be harder to value, less liquid, or more complex than traditional public bonds" (NAIC), which is exactly the territory a monitoring tool operates in: marks, liquidity and complex terms that do not reduce to a single clean number. The U.S. Treasury's Office of Financial Research adds the systemic version of the same worry, noting that the growing links between banks and private credit funds mean "this channel merits close monitoring given the industry's rapid growth" (OFR Brief 26-02). Neither body is talking about a specific product. Both are describing the gap a transparent tool is supposed to close: a complex, hard-to-value asset that needs to be seen clearly, not simplified into a score nobody can unpick.
Evaluation questions, grouped by what a bad answer looks like
A useful evaluation does not ask "is your AI good" and hope for specifics. It puts the same handful of concrete questions in front of every vendor and compares the answers side by side. The checklist above groups thirteen of them by what a bad answer looks like, because a vendor who has thought this through answers in specifics — a section number, a named role, a stated schedule — and a vendor who has not answers in adjectives.
A transparent answer against an evasive one, on the same three questions
What a vendor says: "Our model has a 94% accuracy rate." No comparison set is named, no period, no definition of accurate.
What a transparent answer looks like: "On the fund's own last four closed quarters, the score agreed with the committee's actual decision on 51 of 58 loans; the seven disagreements are listed with the document each side relied on."
What a vendor says: "The AI flags loans that need attention." Does not say who, if anyone, has to act on the flag.
What a transparent answer looks like: "No. It proposes a rating change with its reasons cited; the chief investment officer approves before it takes effect, and the proposal and the decision are both on the record."
What a vendor says: "Real-time monitoring." No mention of what triggers an update or how a document gets read.
What a transparent answer looks like: "New documents are read as they arrive, and every loan is rescored each night; a document uploaded this morning is reflected by tomorrow's run."
Running it on your own closed loan files before you trust it live
The single best test costs nothing extra and no vendor should resist it: hand over a quarter you have already closed, with the amendments, certificates and committee decisions you already made, and ask the tool to replay it. A tool worth buying will show you, loan by loan, where its read agreed with your team's actual decision and where it did not — and for the disagreements, which document each side relied on. A vendor who wants to start on live data instead, with no comparison to check against, is asking you to trust the output before you have any way to verify it. A week-by-week go-live plan starts from exactly this kind of replay, for the same reason.
From a document to a decision: where the software stops and a person starts
Agent
Document arrivesA compliance certificate, an interest notice, an amendment or a lender call note is read and classified.
Agent
Recomputed and scoredCovenants are tested as reported and on the original definition; interest in kind is split designed from added; a driver-by-driver score is produced, each driver linked to its source.
Agent
ProposedA watchlist entry, a rating change or counter-terms to a request are drafted, with the fund's own precedents cited.
Never: it does not send a proposal anywhere on its own
A person decides
SignedThe portfolio manager reviews the proposal against the evidence and signs, with the meaning of the signature recorded.
A person decides
DecidedThe committee chair approves or declines a rating change, a watchlist addition or non-accrual; the valuation team, not the tool, keeps the mark.
What to ask about change control on the scoring settings
A stress score is only as stable as the weights behind it, and those weights are a policy choice, not a fact — in the solution behind this guide, concessions, the original covenant, interest in kind added by amendment, liquidity, sponsor support, mark against peers and reporting carry seven different weights, settable within a published range. Ask who can change those weights, whether a change is versioned, and whether you can see a loan's score under the old weights as well as the new ones after a change. A tool with no change history on its own settings is asking you to trust a number without being able to explain, a year later, why it moved.
Settings as a fund's own policy, not the vendor's default
Judged this way, the evaluation stops being about whether the vendor's AI is impressive in a demo and becomes about whether its answers would survive being read back to an examiner a year from now. Private Credit Portfolio Monitoring is built around that bar: every driver cited, every proposal signed by a named person, and the committee, not the software, deciding what happens next.
Questions people ask
Does monitoring software replace the portfolio manager?
No, in any tool built to this standard. The software reads documents, recomputes covenants, ties interest and produces a cited score; a named person — typically the portfolio manager — reviews and signs before a proposal reaches a committee, and the committee chair decides on rating changes, the watchlist and non-accrual. The work that moves is the mechanical reading and re-spreading, not the judgment.
What is the difference between covenant software and portfolio monitoring software?
Covenant software typically re-tests the ratios in a credit agreement against a borrower's certified figures each period. Portfolio monitoring software does that and adds the history around it — every waiver, reset, interest-in-kind switch and extension, tied together across quarters, and scored as a pattern rather than judged one certificate at a time.
Can monitoring software change a loan's risk rating itself?
In a properly bounded tool, no. It can propose a rating change with its reasons cited, but the change itself is a decision a named person approves — typically the chief investment officer or the equivalent committee chair — and that approval is on the record separately from the proposal.
How is AI monitoring software validated before go-live?
By replaying it against quarters the fund has already closed and comparing its scores and flags with the decisions the committee actually made at the time, not by taking a vendor's accuracy claim on faith. How to validate an AI credit risk score before you trust it works through the three-part test model risk teams use for exactly this.
What questions should I ask a monitoring software vendor?
Start with the three that separate a transparent tool from an opaque one: does every figure cite the document it came from, can the software itself change a rating or a mark, and on what schedule does it re-read the file. A vendor who answers all three in specifics, not adjectives, has usually thought the rest through as well.