Credit agreement review software is bought on a demonstration and lived with on a book. The demonstration is almost always impressive, because reading one clean agreement and quoting a clause is the part of this problem that is solved. What a buying team actually needs to establish is narrower and harder: whether the tool reads each loan as amended by every signed document in the order each took effect, whether the grading standard is the fund's own and not the vendor's, whether every finding carries the clause it rests on, and what the fund can show afterwards if a finding was wrong. LME Protection Review, Botminds' software for private credit funds, is built around those four answers — and this guide is written so that it can be judged against them like anything else on a shortlist.
The sample figures below come from a fictional book inside the solution: Marrowstone Credit Partners, 64 loans, 216 documents, 129 amendments. The supervisory material is quoted from the federal banking agencies' and NIST's own publications, linked under each figure.
Credit agreement review software: the question a demo agreement can't answer
A demonstration uses one agreement, chosen by the vendor, usually clean, usually in text rather than scanned, and usually without the eleven amendments that have modified it. Nothing about that exercise tests the work. The work is a whole book. On the sample book in this guide that is 64 agreements, 129 amendments and 216 documents in all, and any real book adds the awkward cases: scanned amendments, amendments and restatements that replace earlier documents, waivers with no expiry on their face, and borrower requests for amendments that have not been signed and must not be applied.
So the first question to ask is not about accuracy. It is: can we run this on twenty of our own loans, with their full document stacks, before we sign anything? A vendor that cannot do that is asking to be judged on the easy case.
Does the software read amendments, or just the original agreement?
This is the question that eliminates most of a shortlist, and it has three parts, because “yes, we read amendments” is true of almost everything and sufficient for almost nothing.
First, are amendments applied in the order each took effect, by effective date rather than by signature date or file date? Second, is the output the clause as it now reads, rather than the original clause with a note that it was amended? Third, can a reader see the clause at each stop in its history — what it said originally, what each amendment did to it, and what governs today? A tool that answers the first two and not the third is usable; one that answers none of them is producing confident statements about documents that stopped governing years ago. Why that matters so much on this particular work is set out in how a credit agreement drifts from what you signed.
The related question is what the tool does with a document that is not yet operative. A borrower's request for an amendment is not the agreement. It should be read, it should raise an alert, and it should sit visibly outside the applied stack until it is signed and effective.
Can you bring your own protection checklist, not the vendor's?
A grade is a comparison against a standard, so whoever owns the standard owns the grade. If the standard ships with the software, the fund has outsourced its own definition of a strong protection to a company that has never seen its portfolio, its vintages or its sponsors — and when the standard is revised, the book is regraded by a release note.
Graded against the vendor's standard, or against your own
The vendor's built-in checklist: The vendor, from market practice as it reads it.
The fund's checklist, owned by counsel: In-house counsel, from the fund's own credit and documentation policy.
The vendor's built-in checklist: On a product release, usually without a decision by the fund.
The fund's checklist, owned by counsel: When counsel approves a new version, which is kept and dated.
The vendor's built-in checklist: Grades moved; the reason sits in someone else's change log.
The fund's checklist, owned by counsel: The book was regraded against version 8, and the difference from version 7 is readable.
The vendor's built-in checklist: A fixed list, the same for every customer.
The fund's checklist, owned by counsel: The fund's choice — on this sample book, five of the eight are marked must-have.
The vendor's built-in checklist: “The software said it was weak.”
The fund's checklist, owned by counsel: “It is weak against our standard, here is the clause, and here is the element it is missing.”
Ask to see the settings screen in the demonstration, not just the answers. The question behind the question is whether the standard is data the customer owns or logic the vendor ships.
LME Protection Review: the grading standard as a setting the fund owns
Whose responsibility is the finding, if the software is wrong?
The fund's. That is not a contractual observation, it is how supervisors treat the same arrangement in the part of the financial system that is supervised for it. The federal banking agencies' 2023 interagency guidance on third-party relationships states the principle without qualification.
Whether activities are performed internally or via a third party, banking organizations are required to operate in a safe and sound manner and in compliance with applicable laws and regulations.
Federal Reserve, FDIC and OCC, Interagency Guidance on Third-Party Relationships: Risk Management, 88 FR 37920, 9 June 2023
The same guidance adds that a banking organization's “use of third parties does not diminish its responsibility to meet these requirements to the same extent as if its activities were performed by the banking organization in-house”. A private credit fund or a business development company is not a banking organization and this guidance does not bind it; it is quoted here because it is the standard the banking system holds itself to for exactly this decision — handing a piece of work to an outside party and remaining answerable for the result. An investment committee that would not accept “our vendor's model said so” from a bank should not accept it from itself.
What follows practically is that the fund needs its own evidence, not the vendor's assurance: a sample of findings reread against the agreements, by someone who can read them, on a schedule.
Is every finding quoted to its clause, document, section and page?
This is the single most testable property in the category, and the one most worth refusing to compromise on. A grade without a citation cannot be checked; a grade with one can be checked by counsel in under a minute. The target in this software is that 100% of findings carry the document, section and page they rest on, and that clicking a finding opens the agreement at the highlighted passage.
Test it adversarially rather than by asking. Pick a finding in the demonstration, open the agreement at the cited page, and read the surrounding text. Then ask a question about a protection the agreement does not contain and see whether the answer is “there is no such provision” or a paragraph describing one.
What does the tool do with a document it cannot match or read?
Every real book has them: a scanned amendment with a missing page, a joinder whose cover sheet does not name the borrower the fund knows, an intercreditor agreement filed under the wrong loan. The question is whether these become visible exceptions or silent omissions. A tool that reports “three documents are unmatched and are not included in these answers” is telling the truth about its coverage. A tool that simply answers from what it could read is giving a complete-looking answer from an incomplete book, and there is no way to tell from the output which you received.
What ongoing monitoring of a software vendor should look like
Buying is the start of the obligation, not the end of it. The interagency guidance sets out what continuing oversight of a third party involves, and the structure transfers cleanly to a fund even though the rule does not apply to one.
The agencies' ongoing-monitoring structure, applied to a credit agreement review tool
| What the guidance describes | What it means for this software | How often |
|---|---|---|
| Confirm the quality and sustainability of the third party's controls and its ability to meet contractual obligations. | Reread a sample of graded clauses against the agreements: does the quotation exist, and does the grade follow from it? | Quarterly, and after any change to the grading standard. |
| Escalate significant issues or concerns — material or repeat findings, deterioration, compliance lapses. | A log of findings that counsel corrected, with the reason, so a pattern of one kind of error becomes visible. | Continuously; reviewed at each quarterly check. |
| Review of reports regarding the third party's performance and the effectiveness of its controls. | Coverage reporting: how many documents were applied, how many held as pending, how many unmatched. | Every run, and summarised for the committee. |
| Regular testing of the organisation's own controls that manage the risks from the relationship. | Test the fund's own process: did counsel actually approve the checklist version in force, and every ask that left the fund? | Annually, or on any change of personnel in the chain. |
Note what is absent from that table: anything requiring a data science function. The agencies' own model risk letter makes the same point about bought-in software — “an important element of model risk management is the validation of vendor products, either by internal or outside parties”, and the sound practice it describes is understanding the product and monitoring its performance, not reproducing it. For this kind of tool the qualified reviewer is a lawyer with the agreement open.
How is an AI vendor's output audited?
By keeping the record as the work happens, because it cannot be reconstructed afterwards. Three things need to be on the record: every question asked and the answer given, with the documents and the checklist version in force at the time; every grade change, with what caused it — a new signed amendment, a revised checklist, a correction by counsel; and every ask that left the fund, with who decided and who approved the drafting.
Those three make an audit possible in the ordinary sense: someone who was not there can establish what was known, when, and on what basis. Without the checklist version, a historical grade cannot be interpreted at all, because the standard it was measured against is gone.
NIST's AI framework adds the human half of the same requirement, asking that “processes for operator and practitioner proficiency with AI system performance and trustworthiness — and relevant technical standards and certifications — are defined, assessed, and documented”. In plain terms: write down who is competent to check this output, and make sure that is who checks it. The wider question of what the agents may and may not do is covered in the guide to agents and their limits.
What documents does credit agreement software need to read?
Less than vendors usually ask for, and more than buyers usually expect. The minimum set for protection review is the credit agreements and every amendment, waiver, consent and joinder; the intercreditor agreements that sit beside them; the position list saying what the fund holds in each loan and through which vehicle; the compliance certificates that supply the earnings figure the basket arithmetic uses; and the fund's own protection checklist and model clauses.
What is not needed is as informative. This work does not require the fund's valuations, its investor data, its returns or its pipeline, and a vendor asking for them should be asked why. Keeping the input set narrow is both a security answer and a scoping one: it is what makes a pilot possible in weeks rather than quarters, which is the subject of the go-live plan.
The protection map in LME Protection Review, as the committee will read it
A ten-question checklist for the vendor call
Ten questions, each with the reason it is on the list. A vendor who answers all ten specifically is worth a pilot; one who answers in capabilities rather than in mechanics is worth a second call.
Ten questions, and what each one is really testing
- Are amendments applied in effective-date order, and can I see the clause at each stop?
Tests whether the product models the document stack or just indexes files.
- What happens to a borrower's unsigned request for an amendment?
The right answer is: read, flagged, and not applied. Anything else changes what the agreement says.
- Whose grading standard are the grades measured against, and who can change it?
Tests whether the fund owns its own definition of a strong protection.
- Is the standard versioned, and does a new version regrade the book?
Without versions, a historical grade cannot be interpreted later.
- Does every finding carry the document, section and page, and open the agreement there?
A grade that cannot be checked by counsel cannot be used in a negotiation.
- What does the answer say when the provision does not exist?
Tests the failure mode that matters: a fluent description of a clause that is not there.
- How are unreadable or unmatched documents reported?
Distinguishes an honest coverage statement from a complete-looking answer on an incomplete book.
- Can we run this on twenty of our own loans, with full stacks, before signing?
Moves the evaluation off the vendor's chosen agreement.
- What is on the audit trail: questions, grade changes, approvals?
Tests whether an outsider could later reconstruct what was known and when.
- When a signed amendment arrives, what happens to the answers we already have?
An answer that does not refresh is a claim about the date it was produced.
If the shortlist is still tied after those ten, the tie-breaker is unglamorous: ask each vendor to produce, for one of your loans, the clause that governs today and the amendment that last changed it. Then open the documents yourself. To run that test on a sample of your own agreements, start from LME Protection Review.
Questions people ask
How do you evaluate credit agreement review software without a long pilot?
Shrink the sample, not the test. Twenty loans with their full document stacks, including a scanned amendment and an unsigned request, will expose almost everything a 64-loan pilot would: ordering, citation, coverage reporting, and what the tool says when it cannot find something. What a short evaluation cannot tell you is how the product behaves when documents arrive continuously, which is why the first live quarter still needs attention.
Should the grading standard be the vendor's or the fund's?
The fund's, held by counsel and versioned. A vendor's standard is a reasonable starting draft, but once grades are used in negotiations with sponsors, the fund has to be able to say what its own definition of a strong protection is and when it last changed. On the sample configuration here, eight protections each carry written elements for what counts as strong, and five are marked must-have.
Is credit agreement review software subject to model risk rules?
For a private credit fund, no rule applies directly: the agencies' model risk guidance is addressed to banking organizations, and its 2026 revision puts generative and agentic AI outside its scope in any event. That makes the fund's own policy the governing document, which is an argument for borrowing the structure of the supervisory material — validation, ongoing monitoring, independent challenge — rather than ignoring it.
What should be in the contract that is not in the demonstration?
The right to test: a stated ability to run the tool against your own loans, to export findings with their citations, and to take your checklist and model clauses with you. Also worth fixing is what happens to the document stack and the audit trail at the end of the relationship, because both are records of decisions the fund made.
How much document preparation does the fund have to do first?
Less than most buyers fear, but the position list is the exception. Agreements and amendments can arrive as PDFs or scans and be matched to their loans; what cannot be inferred is what the fund holds in each loan and through which vehicle, and without that the answers cannot be weighted by exposure. A tool that produces counts of loans but not dollars held is answering half the question.
What does ongoing monitoring of the software actually involve?
A quarterly reread of a sample of findings against the agreements, a log of what counsel corrected and why, coverage reporting on every run, and an annual check that the fund's own controls were followed — that the checklist version in force was the approved one, and that every ask that reached a borrower had been approved. None of it requires technical expertise; all of it requires someone to open the agreements.