SystemCollections, Recovery & Special Assets

Buying guide · reading the evidence, not managing the file

AI vendor management software, tested on a collection agency panel

Vendor management software typically manages the relationship — contracts, questionnaires, certificates, review dates — and never opens a call or a remittance line. On a collection agency panel the risk lives in that evidence, so the AI worth buying is the kind that reads all of it.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

AI vendor management software is software that applies machine reading to the evidence a third party produces, so that a bank's monitoring covers all of it rather than a sample of it. That is a narrower claim than the category usually makes, and the distinction is the whole buying decision: many vendor management tools manage the relationship — the contract, the questionnaire, the certificate, the review date, the risk tier — and never open a call recording or a remittance line. On a collection agency panel, where the firms talk to the bank's customers on the bank's behalf, much of the risk is in exactly that evidence. Botminds' Collection Agency Management is built for that half: six readers that take in recordings, remittance files, complaints, audit answers and contracts, and produce one graded report card a person signs.

This page is written for the committee paper. It sets out what each reader should do, what it must never be permitted to do, how to test an accuracy claim on your own month of files, and what does not change no matter what you buy. The worked figures come from one week of a fictional sample panel inside the solution — Bellmere Bank, six agencies, 38,420 charged-off accounts — and are sample data, not a customer's measured results.

What does AI vendor management software do?

In the market as it is sold, the phrase covers three fairly different things, and a buyer who does not separate them will compare prices for products that do not overlap.

The first is relationship administration: a repository for contracts, a workflow for due-diligence questionnaires, a calendar of review dates, a risk tier per vendor, and reminders when a certificate expires. The AI in this layer typically summarises a document, drafts a questionnaire, or extracts dates and obligations from a contract. It is genuinely useful and it is not monitoring; nothing it does would have noticed a script that dropped a required sentence.

The second is external risk signal: adverse media, financial health, breach and sanctions screening on the vendor as a company. It answers whether the firm is in trouble, not whether the firm did its job properly on your accounts last week.

The third — the one this page is about — is evidence reading: taking in what the vendor produces in the course of the work and testing each item against a rule or a contract. On a collection panel that means call recordings and dialer logs, remittance and activity files, complaints, audit responses and the contracts the first two are judged against.

Managing the relationship, or reading the evidence

Relationship managementEvidence reading
What it holds

Relationship management: Contracts, questionnaires, certificates, risk tiers, review dates, issues and actions.

Evidence reading: The work product: recordings, dialer logs, remittance files, complaints, audit responses — and the contracts, to test the rest against.

What it can tell you

Relationship management: Whether the vendor is approved, in tier, in date and has answered.

Evidence reading: Whether the vendor did what it promised, on which account, at which minute.

Its unit of work

Relationship management: The vendor.

Evidence reading: The call, the remittance line, the complaint, the filing.

How it fails

Relationship management: Quietly and on time: everything is green because everything was answered.

Evidence reading: Noisily: it raises more than a person can confirm, unless the thresholds are set properly.

What it costs to be wrong

Relationship management: An out-of-date certificate.

Evidence reading: A month of calls that broke a rule, found at the annual audit instead of on the Monday.

Who already owns it

Relationship management: Vendor risk or procurement, usually across every vendor the bank has.

Evidence reading: The business that placed the accounts, because only it knows what the evidence should look like.

Our split of a category sold as one thing. Many programmes need both; the mistake is buying the first and recording in the committee paper that monitoring is covered.

The split that matters: managing the relationship, or reading the evidence

The supervisory guidance is not neutral between the two. Its typical monitoring activities are three: reviewing reports on the third party's performance and the effectiveness of its controls, periodic visits and meetings with the third party's representatives, and regular testing of the bank's own controls over the relationship, especially where it supports higher-risk or critical activities. It then adds the point buyers should underline: depending on the risk, a bank may in some cases test the third party's own controls directly (Interagency Guidance on Third-Party Relationships, June 2023).

Direct testing of the third party's own controls is what reading the evidence is. A monthly compliance report from an agency feeds the first activity; listening to the agency's calls and counting its attempts is the direct testing the last sentence describes. The guidance also says the intensity should rise with risk — monitoring can be periodic or continuous, and higher-risk activities call for more of it, more often. Collections tends to rank high on a bank's conduct-risk register because it is customer contact under pressure, by employees of another company.

Can AI monitor a collection agency's performance?

It can read what the agency produced and test it against a stated rule, which is most of what monitoring is. It cannot decide what the rule should be, and it should not be allowed to decide that a breach occurred. The honest claim is narrow and worth insisting on in writing: coverage and consistency, not judgement.

Coverage is the real gain. Reading every recorded call turns detection from a probability into a property of the method — for calls that were recorded. Consistency is the second: the same test is applied to the sixth agency on a Friday afternoon as to the first on Monday morning, which is not true of a human reviewer working through a sample.

What it does not give you is a judgement that survives contact with the agency. A finding has to be defended with a specific second of a specific call, and the person who defends it has to have heard it. That is why the design that matters is not how clever the reading is but where the person is placed in the chain.

Evidence in, one graded report card out

  1. Agent

    Call reviewer

    Reads every recorded call against the duties in Regulation F, and ties each finding to the collector, the team and the script version in force that day.

    Never: contacts anyone, or decides a critical finding is true

  2. Agent

    Complaint linker

    Reads complaints from every channel — the regulator’s portal, letters, calls, state regulators — and links each to its account, agency and call.

    Never: answers or closes a complaint

  3. Agent

    Remittance reconciler

    Ties every remittance line to a posted payment and to the fee rate in force under the right amendment, and explains each variance.

    Never: moves or offsets money

  4. Agent

    Performance analyst

    Measures liquidation on like-for-like placements by vintage, and prices what a recall would give up in expected recoveries.

    Never: compares agencies on different paper

  5. Agent

    Audit evidence reader

    Reads audit answers and their evidence against each contract, and raises what is missing as an open item with a date.

    Never: closes a finding

  6. Agent

    Placement allocator

    Proposes next month’s shares from the grades, the conduct threshold and the capacity caps, and shows the accounts that move.

    Never: sends a placement file without the signatures

  7. A person decides

    The desk

    Confirms each critical finding from the call itself, decides the correction to request, signs the placement and co-signs a pause.

The six readers on the sample panel and the person at the end of them. Every arrow ends at a decision a named human makes; nothing reaches an agency before that.

What each reader checks, call by call and line by line

A committee paper should be able to say what is being tested, not that the software "monitors compliance". On the conduct side the checks come straight out of the rule text, which is what makes them arguable in both directions.

The disclosure checks follow 12 CFR 1006.18(e): the first communication must say the collector is attempting to collect a debt and that information obtained will be used for that purpose, and every later communication must disclose that it comes from a debt collector. The attempt checks follow the frequency presumption in 1006.14(b)(2)(i), which has two limbs — more than seven calls in seven days, and any call within seven days of a conversation about that debt. Call-time checks compare each attempt with the consumer's own time zone and with anything the consumer said about when to call.

On the money side the test is arithmetic rather than language: each remittance line against a posted payment, and the fee taken against the rate that the contract, as amended, puts on that pool. In the sample week that produced a September file $7,056.90 short — forty-one paid-in-full accounts billed at a secondary rate instead of the primary rate the amendment set, and three payments the bank's own records show were collected but which never appeared on the file.

On the complaints side, the work is linking rather than counting. The supervisory guidance treats complaints as a monitoring input in exactly that shape: among its monitoring factors are how many complaints and inquiries arrive, what they are about and how they trend, how well the third party responds where it handles them, and what remediation followed (Interagency Guidance, ongoing monitoring factors). Volume and nature need the complaint; adequacy and remediation need the account and the call behind it.

Every call that needs a person, by agency and severity

Call review for the week: findings listed by agency and severity, each showing the account, the collector, the date and time, the duty missed and whether it has been confirmed, with linked complaints shown beside the calls they came from and their response clocks.
Collection Agency Management on the fictional sample panel. Of 21,860 calls read, thirty-seven reached this list; the rest were read and left alone.

What it must never do: contact, close, offset

The limits are the part of the paper the committee will actually argue about, and they are easier to agree if they are written as refusals attached to a named reader rather than as a general principle about human oversight. Three run across all of them: never contact a consumer, a collector or an agency; never answer or close a complaint; never move money.

What each reader may not do, written as refusals

Call reviewer

Never

Contacting a consumer, a collector or an agency. Deciding that a critical finding is true. Grading on anything it cannot cite a timestamp for.

Complaint linker

Never

Answering a complaint. Closing one. Changing its received date or its response clock.

Remittance reconciler

Never

Moving or offsetting money. Adjusting a fee rate. Accepting a variance because it is small.

Performance analyst

Never

Comparing two agencies on different paper. Restating a prior month’s liquidation without saying so.

Audit evidence reader

Never

Closing an audit finding. Accepting an assurance in place of the document the contract requires.

Placement allocator

Never

Sending a placement file without the signatures. Placing an account outside its stratum. Overriding the conduct threshold.

The limits as they are set on the sample panel. Writing them per reader rather than as one principle makes each one testable: you can ask to see the refusal, and you can ask what happens when a user tries.

Two of those deserve a sentence of defence, because a vendor will offer to relax them. Automatic offsetting of a short remittance against the next one is attractive and wrong: a variance is a disagreement about a contract, and the lender's negotiating position is better before money has moved than after. Automatic placement changes are the same argument one level up — pausing a firm is a decision with a cost, and the cost should be seen by the person who signs.

The third limit, never deciding a critical finding is true, is the one that makes the rest usable. On the sample panel a critical finding stays provisional until the collections compliance manager opens the call and confirms it, and while provisional it does exactly one thing: holds back new placements to that firm. Nothing else moves — not the published grade, not the fee, not a letter to the agency.

Is it legal to use AI to review a vendor's calls?

The question people mean is usually about the recording, not the review. Whether a call may be recorded at all, and what notice the parties must be given, is settled at the moment of recording by state and federal law and by the notice the agency gives — and that question exists whether a person or a machine listens afterwards. Reading a recording your contract already entitles you to receive is a different act from making one.

Regulation F takes the existence of these recordings for granted and attaches a duty to them. Its retention rule may be the most useful provision in this whole area for a buyer, because it tells you the evidence will be there: a collector that records its collection calls must keep each recording for three years from the date of the call (12 CFR 1006.100(b)).

Three years from the call is a long enough window to re-run a test with a better method, which is why a valuable contract term in this area is nightly delivery of the recordings into the lender's own store. It converts the agency's retention duty into the lender's own evidence, and it is what makes an accuracy test on your own month possible at all.

What does change with machine reading is data protection and access, and those belong in the paper as controls rather than as legal questions: who can play a recording, what is logged when they do, how long the lender keeps its own copy, whether audio or transcripts leave the lender's environment, and what the contract with the software vendor says about training on your files. Ask for the answers in writing; they are all answerable.

One thing does not change. Using software does not move responsibility. The CFPB's service-provider guidance says that entering a business relationship with a service provider does not absolve a supervised bank of responsibility for complying with federal consumer financial law to avoid consumer harm, and that depending on the circumstances legal responsibility may lie with the bank as well as the provider. A reading tool changes what the bank knows, not what it answers for — which is an argument for buying one, not against.

How to test an accuracy claim on your own month of evidence

Every vendor in this area will quote an accuracy number. On this problem a single accuracy number is close to meaningless, and the reason is arithmetic rather than scepticism. If four calls in a week of 21,860 contain a defect, a system that says "no finding" to every call is right 99.98% of the time. The measures that matter are per-check and are about the rare class: of the calls that did breach this duty, how many were raised, and of the calls raised, how many survived a person's review.

Vendor claim, the test, and the evidence you keep

The claimThe test to run on your own filesWhat you keep
"We review 100% of calls."Count the recordings delivered against the dialer log for the same period, per agency. Ask what happens to a call with unusable audio.The two counts and the difference, per agency per week — the gap is the part nobody reviewed.
"Our accuracy is 9x%."Ask for recall per check, measured against a truth set, not one accuracy figure across all checks.The per-check numbers, and the definition of the truth set they were measured on.
"It catches missing disclosures."Build a truth set: have two reviewers listen to 300 of your own calls, including every call you already know was bad, and score each check by hand.The truth set itself. It is reusable at every renewal and against the next vendor.
"Findings are reviewed by a human."Ask how many findings a week that is, for your volume, and have your own compliance manager confirm one day’s worth to time it.The person-hours per week of confirmation, which is the real running cost.
"It reconciles remittances."Run it on a month your team already reconciled by hand — one with a file you know was short — and compare the two variance lists line by line.The variance list from both methods, and the lines only one of them found.
"It compares agency performance."Ask how placements are made comparable: by vintage, balance band, prior placement and product. Then ask it to show the strata.The stratification, in writing. A comparison that cannot show its strata is a ranking of the paper, not the agencies.
Our test plan for an evaluation on a collection panel. Run every test on your own month of evidence rather than the vendor's demonstration data; the second column is worthless on files chosen by the seller.

Two further asks are worth making before signature. First, the false-positive load in person-hours at your volume, because a system that raises six hundred findings a week is not cheaper than a sample — it is a sample with extra steps. Second, what the tool does with a check it cannot run: a call with unusable audio should be reported as unread, not quietly scored as clean. The examiner findings that make this concrete are in collection agency compliance standards.

What a person still has to do

Writing this list down is what turns the committee paper from a procurement document into a control description. On the sample panel it has five items, and none of them are ceremonial.

A person confirms each critical finding from the call itself, and signs their name to it. A person decides what to ask the agency for — a corrected remittance file, a retrained team, a script change, a recall — and what deadline to put on it. A person signs each month's placement. A second person co-signs a pause or a recall above the threshold. And a person argues the finding with the agency when it is disputed, which is where the timestamp and the script version earn their keep.

The structure around those five, and the seats they belong to, is set out in what the agency oversight desk owns.

The evidence a confirmation is made from

One call opened for review: an inbound callback with the collector named and the account masked, the transcript timestamped second by second, markers for right-party verification and the recording notice, and a marker at the point the balance was discussed stating that the debt-collector disclosure was missing and citing the rule, with the two script versions side by side.
A finding is only as good as what it opens onto: the recording, the second, the duty and the script in force that day.

What changes when every call is read instead of a sample

Three things change, and only the first is the one vendors talk about.

The first is detection. A fault present in four calls out of a week's 21,860 has roughly a one-in-five chance of appearing in a random 5% review; read every recorded call and it appears. The second, larger change is that cause becomes findable. One call without a disclosure is a collector; four calls without it, all from the same team, all on the same script version issued on a known date, is a script. The unit of correction moves from the person to the thing that produced the pattern, and the remediation becomes testable — reissue the script, read next week's calls from the same team, see whether the line is there.

The third is that the conversation with the agency changes shape. A sample produces a score the agency can dispute in the abstract; a full read produces a list of calls with timestamps, which the agency can check itself.

What six readers got through in one week on the sample panel

21,860

recorded calls read, from all six agencies — every call, not a sample

6files

monthly remittance files tied to 14,212 posted payments

3new

complaints linked to their account, agency and the call behind them

2audit

responses read against the contract, with the missing evidence raised

Fictional sample panel, week 41, inside Collection Agency Management. Sample data, not a measured customer result. The thirty-seven calls that reached a person and the five critical findings among them all came out of the first column.

What does not change is the shape of the week. The evidence still arrives overnight, the grades still move on a Monday, and the placement still waits for a signature. What is bought here is coverage and a cause, not a decision. The product requirements that follow from that, and how the lender-side buy differs from the agency's own, are in collection agency management software. To see the six readers and the people around them on one panel, start from Collection Agency Management.

Questions people ask

What is third-party risk management software?

Software that holds a bank's inventory of vendors and runs the lifecycle around each one: risk tiering, due-diligence questionnaires, contract and certificate tracking, issue and action management, and periodic review dates. It is the relationship layer. It is complementary to evidence reading rather than a substitute for it — a vendor can be fully in date, fully tiered and fully answered in such a system while its collectors are working from a script that drops a required sentence.

How accurate is AI call review?

Ask for the number per check rather than overall, because overall accuracy on this problem is dominated by the calls where nothing happened. The questions worth putting are: of the calls that genuinely breached this duty, what share was raised, and of the calls raised, what share a person confirmed. Insist both numbers come from a truth set built on your own files, and keep that truth set — it is reusable at every renewal.

What data does AI vendor monitoring need on a collection panel?

Six inputs carry it: placement and recall files, call recordings with dialer logs, remittance and activity reports tied to the payment postings, complaints from every channel including those that reach the agency directly, audit questionnaires with their evidence, and the agency contracts with their fee schedules and amendments. The last one is not optional — without the contract as amended, a fee variance cannot be computed and a missing audit document cannot be named.

What should AI never do in vendor oversight?

Three refusals cover much of the risk: never contact a consumer, a collector or an agency; never answer or close a complaint; never move or offset money. A fourth matters for credibility: never treat a serious conduct finding as settled before a named person has confirmed it from the evidence. On the sample panel an unconfirmed critical finding does exactly one thing — it holds back new placements — and nothing else until someone has listened.

Does buying monitoring software reduce the bank's responsibility for its agencies?

No. The CFPB's service-provider guidance is explicit that entering a business relationship with a service provider does not absolve a supervised bank of responsibility for complying with federal consumer financial law, and that responsibility may lie with the bank as well as the provider. Monitoring software changes what the bank knows and how quickly; the answer it has to give for what its agencies did is unchanged.

How do you test an AI monitoring claim before you buy?

Run it on a month you have already worked by hand, including at least one period you know contained a problem. Compare the recordings delivered against the dialer log to test the coverage claim, score the conduct checks against a hand-built truth set, re-reconcile a remittance file you know was short, and time how long it takes your own compliance manager to confirm one day of findings. Keep every artefact; the next vendor can be measured against the same files.

Collection Agency Management.
See it on your own files.

An agentic solution from Botminds. Each agency and law firm graded weekly on results and conduct, with the evidence behind each grade. Book a demo, or explore how it works.