AI customer service agent accuracy: five measures for asset managers

An AI customer service agent for an asset manager is accurate enough when its replies pass five measures on the firm's own past requests: every figure ties to its source at the stated as-of date, no sentence states a figure or claim without a source, no reply goes to someone not entitled to it, every early or restricted figure is held, and every request it cannot answer goes to a person. "How accurate is the AI?" has no useful single answer, because a reply can be fluent, mostly right and still be a disclosure breach. Botminds' Client Inquiry Management is built so each of the five can be counted: every answer is a block stamped with its system, record and as-of date, and a separate Tie-out agent re-reads every figure before a person approves.

The sample figures below come from Quillfield Capital, a fictional firm in the solution.

How accurate is an AI customer service agent? The wrong question

An overall accuracy figure averages away the errors that matter. Suppose an agent gets 98 replies in 100 exactly right. If one of the two wrong ones carries a transposed fee and the other sends an interim holdings list to a consultant before the trust's other investors, the firm has a client correction and a disclosure problem, and the 98% describes neither. Institutional client service needs error rates by kind, because the kinds have different owners and different consequences.

The research reinforces the point. In a field study of 5,172 customer-support agents, an AI assistant raised issues resolved per hour by 15% on average, but the gains varied widely: newer, less skilled agents improved in both speed and quality, while the most experienced saw small gains in speed and small declines in quality, and the authors found "the gains from AI adoption are largest for moderately rare problems" (Brynjolfsson, Li and Raymond, Quarterly Journal of Economics, 2025). That is productivity, not correctness, and it was measured on customer-support conversations, not on figures a regulator can ask about.

What the research measured, and what it did not

The closest evidence on correctness comes from law, where answers can be checked against sources. A preregistered study of commercial legal research tools built on retrieval found that each still hallucinated in roughly a sixth to a third of answers, fewer than a general chatbot but far from none, and documented "substantial differences between systems in responsiveness and accuracy" (Magesh et al., Journal of Empirical Legal Studies, 2025). An earlier study of general models on legal questions put the hallucination rate at 58% or more (Dahl et al., Journal of Legal Analysis, 2024).

Two lessons carry over. Retrieval reduces errors without removing them, so an agent that "uses your documents" still needs an independent check. And error rates differ sharply between tools, so a vendor's figure from another firm's data says little about yours. What none of these studies measured is the design that matters for client service: every figure taken from a system of record as a block with an as-of date, and every block re-read by a second agent that did not write the draft. That design can be measured directly, which is what the five measures do.

Regulators frame the duty the same way. ESMA's 2024 statement on AI in investment services asks firms for robust controls, set before use, over the accuracy of the information AI systems are given and rely on, so that erroneous information does not reach clients (ESMA, May 30, 2024). The Bank of England and FCA survey found user feedback the most routinely monitored metric for AI systems, at 85% of respondents (Bank of England and FCA, 2026). For institutional replies, feedback is the slowest signal there is: a pension board finds the wrong figure in its pre-read, not in the email.

Measure 1: figure match rate against the source

The figure match rate is the share of figures in replies that equal their source at the as-of date the reply states. It is the measure clients notice and the one a firm can make close to perfect, because every figure has a source to compare against. The sample firm's target is that 100% of figures in every reply are tied to their source system and as-of date and re-checked before a person approves.

On the sample request, a consultant's four questions for Calder Valley PERS, the Tie-out agent re-checked 11 of 11 figures and dates. Over the month it stopped 27 figures that did not match their source, all fixed before sending. One was a fee transposed in the draft as $412,600 against an invoice of $421,600. Counting stops matters as much as counting passes: a figure match rate of 100% on sent replies is only credible if the firm can also show how many mismatches were caught before sending.

One answer in Client Inquiry Management, with its source: the fee block and its working

Measure 2: the uncited-sentence rate

The uncited-sentence rate is the share of sentences that state a figure or a factual claim about the client, the fund or the firm without a source behind them. It catches the error the figure match rate cannot: the plausible sentence that was never sourced at all, such as a positioning comment from memory or an AI-use paragraph written fresh. Its pass mark is zero, because an unsourced sentence cannot be checked.

The sample firm enforces it in design. The Drafting agent may not write a figure without a block, every paragraph of the reply carries its block number, and the AI-use paragraph in the sample reply was taken verbatim from approved answer AI-USE-03. A reviewer measures the rate by sampling replies and marking any sentence with a claim but no block, which takes minutes per reply once blocks exist and hours without them.

Measure 3: entitlement errors

An entitlement error is a reply that reaches anyone, including a person copied, who may not receive that data type today. It is the measure most likely to be missed in a vendor's accuracy figure, because the reply itself can be perfect. Its pass mark is zero, and the sample firm's target is zero holdings, estimates or licensed data sent early or to someone not entitled.

The error usually starts at intake. A consultant's authorization from the client expires; the consultant keeps asking; the reply goes out because the consultant is a familiar name. On the sample Thursday three requests for Lakehaven Teachers were held because the consultant's authorization was missing or expired, and each proposed the right next step: ask the plan to renew. To measure the rate, take a sample of sent replies, list every recipient and data type, and check each against the authorization in force on the send date.

Measure 4: disclosure holds raised, kept and missed

The disclosure measure has two halves. Caution is the share of holds that compliance keeps; a low figure means the agent holds too much and slows the desk. Misses are the restricted figures that went out unheld, found by reviewing a sample of sent replies against the disclosure policy; the pass mark for misses is none. Both matter, but only misses become a problem outside the firm.

In the sample month the Disclosure agent raised 46 holds and checked 3,862 answer blocks. Of the 32 holds decided, compliance released 5, and the rest were answered with an alternative or kept. That tells the firm the agent is cautious, which is the right direction for a disclosure check. It does not tell the firm about misses, which only a review sample can show. Performance has a rule behind it too when a reply counts as an advertisement, as material a consultant passes to a client's board can: under the SEC's Marketing Rule, an adviser's advertisement may not show gross performance without net performance of at least equal prominence. The sample reply states 4.37% net of fees beside 4.45% gross, with the period and benchmark named, so the question never arises.

Measure 5: hand-offs to a person, and whether they were right

The hand-off rate is the share of requests or answer blocks the agents route to a person instead of completing. A low rate is not automatically good: an agent that never hands off is guessing. The sample firm sets two thresholds. Below 85% confidence in classification, a person classifies. Below 95% confidence for the Fetch and Tie-out agents, a routine block reaches the associate unconfirmed.

Routing thresholds the Head of Client Service approves, not a vendor default

Measure the second half by sampling hand-offs and asking whether a person was needed. Unknown senders, instruction changes and complaints should always hand off; a routine fee copy usually should not. On the sample Thursday the Intake agent held 2 unknown senders for a person rather than guess, which is a hand-off the firm wants every time.

How to run the five measures on last quarter's inquiries

Run them on a replay of requests the team has already answered, where the right reply is known. The sample firm's go-live plan has the agents draft the previous month's 3,212 requests in shadow mode, with nothing sent; 2,849 drafts, or 88.7%, matched what the team sent. That match rate is a useful start, but it is not accuracy. The 363 drafts that differed must each be classed: the draft was wrong, the team's reply was wrong, or both were acceptable.

Sample size decides what a clean result can prove. When a sample shows no errors, the rule of three gives the 95% upper bound on the true error rate as about 3 divided by the sample size. So 300 figures checked with no mismatch shows the mismatch rate is likely under 1%, and 3,000 shows it is likely under 0.1%. A vendor who reports "zero errors" on 50 replies has shown only that the rate is probably under 6%.

Draw the sample by kind, not at random from the whole mailbox. Holdings, performance and fee requests carry the disclosure and figure risk; a random sample dominated by routine document copies will flatter every measure. The 30-day pilot on last quarter's mailbox sets this out as a schedule with pass marks fixed before it starts.

What to ask an AI customer service agent vendor

Ask for the five measures on your data, and for the design that makes each one countable. A vendor's general accuracy figure, a customer satisfaction score from another industry or a reference call does not answer any of them.

Where an AI customer service agent fits, and where it does not

It fits where the answer already exists in a system the firm trusts, which is most institutional client requests: holdings, returns, fees, statements, fund data and approved answers. There the five measures can be driven close to their pass marks, because every figure has a source to tie to. On the sample request four questions were answered and approved in 6 minutes against a 4-hour Tier 1 target.

It fits less well where the answer is a judgement, such as why a strategy lagged or whether a position still makes sense; there the agent drafts from the attribution system and a product specialist decides. And it does not replace the people who decide on holds, complaints and instruction changes. What each agent may never send sets out those lines, and the evidence on replacing service desks with AI sets out why the research points to reassignment. The solution is at Client Inquiry Management.

Client Inquiry Management on the sample request: tie-out 11 of 11, every block cleared or replaced

Questions people ask

Why do people say AI customer service is bad?

Mostly because of chatbots that answer confidently from nowhere or cannot hand a customer to a person. Studies of retrieval-based legal tools still found hallucinations in roughly a sixth to a third of answers. An agent that cites a stamped source for every figure, is checked by a second agent and hands off below a threshold is a different design, and it can be measured as such.

What is the difference between an AI chatbot and an AI customer service agent?

A chatbot answers in a conversation; an agent completes a task, here fetching the data, checking what may go out and drafting the reply. For an asset manager the distinction matters less than the controls: FINRA holds the firm responsible for the communication either way, and ESMA expects firms to tell clients when AI handles the interaction. The same FINRA positions, read for a sales desk, are set out in the guide to FINRA AI rules.

How do you measure the accuracy of an AI agent's replies?

Count error kinds separately on a replay of requests your team already answered: figures that do not match their source, sentences without a source, entitlement errors, disclosure misses and wrong hand-offs. Draw the sample by request type and size it so a clean result means something; with no errors in 300 figures the 95% ceiling on the error rate is about 1%.

Can an AI agent make up a number?

A general model can, which is why the design matters more than the model. In the sample firm the Drafting agent may not write a figure without a source block, the Fetch agents compute only stated arithmetic and show the working, and any figure that does not tie stops the reply. This month 27 such figures were stopped before sending.

What match rate between AI drafts and the team's replies is good enough?

There is no regulatory figure, and the match rate is not accuracy. In the sample firm's shadow run, 88.7% of drafts matched what the team sent; the useful work is classing the rest as agent errors, team errors or acceptable differences. Decide the pass marks for each of the five measures before the run, not after seeing the results.

Do we have to tell clients an AI drafted the reply?

For retail investment services in the EU, ESMA's 2024 statement says firms using AI for client interactions should disclose it during those interactions. US rules do not require a label on a reply a person has reviewed and sent, but they require supervision and retention either way. At the sample firm, 58 requests this month asked how it uses AI; it answers them with approved text, AI-USE-03.