Consumer Duty outcomes testing with AI: synthetic customers, real checks

Consumer Duty outcomes testing is the evidence that customers got good outcomes in practice: testing communications before and after they go out, as PRIN 2A.5.10R requires where appropriate, and testing real customer journeys, calls and files against a written definition of a good outcome. AI can now help with both. The FCA's July 2026 review of outcomes monitoring described a firm that piloted AI-based synthetic testing of its communications and found the tool's outputs largely aligned with its human testing; its March 2026 review of consumer understanding found many firms whose testing was superficial, one-off or missing. The safe pattern is AI as a first pass and a second reader, validated against human testing and real customer behaviour, split by customer group, with people deciding. Botminds' Consumer Duty Outcomes Monitoring takes the outcome-testing side: it reads what customers were actually told across letters, app messages and calls, flags key communications never tested for comprehension, and drafts plain notices, but it does not replace testing with real customers.

Examples from practice on this page come from a fictional sample bank, Corvane Bank UK, as it runs through the solution. The rules and findings are the FCA's, linked where they appear.

What is Consumer Duty outcomes testing?

Consumer Duty outcomes testing is any test that checks what customers actually understood, did or received, rather than what the firm intended. It comes in two kinds. The first is communications testing, which PRIN 2A.5.10R(1) requires: “Where appropriate, a firm must: (a) test communications before communicating them to retail customers; and (b) (as set out in PRIN 2A.9) regularly monitor the impact of the communications once they have been communicated” (PRIN 2A.5.10R). The second is testing of outcomes on real cases: file reviews, call monitoring, mystery shopping and journey audits, which FG22/5 lists among the information firms can use, often on a risk-based sample (FG22/5, chapter 11).

The FCA's review of second-year board reports said boards were still not seeing enough of either. Reports focused more on products and value than on understanding and support, and the FCA asked that firms be able to evidence how they “test communications”, “assess consumer comprehension” and “respond where consumer behaviours indicate misunderstanding or friction” (FCA, 16 April 2026). That is the definition of outcomes testing in three lines.

When must communications be tested under the Consumer Duty?

Communications must be tested “where appropriate”, and PRIN 2A.5.12G lists what decides it: the purpose of the communication, especially if it prompts a decision; its context, timing and frequency, with tests more appropriate for communications reaching many customers; customers' information needs and characteristics of vulnerability; the scope for harm if it is misunderstood; and whether urgency outweighs testing first. A savings letter telling customers a better rate exists ticks most of those boxes.

The rules also say how to read the results. PRIN 2A.5.13G(2): “if there is a notably different response by retail customers than was reasonably anticipated by the firm or ought to have been reasonably anticipated, including a notably lower response rate, following a communication prompting retail customers to take action, then this would suggest that the communication has not been understood.” At the sample bank, the estimate for a letter inviting holders of a closed savings issue paying 1.10% to move to 3.85% was that about 3% would act. Under that guidance, a 3% response to a letter that offers more than three times the interest is itself a test result.

What the FCA found when it looked at testing

The FCA's March 2026 consumer understanding review, which drew on a September 2025 survey of 38 firms as well as supervisory findings and research, found four recurring weaknesses (FCA, Consumer understanding: good practice and areas for improvement). Some firms said they had tested but provided little evidence, and testing was often superficial, one-off or poorly documented. Some collected MI such as drop-off rates or chat logs but could not explain how it informed their view of understanding. Some did not test with people with accessibility needs, language requirements or lower financial capability. And some changed communications without checking whether the change worked.

One finding goes to the heart of how firms evidence understanding: “Several firms continued to rely on sales data or the absence of complaints as evidence of understanding. This does not provide reliable assurance, even taking into account the size and complexity of firms.” The good practice it described was proportionate and practical: short surveys, comprehension checks, A/B testing before and after, customer callbacks, and a record of what changed, why, and with what effect.

Can AI test customer communications?

Yes, as a first pass and a second reader, provided its results are checked against human testing. The FCA's July 2026 outcomes monitoring review described an example: “One firm piloted synthetic testing to identify where communications may confuse or mislead different customer groups” (FCA outcomes monitoring review). The AI tool assessed communications against themes including vulnerability, fair treatment, root cause indicators and the post-advert customer experience, and produced a comprehension score, an engagement heatmap, a risk matrix and recommendations.

What made the pilot credible was the check. The firm ran the tool on a handful of communications that had already been through human testing, at the awareness, purchase and onboarding stages, and “It reported a high average match rate, with the AI tool’s outputs largely aligned with the human-testing findings.” The FCA's suggestion for smaller firms set the boundary: AI can highlight potential comprehension risks and “it can be used to support and supplement human review and judgement with appropriate guardrails.”

What is synthetic testing, and what can it not see?

Synthetic testing uses an AI model to read a communication as different kinds of customer might, predicting where they would misread it, miss a key point or fail to act. It is fast, cheap, repeatable and tireless, which makes it good at finding jargon, buried key information, missing calls to action and reading-age problems across hundreds of letters. What it cannot do is be a customer: it has no real circumstances, no history with the firm, and no way to show what a real person would actually do.

The two are complements, not substitutes. The FCA's consumer understanding review cited research that 1 in 7 adults have literacy skills at or below those expected of a 9 to 11 year-old, and its 2024 Financial Lives survey found 34% of adults with poor or low financial numeracy. A model can flag a letter that is likely to defeat them; only testing with people from those groups shows whether the rewrite works.

Does the FCA accept AI testing of communications?

The FCA has no approval regime for testing tools; it judges the evidence. Its October 2026 publication on AI in financial services is clear on the firm's position: a firm that uses AI tools stays responsible for the outcomes its customers get (FCA, AI in financial services: good practice; AI and vulnerable customers quotes it). Among its areas for improvement it listed firms that “assumed their AI outputs were clear and helpful without testing whether different consumer groups understood them”.

Its good-practice case studies show the shape the FCA looks for. In one, a firm used a large language model to draft clearer replies to queries and complaints, and before deploying the new style it tested the AI-generated letters with a small sample of different consumers to check they helped people understand and improved outcomes against its measures. In another, on affordability, reviewers assessed each case “without being shown the AI's proposed outcome, allowing the firm to compare independent human and AI judgements and reduce the risk that reviewers would defer automatically to the system.” The same blind comparison is how an AI communications test should be validated.

How to use AI in outcomes testing safely

Use AI where it is strong, measure it against people where it is weak, and keep the decision with a named person. The checklist below turns the FCA's examples into a routine a conduct or communications team can adopt.

Testing outcomes, not only letters: what customers were actually told

The other half of outcomes testing looks at what happened to real customers, and here AI's advantage is coverage rather than imagination. A file review or call-listening programme usually tests a sample; FG22/5 expects some monitoring, such as file reviews and distributional analysis, to be risk-based samples. An agent that reads transcripts can test every call on a topic, which changes what a firm can say to its board.

At the sample bank, the Communications Trace agent tested a single outcome for the 41,800 holders of a closed savings issue: were they told a better rate existed? Across four years it found two mentions in letters, one a line on page two of a savings summary and one a boxed reminder in an annual statement that had never been tested for comprehension; no app message; and, across all 312 calls in the year in which holders asked about their rate, 204 where the adviser did not mention a better account. That is an outcome test on the whole population, with the evidence attached, and it led the bank to a rate rise rather than another letter.

How do you test communications with different customer groups?

Test with people from the groups most likely to misunderstand, and report the results by group. The FCA's March 2026 review found firms that tested with nobody from those groups, so it was unclear whether changes helped them, and it described a smaller firm that tested its renewal letter with a small sample including two customers with sight impairments, then introduced a large-print design, a 100-word summary and clearer next steps, and checked afterwards with follow-up calls and a short survey. PRIN 2A.5.8R already requires communications to be tailored to customers' characteristics, including vulnerability.

For an AI first pass, “groups” means the tool's results should be checked group by group against human testing, not averaged. For outcome tests on real customers, it means splitting the result: at the sample bank, the closed issue whose holders were rarely told is also one where 62% of holders are aged 70 or over. Testing and supporting customers in vulnerable circumstances is covered in depth in the Vulnerable Customer Support series; FCA Consumer Duty outcomes monitoring sets out the group comparison the rules require.

Where Consumer Duty Outcomes Monitoring fits, and where it does not

The solution fits the outcome-testing half and the evidence around communications. Its Communications Trace agent tests “were they told?” across letters, app messages and redacted call transcripts for a product's holders; its outcome measures include “key letters tested for comprehension”, which on the sample bank's report reads 0 of 1 and is marked “No evidence yet”, so an untested key letter reaches the board as a gap with an owned action; and its Drafting agent drafts customer notices in plain words, such as the sample bank's rate-change notice at a reading age of 9, for a person to check and send.

An untested key letter reaches the board as a gap, not a blank

It does not run consumer panels, recruit testers or measure comprehension with real customers, and it does not offer synthetic customer testing of the kind the FCA's pilot firm used. Those are jobs for a firm's own research and communications teams or specialists; the solution records their results as evidence when they exist and reports their absence when they do not. A firm buying for comprehension testing alone should look elsewhere. A firm whose board asks what customers were actually told, and what happened next, is the buyer it is built for.

A plain customer notice drafted for a person to check and send

To see the trace and the measures on the sample bank, start from Consumer Duty Outcomes Monitoring; how the results reach the board is set out in Consumer Duty board reporting requirements.

Questions people ask

How many customers should a communications test use?

The FCA sets no number. Its examples range from a smaller firm testing a renewal letter with a small sample that included two customers with sight impairments, to a firm testing AI-drafted letters with a small sample of different consumers before deployment. What matters is that the sample includes the groups most likely to misunderstand and that the test is repeated after changes.

What is comprehension testing?

Comprehension testing checks whether customers understood the key points of a communication, usually by asking them questions afterwards, by short surveys or by callbacks. The FCA's March 2026 review lists comprehension checks with surveys, A/B testing and customer callbacks as proportionate methods, alongside documenting what changed and its effect.

Is A/B testing acceptable under the Consumer Duty?

Yes, as one method among several. The FCA's consumer understanding review describes A/B testing before and after a change as good practice, provided the firm verifies whether the change improved understanding and adapts based on the results.

Can the absence of complaints show a letter was understood?

No. The FCA's March 2026 review found firms still relying on sales data or the absence of complaints as evidence of understanding and said this does not provide reliable assurance. A notably lower response than expected to a letter that asks customers to act suggests it was not understood, under PRIN 2A.5.13G.

Do AI-drafted letters need to be tested?

Yes, like any other communication where testing is appropriate. The FCA's October 2026 good practice on AI criticised firms that assumed AI outputs were clear without testing whether different consumer groups understood them, and described a firm that tested AI-drafted letters with a small sample of consumers before deploying them.