Chatbot conversation analytics that reads every chat, checks every answer against your tariff and turns each failure into a fix
Every chat read, and every failure turned into a skill or a fix the bot team can ship.
See one case, screen by screen ↓Virtual Agent Quality Audit is software that reads every conversation your chatbot or virtual agent has, marks the turn where it failed and turns each kind of failure into work the bot team can ship. Today the bot platform reports containment: every chat that did not reach the transfer queue counts as a success. A customer who gave up after the third “Sorry, I didn’t get that” is contained. So is one told the wrong reconnection charge, and one whose burst-main report was acknowledged and never created.
Nobody reads the chats at scale. A CX analyst reads 40 a week by hand out of 4,860 and writes the monthly bot quality note. The knowledge base drifts behind the tariff: when the rates change on 1 July, the bot keeps quoting last year’s. The cost lands somewhere else — in the contact centre on Monday morning, in a complaint about a charge, in a hazard that reached the field only because the customer phoned.
Estimated hours a week for a digital channel team reviewing its virtual agent by hand on a sample, against reviewing the audit’s verdicts on every chat.
Six agents read every chat, mark where the bot failed and turn each cluster into a skill or a fix; people approve.
Lumina Utilities runs Lumi, a virtual assistant on its app, website and WhatsApp for electricity and water customers. On Sunday 12 October at 19:52 a water customer in Cedar Hill tries to send a meter reading. Here is what the audit found, screen by screen, in the working solution.
Hana reports containment to the CX director every Monday. The replay opens on the week of 6–12 October: the bot dashboard says 76.4 % contained; reading every chat, 61.0 % were truly resolved. 918 of 4,860 chats failed, ranked by chats spoiled — loops on “I didn’t get that” 312, wrong answers 214, dropped hand-offs 168, abandoned mid-task 131, out-of-date answers 93.
“Lumi’s dashboard says 76 % contained. I want to know what the other customers went through.”
Chat CH-1012-0417, Lumina app, water account ···8106. The customer writes “I want to give my water meter reading”. Lumi has no skill for it; its top guess is “How to read your meter” at 41 %, and it answers “Sorry, I didn’t get that.” She tries “meter reading. water. 04812” — a reading in range for this meter, last actual 04731 on 11 June. Fallback 2 of 3.
Every turn is marked in the margin: fallback 1 of 3, no intent matched, gave the reading: 04812, in range for this meter.
“How do I submit a reading, my bill was estimated again.” It is her third estimated bill in a row — July, August and September — and October’s is $142.60 against a usual $88. Lumi explains how to read the black numbers and says readings are not needed. The answer checker flags it: article KB-118 answered how to read the dials, not how to send a reading, and her meter was missed three times.
“Wrong answer · Answered how to read the dials, not how to send a reading.”
“I KNOW how to read it. I want to SEND it. 04812.” Lumi answers “Sorry, I didn’t get that” — fallback 3 of 3, a loop. The replay pauses on that turn and a chip appears under it: Missing skill: take a meter reading, 37 chats this week, 12 phrasings from real chats.
“Missing skill: take a meter reading · 37 chats this week · 12 phrasings from real chats”
She types “agent”. Lumi promises a customer care agent; live chat closes at 20:00 on Sundays, and at 19:58 Lumi says agents are available Sunday to Thursday, 7:30 to 20:00. No ticket, no callback, no reading saved. On Monday at 08:06 she calls: 9 min 40 s, agent Rami Aziz keys reading 04812, and the October bill is reissued from $142.60 to $91.20.
Bot platform logged “Transferred to an agent” · Audit verdict: Failed · loop, wrong answer, dropped hand-off · follow-up contact $9.54
Of the 312 chats that looped this week, 134 fall into six proposed skills and 178 are one-offs. SK-14 Take a meter reading has 37 chats, up 12 on last week. Others: a payment plan for a water bill (29), report a street light out (24), stop an EV charging session (11). SK-17, changing the account holder after a death, is proposed as a route straight to a person, never automation.
The long tail is reviewed again each week; a phrasing joins a cluster when 5 or more chats share it.
SK-14 lists 12 phrasings, personal data removed, and five details to collect: account, meter, reading, read date and an optional photo. The conversation draft checks the reading against the expected range, 04790–04860; out of range twice, or after hours, it hands to a person or saves a callback with the reading attached. Replayed against the 37 chats: 31 resolved by the draft, 4 handed to a person, 2 out of scope.
Expected outcome: “31 of 37 chats a week resolved by the bot; about 16 calls a week avoided” — estimated, measured after release.
Hana approves SK-14. Leila Farouk’s backlog gets skill request BT-2291 · Take a meter reading, with the 12 phrasings as a training file, the 5 details to collect, the conversation draft, the hand-off rule and the 37 source chats, for Lumi release 26.21 on 23 October. The approval is recorded in the audit trail.
“The audit only proposes; nothing changes in Lumi until the bot team ships.”
Lumi told 64 customers this week that “there’s no charge to reconnect”, from FAQ-27 (2023). Tariff & Charges Schedule 2026 §4.2, effective 1 July: $35.00 for restoration within 24 hours, $70.00 same-day if requested before 14:00. The knowledge fixer drafts the change as a marked diff and proposes retiring FAQ-27. The fix goes to Daniel to approve; when he publishes, the answer checker re-tests the 64 chats.
“Money answers are fixed first: what the bot says about charges is what Lumina says.”
168 hand-offs never arrived this week, and 119 of those customers called within 48 hours. Nine were hazards answered “I’ve passed this on”: five water leaks or burst mains, two sparking meter boxes, a fallen overhead line and an exposed cable. The burst main on Cedar Hill Road at 23:41 on Friday reached the field only as WO-58821, from a phone call 29 minutes later. Omar owns the hand-off rules; the hazard rule is added from this screen.
Hazard rule: the 24-hour emergency number in the first reply; never “passed on”.
The same week of chats, seen by the five people who own the virtual agent — what their week looked like, and what it looks like now.
Six agents read every chat, mark where the bot failed and turn each cluster into a skill or a fix; people approve.
Reads every chat from the app, website and WhatsApp, masks personal data, marks each turn — understood, fallback, answered, hand-off promised, hand-off delivered — and gives the chat one verdict with the turn that decided it.
Checks every factual answer — a charge, a rate, a time, a rule — against the Tariff & Charges Schedule, the Customer Service Code and the articles, and flags wrong and out-of-date answers with the passage that proves it.
Follows each hand-off and each “I’ve passed this on” to the agent join, the ticket, the work order or the next contact from the same account within 48 hours, and marks it delivered or dropped with the reason.
Groups the phrasings that led to fallbacks or mis-routed answers into proposed missing skills — new, an extension of an existing one, or a route to a person — with up to 12 phrasings, the details to collect, a conversation draft, the hand-off rule and a test against the cluster’s chats.
For each group of wrong or out-of-date answers, finds the article the bot used and drafts the smallest change that makes it match the governing passage, showing old and new text, the source and the chats affected.
Every Sunday at 22:00 reports chats, truly resolved, handed off and failed by type against the bot platform’s containment, and what changed after each shipped skill or fix.
Approves every skill request. The agents propose; a named person decides.
Ask about any chat, failure, missing skill or knowledge fix — or say what to change. Every answer points to its chats and sources.
Why does Lumi fail on meter readings?
Lumi has no skill to take a reading. This week 37 customers tried to send one — 25 last week — mostly on estimated bills. Lumi either loops or matches “How to read your meter”, which tells them a reading is not needed. Proposal SK-14 is open: 12 phrasings, tested 31 of 37 resolved.
Which wrong answers carry a money risk?
One stands out: reconnection is “free”. Lumi said it 64 times this week, from FAQ-27 (2023), which is still live. The 2026 tariff charges $35, or $70 same-day. One customer has already complained. The fix KF-31 is drafted for Daniel Okoro.
Show hazard reports the bot never passed on
9 hazard reports this week were answered “I’ve passed this to our team” and never created — 29 field reports in all. The worst: a burst main on Cedar Hill Road at 23:41 on Friday; the first work order came from a phone call 29 minutes later.
Route bereavement chats straight to a person
Done, as a routing rule: “Bereavement, death of the account holder → straight to a person, no menus.” 18 chats this week would have reached a person in the first reply instead of looping. It matches SK-17, which is proposed as a person route, never automation.
13 screens from the working solution, on its sample data. Pick one to see it large.
What counts as a failure, which source wins and who approves are settings the digital channel owner sets; saved rules apply to chats from then on.
| Setting | Default | Choose from |
|---|---|---|
| Loop | 3 fallbacks in one chat | Any number of fallbacks you set |
| Dropped hand-off | 5 minutes with no agent joining | Any number of minutes you set |
| Report never created | 15 minutes after “I’ve passed this on” | Any number of minutes you set |
| Abandoned mid-task | Any task flow | Any task flow · Payment only — left inside a payment, move or reading |
| Follow-up window | 48 hours | Hours to match the next contact, as you set |
| Which source wins | Latest effective date first | Tariff & Charges Schedule 2026 · Customer Service Code 2026 · knowledge base articles · FAQs |
| Always to a person | Bereavement · life support · hazards — On | Add a topic, such as hardship; it goes straight to a person |
| Coverage and privacy | Every chat, every 15 minutes | App, website and WhatsApp · account numbers masked to the last 4 digits · chats kept 13 months |
Every figure is labelled: a target the solution is built to, an estimate, a typical published result, or a proven one.
“demo” = seen in the working solution, on its sample data for the week of 6–12 Oct 2026 · “estimated” = our arithmetic from published inputs ($187K = 376 follow-up calls a week × 52 × $9.54) or, for the hours table, our estimate · “typical” = published figures (Gartner customer survey, Aug 2024: 14 % of service issues fully resolved in self-service; E Source: $9.54 average per escalated contact, across 15 utilities). Context: in Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024) a company was held responsible for its chatbot’s answer. Lumina Utilities is a fictional demo utility; the people, accounts, chats and records named on this page are characters and sample data in the working solution.
It is reading what a utility’s chatbot or virtual agent actually said to customers, to find where it failed and why. Virtual Agent Quality Audit reads every chat, gives each a verdict — resolved, handed off properly, or failed by loop, wrong answer, dropped hand-off, abandoned task or out-of-date answer — and turns each cluster of failures into a missing skill or a knowledge fix.
Containment counts every chat that did not reach the transfer queue, so a customer who gave up counts as a success. True resolution comes from reading every chat. In the demo week the bot dashboard said 76.4 % contained while 61.0 % were truly resolved: containment counted 750 failed chats as successes.
The skill finder groups the phrasings that led to fallbacks or mis-routed answers, names each group as a skill in plain words and picks up to 12 real phrasings, with personal data removed. It drafts what the skill must collect and its hand-off rule, and tests the draft against that week’s chats before anyone approves it.
Yes. The answer checker checks every answer that states a charge, rate, time or rule against the tariff and charges schedule, the customer service code and the articles; the source with the latest effective date wins. It quotes the passage, and the knowledge fixer drafts the article change for the knowledge manager to publish.
The hand-off tracer follows every “I’ve passed this on” to the work order or the next call, so a hazard the bot never reported shows up the same week. Bereavement, life-support and hazard topics are never proposed for automation, and the hazard rule gives the emergency number in the first reply.
Yes. The agents read, check, trace and propose; they never change the bot or publish an article. The digital channel owner approves skill requests, the knowledge manager approves knowledge fixes, the contact centre manager owns hand-off rules, and every approval is on the audit trail.
It reads the conversation logs your virtual agent already produces — in the working solution, chats from the app, website and WhatsApp — alongside contact-centre records, work orders, the knowledge base and your tariff and policy documents. The audit runs every 15 minutes over every chat.
The Agentic Solution Engine builds and deploys it from your requirements and documents — your conversation logs, knowledge base, tariff schedule and service code — and it goes live once every quality gate has passed. We will walk you through it on a week of your own chats first.
We’ll run Virtual Agent Quality Audit on a week of your own virtual-agent conversations.