Complaint management software for a medical information desk is bought on a demonstration and judged, eighteen months later, on the cases the demonstration did not contain. The way out of that is unglamorous and cheap: take a fortnight of inquiries you have already closed, replay them through the system with nothing connected downstream, and score what it found against what your own people recorded at the time. This guide sets out that fortnight day by day, the four things worth scoring, how to seed the awkward cases deliberately, and the ten questions to put to a vendor before any of it starts. Botminds' Medical Inquiry Intake & Triage is designed to be bought this way, and the page says plainly which of its own numbers a pilot should expect to reproduce and which it should not.
The screens here are the product's sample data on fictional people and medicines. One boundary applies throughout: this work routes product complaints to quality and access requests to patient support, and it does not classify, assess or report adverse events — that stays with the safety team the desk hands over to, and a pilot should not be scored on it.
What a replay needs, and what the published literature on machine triage actually rests on
4
things to bring: intake procedures, approved answers, your product dictionary, and a sample of closed inquiries
2weeks
from freezing the sample to a written decision, with no downstream connection
10
studies survived screening in the systematic review of language processing for triaging referrals
from 4,225 titles and abstracts
2,937
questions decomposed from 1,467 consumer health requests — the baseline for the second-request score
What should complaint management software do?
For a medical information desk, five things, and they are worth stating before any feature list arrives. It should take an inquiry from every channel the desk runs and stamp the minute it was first received, not the minute a form was opened. It should split the inquiry into every separate request it holds. It should recognise when a product defect has been described, and capture the lot, the expiry and whether a sample can be returned while the caller is still reachable. It should route each request to the team that owns it with its own clock. And it should leave a file another department can act on, with the requester's own words behind each field and a named person confirming before anything is sent.
Nothing on that list is about the user interface, and all of it is testable on inquiries that have already happened. That is the point of what follows. What inspectors actually cite on complaint handling is a useful second list to read beside it, because the citations are about exactly these properties of the file.
A two-week pilot on inquiries you have already closed
A replay has one advantage no demonstration can match: you already know the right answer. Your own people worked these inquiries, found the questions they found, sent what they sent. Running the system over the same material turns an argument about capability into a count of agreements and disagreements, with every disagreement readable in full.
It is also the posture standards bodies ask for. NIST's AI risk framework puts it in one line: test an AI system before it goes live, and keep testing it at intervals once it is running (NIST AI 100-1, January 2023). Before deployment is the cheap half; the fortnight below is one way to spend it.
A fortnight on one page: who owns each step and what it leaves behind
Days 1–2
Pick the sample and freeze it
A fortnight of closed inquiries across every channel, with the outcome your own people recorded for each — the questions found, the route taken, the lot, the received date. Frozen, so nothing changes under the test.
OwnerMedical information lead
Day 2
Write down the role and scope before anyone sees a result
One page: which decisions the system is being tested on, which it is not, and what would count as a pass. Signed before the replay starts.
OwnerLead with quality
Days 3–4
Load your own procedures and answers
Intake procedures, approved standard responses, the product dictionary with your misspellings and nicknames, and the triage rules and clocks as your desk actually runs them.
OwnerVendor with your lead
Day 5
Seed the hard cases
A short list of deliberately awkward inquiries added to the sample: a missing reply contact, two plausible medicines, a lot the register does not hold, a second question in a second paragraph.
OwnerQuality and a second-line reviewer
Days 6–8
Replay, read-only
Every inquiry read and triaged, with nothing sent anywhere. Each output carries the passage it came from, so a disagreement can be adjudicated rather than argued.
OwnerThe system, nobody downstream
Days 9–10
Score the four things
Agreement, per class, with the denominator beside each number: second requests, identifiers, received dates, routes.
OwnerA reviewer who did not run the pilot
Day 11
Adjudicate the disagreements
Every difference read in full, and classified: the system was wrong, your own file was wrong, or the case was genuinely ambiguous. The third pile is the interesting one.
OwnerLead, quality, second-line reviewer
Day 12
Write down what you could not measure
The list of things a replay cannot test — tone on a live call, whether a prompt would have changed what the caller said, what quality did with the complaint afterwards.
OwnerLead with quality
Days 13–14
Decide, in writing
A recommendation against the pass criteria written on day 2, with the adjudicated scores, the unmeasurable list, and the go-live order if the answer is yes.
OwnerThe buying group
Day 2 is a step pilots often skip, and the one that decides whether the result means anything. FDA's draft guidance on AI supporting regulatory decision-making asks for the context of use to be written down first — what job the model does, where that job stops, what exactly is modelled and how its outputs will be used (FDA, draft guidance, January 2025, step 2). Write the pass criteria on day 2 and a disappointing score is information. Write them on day 13 and they will be whatever the score turned out to be.
What data do you need to start a complaint management pilot?
Four things, and none of them is a system connection. Your intake procedures, so the system routes the way your desk is meant to. Your approved, current standard responses, so a match can be judged as a match. Your product dictionary, including the misspellings and nicknames your callers actually use — the product's own sample dictionary takes "veltremab" and "corvanta" for exactly this reason. And a sample of past inquiries across every channel, each with the outcome your own people recorded.
That last one is where the effort is. The sample has to carry not only the inquiry but the answer key: which questions were found, which were routed where, what lot was captured, what date was recorded as the receipt. If your desk cannot export that, the pilot has found something before it has run — because the same gap is what makes a complaint file hard to produce for an inspection.
The rules the pilot is configured with, before anything is replayed

The four things to score, and how to score them
Resist scoring everything. Four measures carry most of the decision, and each has a denominator the sample can supply.
The scorecard a replay fills in — the denominators are the pilot's, the numbers are yours
Second requests found
How to score itOf the inquiries your own people recorded as holding more than one request, how many did the system split the same way? And how many did it split that your people did not?
The second number matters as much as the first: a request your file never recorded is not automatically a false positive.
Out of__ of __Inquiries in the sample
Product complaints recognised
How to score itOf the inquiries your file records as having produced a complaint, how many did the system open one for — and did it decline the storage mistakes your people also declined?
Score the declines as well. A system that opens a complaint on everything is not accurate, it is loud.
Out of__ of __Complaints in the sample
Identifiers captured
How to score itWhere a lot, expiry or device identifier exists in the file, did the system read the same one — and did it flag the ones that are not in your batch register?
Transpositions are the failure to look for. Compare character by character, not by eye.
Out of__ of __Inquiries with an identifier
Received date and route
How to score itDid the received date match the date of first receipt in your own file, and did each request go to the team your people sent it to?
A route disagreement is often a procedure disagreement. Read ten of them before calling any of them an error.
Out of__ of __Requests in the sample
Two rules make the numbers honest. Print the denominator next to every score, because "94% accurate" over eleven cases is not a measurement. And count the disagreements in both directions: a second request the system found that your file never recorded is not automatically wrong, and in the replays worth running it is often the most valuable row on the sheet.
Seeding the hard cases on purpose
A fortnight of ordinary inquiries will be mostly ordinary, and a system can score well on it while failing every case that matters. So add the awkward ones deliberately, from your own history where you have them: an inquiry with no reply contact at all; one where two of your medicines could plausibly match what the caller said; a lot number that is not in your batch register; a second question buried in the second paragraph of an email about something else; a question about a dose or a route the label does not describe; a misspelling nobody has added to the dictionary.
Score those separately, and expect a worse result on them. The useful question is not whether the system gets them right but what it does when it does not — whether it names the open check or quietly fills it in. Which of the agents may decide what sets out how that line is drawn in this product.
Who runs the scoring matters as much as what is scored. NIST asks that the regular checks on a system be done by someone other than the people who built it day to day — an internal expert from outside the build team, an independent assessor, or both (NIST AI 100-1, MEASURE 1.3). The pilot's equivalent is simple: the person who configured the replay should not be the person who scores it.
How do you measure whether triage is accurate?
By agreement with a known answer, per class, on your own material — not by a single accuracy number. Take each of the four measures above, compare the system's output with what your file records, and put every disagreement in front of someone who can decide which side is right. Three piles come out: the system was wrong, your own file was wrong, and the case was genuinely ambiguous. A pilot that produces no third pile has not looked hard enough at its sample.
The unit of assessment is the desk, not the model. The joint EMA and FDA principles are explicit about it. Performance should be assessed according to risk and across the complete system, including the way people and the AI work together, with data and measures fit for the intended use (EMA and FDA, guiding principles, principle 8, January 2026). In practice that means scoring what a specialist would have confirmed, corrected or rejected — the draft plus the person — rather than the draft alone.
Where a replay's numbers come from once the system is running

What published accuracy numbers do and do not tell you
They tell you the task is tractable, and almost nothing else. In a peer-reviewed comparison against physician performance on clinical vignettes, the frontier models triaged 133 of 144 cases correctly (92.4%), and a collaborative set-up 44 of 48 (92%) (Sorich et al., 2024). That is a real result on 144 cases of a different task, with a different definition of correct, in a different setting. It is not a forecast of what your desk will see.
The systematic review of the closest published work makes the point from the other direction. Screening the literature on natural language processing for triaging outpatient referrals, the authors report that "A total of 4,225 titles and abstracts were reviewed resulting in 26 full-text reviews. A total of 10 studies were used for data extraction and synthesis" (Fung et al., 2026) — ten studies, for an entire field. And their conclusion is the sentence every buyer should carry into a vendor meeting: "As most studies relied solely on retrospective data, further studies focused on prospective effectiveness and clinical impact should be prioritized."
Retrospective is exactly what a two-week replay is. That is not a reason to skip it — it is the cheapest honest evidence available before go-live — but it is a reason to write down that the replay measured agreement with a past file and not effect on a live desk, and to plan the prospective check that follows.
A vendor demonstration against a replay on your own closed inquiries
What a demonstration shows: Sample data chosen by the vendor, where every case has a clean answer.
What a replay shows: Your own inquiries, including the ones your people argued about.
What a demonstration shows: None. Nothing on screen can be wrong.
What a replay shows: What your own people recorded at the time, case by case.
What a demonstration shows: Absent, because the sample was built to work.
What a replay shows: Seeded on purpose, and scored separately.
What a demonstration shows: The vendor’s product names and categories.
What a replay shows: Your medicines, your misspellings, your routes, your clocks.
What a demonstration shows: That the system is confident.
What a replay shows: What it does when a check cannot be closed — name it, or fill it in.
What a demonstration shows: Notes and an impression.
What a replay shows: Four scores with their denominators, an adjudicated disagreement list, and a written record of what you could not measure.
What questions should you ask a complaint management vendor?
Ten, and each has a right answer that is specific rather than reassuring.
Ten questions to ask, and what a good answer sounds like
- Will you run a read-only replay on our own closed inquiries, before any connection?
A good answer is yes, with a named data requirement and no downstream integration. A bad one defers it to implementation.
- What exactly do you need from us to start?
Procedures, approved answers, a product dictionary and a sample. If the list includes a live connection to your quality system, the pilot is not read-only.
- How do we add our own misspellings and nicknames?
It should be a setting you edit, not a change request. Your callers will not spell your medicines the way your label does.
- Show me an inquiry where a required check could not be closed.
Watch whether the product has a state for it. A system that always produces a complete output has no way to tell you it was guessing.
- Which outputs can leave without a person, and can that be switched off?
Ask per output: the answer to the requester, the quality hand-off, the support request. Get the answer in writing.
- What is written down as the role and scope of each model?
One page, in the terms the regulators’ own AI documents use. A vendor who has never written it will improvise, and you will hear it.
- How is the received time recorded, and who can change it afterwards?
It should be stamped at first receipt, in your system, and be unchangeable by any later step. It is one of the most consequential fields in the file.
- How would we find a hand-off that was sent and never arrived?
The only honest answer is a match against the receiving systems. A report of what the product sent cannot find it.
- What do you monitor after go-live, and on what schedule?
The joint EMA and FDA principles ask for scheduled monitoring and periodic re-evaluation. Ask what the schedule is and who sees the result.
- If we leave, what comes with us?
Your standard responses, your dictionary, the inquiry history and the audit trail, in a form another system can load.
Writing down what you cannot measure
A replay cannot test everything, and the honest move is to list what it missed rather than let the gaps pass as passes. NIST makes this a measurement step in its own right: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented" (NIST AI 100-1, MEASURE 1.1).
On a medical information replay the list is short and predictable. A closed inquiry cannot tell you whether a live prompt would have changed what the caller said — a specialist who is asked to request the lot number mid-call gets an answer that does not exist in any historical file. It cannot tell you what quality or patient support did with a better hand-off, because the downstream system was never connected. It cannot measure tone, and it cannot measure weekend behaviour unless your sample contains a weekend. And it cannot tell you what happens at four times the volume.
Write those down before the decision, and write them in words the people reading the decision will understand — the joint principles ask that "Plain language is used to present clear, accessible, and contextually relevant information to the intended audience, including users and patients, regarding the AI technology's context of use, performance, limitations, underlying data, updates, and interpretability or explainability" (EMA and FDA, guiding principles, principle 10). A limitations list written in the vendor's vocabulary is a limitations list nobody will read.
The same discipline applies to this product's own figures. The solution's stated median of 11 minutes from receipt to a sent hand-off, and its 99.6% of hand-offs inside the one-business-day clock, are design targets rather than measured results on any customer's desk. Treat them as two of the things a pilot should test, not two of the things it will reproduce.
How long does it take to go live with complaint management software?
Longer than the pilot and shorter than procurement timetables often assume, and the order matters more than the total. The system is built and deployed from the customer's own procedures and documents — intake procedures, standard responses, the product dictionary and a sample of past inquiries — and goes live once every check has passed. From there, turn capability on in four steps rather than one.
The order to switch things on, least consequential first
- Read-only replay2 weeks
- Drafting, nothing sentin parallel
- Hand-offs, confirmedlive
- First signed monthmonth end
Two details are worth holding firm on. Hand-offs should go live with human confirmation from the first day and stay that way — in this product, no agent can send one at all. And the first monthly reconciliation should be run even if nothing seems wrong, because it is the only step that can find a hand-off that was sent and never arrived, which is precisely the failure that leaves no trace anywhere else.
After go-live the measurement does not stop. The joint principles ask that "The AI technologies undergo scheduled monitoring and periodic re-evaluation to ensure adequate performance (e.g., to address data drift)" (EMA and FDA, guiding principles, principle 9). The simplest schedule is the one the desk already has: the monthly match, read as a measurement rather than as an administrative chore, plus a small repeat replay each quarter on inquiries closed since the last one.
What a scored replay looks like inquiry by inquiry

Questions people ask
What is a complaint management system?
For a medical information desk it is the system that takes a described product problem, captures the lot, expiry and whether a sample can be returned, records the complainant's own words and the date the company first became aware, and passes a complete file to the quality team that investigates it. In a desk like this one the intake side and the quality system are separate: the desk recognises and hands over, and quality investigates.
How do you evaluate complaint management software?
On your own closed inquiries, against what your own people recorded, with the pass criteria written before the first result is seen. Score four things with their denominators — second requests found, complaints recognised and declined, identifiers captured, received date and route — adjudicate every disagreement, and have someone who did not run the pilot do the scoring. A demonstration cannot produce any of that.
Can you test AI on past calls?
Yes, and it is the cheapest useful test available. Replay transcripts, emails and forms you have already closed, with nothing connected downstream, and compare what the system found with what was recorded at the time. The limit is real and should be written down: a past call cannot tell you whether a live prompt would have changed what the caller said, because the answer to a question nobody asked is not in the file.
Do you need live system connections for a pilot?
No, and you should refuse them for the first fortnight. A read-only replay needs your procedures, your approved answers, your product dictionary and an exported sample. Connecting the quality or patient support system turns an evaluation into an implementation, and makes every disagreement expensive to investigate instead of free.
What score should a pilot be looking for?
Whatever you wrote down on day 2, which should be set against what your desk achieves today rather than against a published figure from another setting. Published results on machine triage come from small literatures — one systematic review of language processing for triaging referrals found ten usable studies out of 4,225 screened — and from tasks that are not yours. The comparison that decides the purchase is the system against your current desk, on the same inquiries.
Who should be in the room when the replay is scored?
The medical information lead, someone from quality, a second-line or medical reviewer for the cases that turn on clinical judgement, and a scorer who did not configure the pilot. Standards guidance asks for assessors who were not front-line developers of the system for exactly this reason: the person who tuned the configuration is the worst-placed person to decide whether its output was right.