WorkflowMedical Affairs

Buying guide · a fortnight you run yourself

Complaint management software: a two-week pilot on your own calls

Nobody can judge a medical information system from a demonstration, because the interesting cases are the ones the sample data does not contain. This guide sets out a fortnight's replay on inquiries you have already closed, the four things to score, and the questions to put to a vendor — including how Medical Inquiry Intake & Triage would be tested.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

Complaint management software for a medical information desk is bought on a demonstration and judged, eighteen months later, on the cases the demonstration did not contain. The way out of that is unglamorous and cheap: take a fortnight of inquiries you have already closed, replay them through the system with nothing connected downstream, and score what it found against what your own people recorded at the time. This guide sets out that fortnight day by day, the four things worth scoring, how to seed the awkward cases deliberately, and the ten questions to put to a vendor before any of it starts. Botminds' Medical Inquiry Intake & Triage is designed to be bought this way, and the page says plainly which of its own numbers a pilot should expect to reproduce and which it should not.

The screens here are the product's sample data on fictional people and medicines. One boundary applies throughout: this work routes product complaints to quality and access requests to patient support, and it does not classify, assess or report adverse events — that stays with the safety team the desk hands over to, and a pilot should not be scored on it.

What a replay needs, and what the published literature on machine triage actually rests on

4

things to bring: intake procedures, approved answers, your product dictionary, and a sample of closed inquiries

2weeks

from freezing the sample to a written decision, with no downstream connection

10

studies survived screening in the systematic review of language processing for triaging referrals

from 4,225 titles and abstracts

2,937

questions decomposed from 1,467 consumer health requests — the baseline for the second-request score

The first two are this guide's own plan. The third is the yield of a 2026 systematic review of natural language processing used to triage outpatient referrals; the fourth is the decomposition that gives a replay something to compare its second-request rate against.Source: Fung et al., Front Health Serv 6:1797583, April 2026; Roberts et al., AMIA Annu Symp Proc 2014:1018–1027

What should complaint management software do?

For a medical information desk, five things, and they are worth stating before any feature list arrives. It should take an inquiry from every channel the desk runs and stamp the minute it was first received, not the minute a form was opened. It should split the inquiry into every separate request it holds. It should recognise when a product defect has been described, and capture the lot, the expiry and whether a sample can be returned while the caller is still reachable. It should route each request to the team that owns it with its own clock. And it should leave a file another department can act on, with the requester's own words behind each field and a named person confirming before anything is sent.

Nothing on that list is about the user interface, and all of it is testable on inquiries that have already happened. That is the point of what follows. What inspectors actually cite on complaint handling is a useful second list to read beside it, because the citations are about exactly these properties of the file.

A two-week pilot on inquiries you have already closed

A replay has one advantage no demonstration can match: you already know the right answer. Your own people worked these inquiries, found the questions they found, sent what they sent. Running the system over the same material turns an argument about capability into a count of agreements and disagreements, with every disagreement readable in full.

It is also the posture standards bodies ask for. NIST's AI risk framework puts it in one line: test an AI system before it goes live, and keep testing it at intervals once it is running (NIST AI 100-1, January 2023). Before deployment is the cheap half; the fortnight below is one way to spend it.

A fortnight on one page: who owns each step and what it leaves behind

Days 1–2

Pick the sample and freeze it

A fortnight of closed inquiries across every channel, with the outcome your own people recorded for each — the questions found, the route taken, the lot, the received date. Frozen, so nothing changes under the test.

OwnerMedical information lead

Day 2

Write down the role and scope before anyone sees a result

One page: which decisions the system is being tested on, which it is not, and what would count as a pass. Signed before the replay starts.

OwnerLead with quality

Days 3–4

Load your own procedures and answers

Intake procedures, approved standard responses, the product dictionary with your misspellings and nicknames, and the triage rules and clocks as your desk actually runs them.

OwnerVendor with your lead

Day 5

Seed the hard cases

A short list of deliberately awkward inquiries added to the sample: a missing reply contact, two plausible medicines, a lot the register does not hold, a second question in a second paragraph.

OwnerQuality and a second-line reviewer

Days 6–8

Replay, read-only

Every inquiry read and triaged, with nothing sent anywhere. Each output carries the passage it came from, so a disagreement can be adjudicated rather than argued.

OwnerThe system, nobody downstream

Days 9–10

Score the four things

Agreement, per class, with the denominator beside each number: second requests, identifiers, received dates, routes.

OwnerA reviewer who did not run the pilot

Day 11

Adjudicate the disagreements

Every difference read in full, and classified: the system was wrong, your own file was wrong, or the case was genuinely ambiguous. The third pile is the interesting one.

OwnerLead, quality, second-line reviewer

Day 12

Write down what you could not measure

The list of things a replay cannot test — tone on a live call, whether a prompt would have changed what the caller said, what quality did with the complaint afterwards.

OwnerLead with quality

Days 13–14

Decide, in writing

A recommendation against the pass criteria written on day 2, with the adjudicated scores, the unmeasurable list, and the go-live order if the answer is yes.

OwnerThe buying group

This guide's own plan, written for a medical information desk evaluating a triage and intake system. The day numbers are working days and assume the sample can be exported; where it cannot, days 1–2 are the longest part of the exercise.

Day 2 is a step pilots often skip, and the one that decides whether the result means anything. FDA's draft guidance on AI supporting regulatory decision-making asks for the context of use to be written down first — what job the model does, where that job stops, what exactly is modelled and how its outputs will be used (FDA, draft guidance, January 2025, step 2). Write the pass criteria on day 2 and a disappointing score is information. Write them on day 13 and they will be whatever the score turned out to be.

What data do you need to start a complaint management pilot?

Four things, and none of them is a system connection. Your intake procedures, so the system routes the way your desk is meant to. Your approved, current standard responses, so a match can be judged as a match. Your product dictionary, including the misspellings and nicknames your callers actually use — the product's own sample dictionary takes "veltremab" and "corvanta" for exactly this reason. And a sample of past inquiries across every channel, each with the outcome your own people recorded.

That last one is where the effort is. The sample has to carry not only the inquiry but the answer key: which questions were found, which were routed where, what lot was captured, what date was recorded as the receipt. If your desk cannot export that, the pilot has found something before it has run — because the same gap is what makes a complaint file hard to produce for an inspection.

The rules the pilot is configured with, before anything is replayed

The settings screen: triage rules shown as switches with their options — split every inquiry into its questions, ask for a reply contact when none is given, device problems always open a complaint, prompt the specialist during live calls, read back every question before the call ends. Below them the clocks: quality and patient support hand-offs in one business day, standard answers in two business days, custom and medical review answers in five. Then who receives each hand-off and who may confirm one, and the product dictionary.
Sample data in the product. Each row is a setting rather than a behaviour fixed in the software, which is what makes a replay meaningful: the pilot runs your desk's rules, not the vendor's defaults.

The four things to score, and how to score them

Resist scoring everything. Four measures carry most of the decision, and each has a denominator the sample can supply.

The scorecard a replay fills in — the denominators are the pilot's, the numbers are yours

What to scoreHow to score itOut of

Second requests found

How to score itOf the inquiries your own people recorded as holding more than one request, how many did the system split the same way? And how many did it split that your people did not?

The second number matters as much as the first: a request your file never recorded is not automatically a false positive.

Out of__ of __Inquiries in the sample

Product complaints recognised

How to score itOf the inquiries your file records as having produced a complaint, how many did the system open one for — and did it decline the storage mistakes your people also declined?

Score the declines as well. A system that opens a complaint on everything is not accurate, it is loud.

Out of__ of __Complaints in the sample

Identifiers captured

How to score itWhere a lot, expiry or device identifier exists in the file, did the system read the same one — and did it flag the ones that are not in your batch register?

Transpositions are the failure to look for. Compare character by character, not by eye.

Out of__ of __Inquiries with an identifier

Received date and route

How to score itDid the received date match the date of first receipt in your own file, and did each request go to the team your people sent it to?

A route disagreement is often a procedure disagreement. Read ten of them before calling any of them an error.

Out of__ of __Requests in the sample

This guide's scorecard, left blank on purpose. Published accuracy figures from other settings do not transfer to your inquiry stream; the only number that means anything is the one your own sample produces, with the denominator printed beside it.

Two rules make the numbers honest. Print the denominator next to every score, because "94% accurate" over eleven cases is not a measurement. And count the disagreements in both directions: a second request the system found that your file never recorded is not automatically wrong, and in the replays worth running it is often the most valuable row on the sheet.

Seeding the hard cases on purpose

A fortnight of ordinary inquiries will be mostly ordinary, and a system can score well on it while failing every case that matters. So add the awkward ones deliberately, from your own history where you have them: an inquiry with no reply contact at all; one where two of your medicines could plausibly match what the caller said; a lot number that is not in your batch register; a second question buried in the second paragraph of an email about something else; a question about a dose or a route the label does not describe; a misspelling nobody has added to the dictionary.

Score those separately, and expect a worse result on them. The useful question is not whether the system gets them right but what it does when it does not — whether it names the open check or quietly fills it in. Which of the agents may decide what sets out how that line is drawn in this product.

Who runs the scoring matters as much as what is scored. NIST asks that the regular checks on a system be done by someone other than the people who built it day to day — an internal expert from outside the build team, an independent assessor, or both (NIST AI 100-1, MEASURE 1.3). The pilot's equivalent is simple: the person who configured the replay should not be the person who scores it.

How do you measure whether triage is accurate?

By agreement with a known answer, per class, on your own material — not by a single accuracy number. Take each of the four measures above, compare the system's output with what your file records, and put every disagreement in front of someone who can decide which side is right. Three piles come out: the system was wrong, your own file was wrong, and the case was genuinely ambiguous. A pilot that produces no third pile has not looked hard enough at its sample.

The unit of assessment is the desk, not the model. The joint EMA and FDA principles are explicit about it. Performance should be assessed according to risk and across the complete system, including the way people and the AI work together, with data and measures fit for the intended use (EMA and FDA, guiding principles, principle 8, January 2026). In practice that means scoring what a specialist would have confirmed, corrected or rejected — the draft plus the person — rather than the draft alone.

Where a replay's numbers come from once the system is running

The dashboard: inquiries read, hand-offs made, time from first receipt to hand-off, hidden asks caught, a breakdown of who is asking, findings by product, and the top topics with the standard response used for each, all for the selected period against the previous one.
Sample data in the product. In a read-only replay these panels are the pilot's output: the counts come from the sample you froze, and each one can be opened to the inquiries behind it.

What published accuracy numbers do and do not tell you

They tell you the task is tractable, and almost nothing else. In a peer-reviewed comparison against physician performance on clinical vignettes, the frontier models triaged 133 of 144 cases correctly (92.4%), and a collaborative set-up 44 of 48 (92%) (Sorich et al., 2024). That is a real result on 144 cases of a different task, with a different definition of correct, in a different setting. It is not a forecast of what your desk will see.

The systematic review of the closest published work makes the point from the other direction. Screening the literature on natural language processing for triaging outpatient referrals, the authors report that "A total of 4,225 titles and abstracts were reviewed resulting in 26 full-text reviews. A total of 10 studies were used for data extraction and synthesis" (Fung et al., 2026) — ten studies, for an entire field. And their conclusion is the sentence every buyer should carry into a vendor meeting: "As most studies relied solely on retrospective data, further studies focused on prospective effectiveness and clinical impact should be prioritized."

Retrospective is exactly what a two-week replay is. That is not a reason to skip it — it is the cheapest honest evidence available before go-live — but it is a reason to write down that the replay measured agreement with a past file and not effect on a live desk, and to plan the prospective check that follows.

A vendor demonstration against a replay on your own closed inquiries

What a demonstration showsWhat a replay shows
The material

What a demonstration shows: Sample data chosen by the vendor, where every case has a clean answer.

What a replay shows: Your own inquiries, including the ones your people argued about.

The answer key

What a demonstration shows: None. Nothing on screen can be wrong.

What a replay shows: What your own people recorded at the time, case by case.

The hard cases

What a demonstration shows: Absent, because the sample was built to work.

What a replay shows: Seeded on purpose, and scored separately.

Vocabulary

What a demonstration shows: The vendor’s product names and categories.

What a replay shows: Your medicines, your misspellings, your routes, your clocks.

What you learn about doubt

What a demonstration shows: That the system is confident.

What a replay shows: What it does when a check cannot be closed — name it, or fill it in.

What you can show afterwards

What a demonstration shows: Notes and an impression.

What a replay shows: Four scores with their denominators, an adjudicated disagreement list, and a written record of what you could not measure.

Our comparison, written for a selection committee. It is an argument about evidence, not a claim about any particular vendor's demonstration.

What questions should you ask a complaint management vendor?

Ten, and each has a right answer that is specific rather than reassuring.

Ten questions to ask, and what a good answer sounds like

  1. Will you run a read-only replay on our own closed inquiries, before any connection?

    A good answer is yes, with a named data requirement and no downstream integration. A bad one defers it to implementation.

  2. What exactly do you need from us to start?

    Procedures, approved answers, a product dictionary and a sample. If the list includes a live connection to your quality system, the pilot is not read-only.

  3. How do we add our own misspellings and nicknames?

    It should be a setting you edit, not a change request. Your callers will not spell your medicines the way your label does.

  4. Show me an inquiry where a required check could not be closed.

    Watch whether the product has a state for it. A system that always produces a complete output has no way to tell you it was guessing.

  5. Which outputs can leave without a person, and can that be switched off?

    Ask per output: the answer to the requester, the quality hand-off, the support request. Get the answer in writing.

  6. What is written down as the role and scope of each model?

    One page, in the terms the regulators’ own AI documents use. A vendor who has never written it will improvise, and you will hear it.

  7. How is the received time recorded, and who can change it afterwards?

    It should be stamped at first receipt, in your system, and be unchangeable by any later step. It is one of the most consequential fields in the file.

  8. How would we find a hand-off that was sent and never arrived?

    The only honest answer is a match against the receiving systems. A report of what the product sent cannot find it.

  9. What do you monitor after go-live, and on what schedule?

    The joint EMA and FDA principles ask for scheduled monitoring and periodic re-evaluation. Ask what the schedule is and who sees the result.

  10. If we leave, what comes with us?

    Your standard responses, your dictionary, the inquiry history and the audit trail, in a form another system can load.

Our questions, written for a first technical session with any vendor in this category. Each is a case a product either has a specific answer for or does not.Source: EMA and FDA, Guiding principles of good AI practice in drug development, January 2026; FDA, Considerations for the Use of AI to Support Regulatory Decision-Making, draft guidance, January 2025

Writing down what you cannot measure

A replay cannot test everything, and the honest move is to list what it missed rather than let the gaps pass as passes. NIST makes this a measurement step in its own right: "The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented" (NIST AI 100-1, MEASURE 1.1).

On a medical information replay the list is short and predictable. A closed inquiry cannot tell you whether a live prompt would have changed what the caller said — a specialist who is asked to request the lot number mid-call gets an answer that does not exist in any historical file. It cannot tell you what quality or patient support did with a better hand-off, because the downstream system was never connected. It cannot measure tone, and it cannot measure weekend behaviour unless your sample contains a weekend. And it cannot tell you what happens at four times the volume.

Write those down before the decision, and write them in words the people reading the decision will understand — the joint principles ask that "Plain language is used to present clear, accessible, and contextually relevant information to the intended audience, including users and patients, regarding the AI technology's context of use, performance, limitations, underlying data, updates, and interpretability or explainability" (EMA and FDA, guiding principles, principle 10). A limitations list written in the vendor's vocabulary is a limitations list nobody will read.

The same discipline applies to this product's own figures. The solution's stated median of 11 minutes from receipt to a sent hand-off, and its 99.6% of hand-offs inside the one-business-day clock, are design targets rather than measured results on any customer's desk. Treat them as two of the things a pilot should test, not two of the things it will reproduce.

How long does it take to go live with complaint management software?

Longer than the pilot and shorter than procurement timetables often assume, and the order matters more than the total. The system is built and deployed from the customer's own procedures and documents — intake procedures, standard responses, the product dictionary and a sample of past inquiries — and goes live once every check has passed. From there, turn capability on in four steps rather than one.

The order to switch things on, least consequential first

From replay to a signed month
Read-only replay2 weeks
Drafting, nothing sentin parallel
Hand-offs, confirmedlive
First signed monthmonth end
  • Read-only replay2 weeks
  • Drafting, nothing sentin parallel
  • Hand-offs, confirmedlive
  • First signed monthmonth end
This guide's recommended order, not a delivery schedule. Read-only replay proves the reading; drafting beside your live desk proves the writing without risk; confirmed hand-offs put a person between the system and another department; and the first monthly match is the step that tells you whether anything has been quietly going missing all along.

Two details are worth holding firm on. Hand-offs should go live with human confirmation from the first day and stay that way — in this product, no agent can send one at all. And the first monthly reconciliation should be run even if nothing seems wrong, because it is the only step that can find a hand-off that was sent and never arrived, which is precisely the failure that leaves no trace anywhere else.

After go-live the measurement does not stop. The joint principles ask that "The AI technologies undergo scheduled monitoring and periodic re-evaluation to ensure adequate performance (e.g., to address data drift)" (EMA and FDA, guiding principles, principle 9). The simplest schedule is the one the desk already has: the monthly match, read as a measurement rather than as an administrative chore, plus a small repeat replay each quarter on inquiries closed since the last one.

What a scored replay looks like inquiry by inquiry

The inquiries list: every question from every channel with the channel it arrived on, who is asking, the product, what the agents found, where it is routed, its clock and its stage. Filters across the top narrow the list by channel, by product, or to inquiries that held a request the requester did not put up front.
Sample data in the product. The filter a pilot lives in is the one for inquiries that held more than one request: those rows are where agreement with your own file is worth reading case by case.

Questions people ask

What is a complaint management system?

For a medical information desk it is the system that takes a described product problem, captures the lot, expiry and whether a sample can be returned, records the complainant's own words and the date the company first became aware, and passes a complete file to the quality team that investigates it. In a desk like this one the intake side and the quality system are separate: the desk recognises and hands over, and quality investigates.

How do you evaluate complaint management software?

On your own closed inquiries, against what your own people recorded, with the pass criteria written before the first result is seen. Score four things with their denominators — second requests found, complaints recognised and declined, identifiers captured, received date and route — adjudicate every disagreement, and have someone who did not run the pilot do the scoring. A demonstration cannot produce any of that.

Can you test AI on past calls?

Yes, and it is the cheapest useful test available. Replay transcripts, emails and forms you have already closed, with nothing connected downstream, and compare what the system found with what was recorded at the time. The limit is real and should be written down: a past call cannot tell you whether a live prompt would have changed what the caller said, because the answer to a question nobody asked is not in the file.

Do you need live system connections for a pilot?

No, and you should refuse them for the first fortnight. A read-only replay needs your procedures, your approved answers, your product dictionary and an exported sample. Connecting the quality or patient support system turns an evaluation into an implementation, and makes every disagreement expensive to investigate instead of free.

What score should a pilot be looking for?

Whatever you wrote down on day 2, which should be set against what your desk achieves today rather than against a published figure from another setting. Published results on machine triage come from small literatures — one systematic review of language processing for triaging referrals found ten usable studies out of 4,225 screened — and from tasks that are not yours. The comparison that decides the purchase is the system against your current desk, on the same inquiries.

Who should be in the room when the replay is scored?

The medical information lead, someone from quality, a second-line or medical reviewer for the cases that turn on clinical judgement, and a scorer who did not configure the pilot. Standards guidance asks for assessors who were not front-line developers of the system for exactly this reason: the person who tuned the configuration is the worst-placed person to decide whether its output was right.

Medical Inquiry Intake & Triage.
See it on your own files.

An agentic solution from Botminds. Every inquiry from every channel read, split into its questions and routed, including the complaint or second question mentioned on the way out. Book a demo, or explore how it works.