AI analysis of field medical insights is qualitative data analysis: letting a model do what a human coder does to free text: read each passage, assign it to a code in a scheme somebody wrote, and group the coded passages into themes. It works, within limits that recent peer-reviewed studies have begun to quantify — and the quantification is where buying decisions go wrong, because three quite different measurements are all reported as accuracy and only one of them is about agreeing with an expert. The work is hard for the same reason it is valuable: a quarter of medical affairs insights is thousands of passages written by dozens of people, and consistency across that volume is exactly what tired humans lose and models keep. Botminds' Medical Insights Management puts a coder, a clusterer and a report writer over five channels of that free text, with a stated limit on each: uncertain codes go to the insights lead, a merge or split is proposed rather than applied, and no sentence of a theme statement exists without a citation.
This page is written to be used against any product that codes your notes, including this one. It covers what the published figures measure, why self-consistency is the most misleading of them, where models drift, and the test protocol to run on a quarter you have already coded by hand.
What two peer-reviewed agreement studies actually reported
0.76 · 0.78
chance-corrected agreement with human coders on two categories of 778 clinician free-text comments, after structured prompting and calibration
0.67
the same study’s starting point on the first category with a simple prompt — while raw agreement read 76%
71%
of the themes human analysts found that two general models also found, coding 40 text messages inductively
36–47%
coding agreement between those models and the human coders, even on the themes both found
Can AI do qualitative analysis of free-text notes?
For the coding step, yes, and the published evidence supports it at a useful level. Coding 778 clinicians' free-text comments against a scheme written in advance, a general model "achieved substantial agreement with human coding (κ = 0.76 for Category 1; κ = 0.78 for Category 2)" once the prompts were structured and calibrated (Jeong and colleagues, JAMIA Open 2025). That is a real result on clinical free text, the closest published material to a field insight note.
Two caveats sit inside that sentence and both matter commercially. The first is "with structured prompts and calibration": the figure is post-tuning, so it describes what the method can reach with effort on a known corpus, not what it gives you on day one on yours. The second is the kind of reading involved. A study comparing human and model thematic analysis of health text messages found the models recovered most of the human themes but agreed with the human coding far less often, and concluded that "human coders appeared to be better than GenAI at identifying nuanced and interpretative themes" (Prescott and colleagues, JMIR AI 2024) — and a medical affairs scheme is full of broad, interpretative codes such as "access" or "unmet need".
What the evidence does not support is the step after coding. Deciding that a theme matters, that it is worth a medical director's attention and that it should change the evidence plan is not a reading task, and no agreement figure speaks to it. That is the division the solution draws, and it is why the test for what counts as an insight is a human instrument rather than a model's output.
What the published agreement numbers actually measure
Each study answers a different question, on a different corpus, with a different comparator. Lining them up makes the shape of the evidence obvious: it is encouraging, it is thin, and almost none of it is from this field.
Three published results side by side, and what each one does not prove
| The study | What was coded | What was compared | The result | What it does not prove |
|---|---|---|---|---|
| Jeong and colleagues, JAMIA Open (2025) | 778 free-text comments clinicians left on a clinical deterioration alert, in two categories | A general model against human coding, with prompt strategies tested on a 20% subsample first | κ = 0.76 and 0.78 after calibration; 0.67 on the first category with the simplest prompt | That an untuned model reaches the same level, or that a field insight corpus behaves like alert comments |
| Prescott and colleagues, JMIR AI (2024) | 40 text messages from a health intervention, analysed inductively and deductively by two human teams | Two general models, run by an analyst, against the human themes and coding | 71% of human themes found inductively, 50–58% deductively; coding agreement of 36–47% | That themes a model finds are coded the same way a person would code them |
| McHugh, Biochemia Medica 22(3) (2012) | Not a model study: the methodological reference for how agreement is judged | Percent agreement against agreement corrected for chance | Below 0.60 "one can surmise that about half the data may be incorrect" | Anything about models. It is the yardstick, and it predates all of this |
Self-consistency is not agreement with an expert
A figure often quoted for these tools is not in that table at all, and the distinction is the single most useful thing a buyer can learn. Run a model twice over the same notes and compute agreement between the two runs, and you get a self-consistency figure. It measures whether a model says the same thing twice. It does not measure whether what it says is right.
That is a worthwhile property — a coder whose answers change between Tuesday and Thursday is unusable. But a confidently and consistently wrong coder scores extremely well on this measure. If a vendor quotes a figure above 0.9, the first question is: agreement with whom?
The confusion is old. Chance-corrected agreement exists because raw percentages flatter. Jacob Cohen objected to plain percent agreement as long ago as 1960, because it cannot allow for the agreement two raters reach by chance (McHugh, 2012). With an unbalanced scheme — one code covering a third of your insights, a dozen covering a few each — two raters who both guess the common code will agree most of the time and have demonstrated nothing.
How accurate is AI thematic analysis, and how would you know?
One test yields several numbers, and they are not interchangeable. Running them separately is what stops a demonstration from being an argument about a single figure nobody can define.
One test of 300 insights, four numbers that all get called accuracy
96%
Run-to-run consistency
The model given the same 300 insights twice, agreeing with itself. Says nothing about whether either run was right.
94%
Raw agreement with the insights lead
The share of the 300 where the model and the lead chose the same topic code. Flattered by any code that is common.
0.71
Agreement corrected for chance
The same 300, with agreement you would expect by chance removed. The number a methodologist will ask for.
78%
Matched the decision actually taken
Of the themes the lead promoted, the share the system also surfaced as worth a decision. The only one the medical director feels.
Read down that list and the ranking inverts. The first figure is the easiest to achieve and the least informative. The last is the hardest to measure, because it requires knowing what you decided at the time, and it is the only one a medical director would recognise as the thing they care about. A product that scores well on the first three and badly on the last is a tidy coder that surfaces the wrong themes.
The middle two deserve one more warning. Raw agreement and chance-corrected agreement are computed from the same test and can disagree by a wide margin — 76% against 0.67 in the alert-comment study — which is why a quoted percentage with no stated method should be treated as uninformative rather than as good news. Where both are reported, the gap between them tells you how unbalanced the scheme is — which is itself worth knowing, and is a design question taken up in building a coding scheme that still holds next quarter.
The floor to insist on, and why 0.60 is not a generous bar
For health-related work the conventional floor is 0.60 on the chance-corrected measure, and the reason is arithmetic rather than custom: below 0.60, the confidence interval around a kappa is so wide that roughly half the data may be wrong (McHugh, 2012). Half the data. That is not a cautious academic's preference; it is a statement about how much of a quarter you would be willing to get wrong.
Two practical consequences follow. First, 0.60 is a floor for a dimension, not for a system: topic, sentiment and segment should each clear it separately, because a system that codes topic at 0.78 and sentiment at 0.41 will report an average that conceals a dimension you should not use. Second, the floor applies to the codes that matter rather than to the corpus average — a rare code that appears in 2% of insights can be coded badly without moving any overall figure, and rare codes are frequently the interesting ones.
Sample size has a floor too, and it is lower than people fear: "as a general heuristic, sample sizes should not consist of less than 30 comparisons", with larger samples giving a more precise estimate. Thirty is the minimum for the statistic to mean anything; a few hundred insights is a realistic test and well within what a team can hand-code in a week.
Where models drift: broad codes, rare codes, sentiment, sarcasm
Four failure modes recur, and knowing them lets you design a test that finds them rather than one that passes.
Broad codes. The published comparisons found models weakest on the nuanced, interpretative themes, and medical affairs schemes are full of them. "Unmet need" and "access" absorb anything, and a model will put things there for the same reason a tired human does. The remedy is definitional: every code needs a boundary statement and an example of something that is near it and excluded.
Rare codes. A code with twenty instances in a quarter is both the hardest to learn from examples and often the most valuable — a safety signal, a competitor's new data, a reimbursement change. Test them deliberately by over-sampling them, or they will be invisible in the overall figure.
Sentiment. Clinicians are polite, and politeness defeats sentiment coding. "That is certainly an interesting approach" is not positive. Sentiment is the dimension most likely to fall below the floor while topic coding looks fine, which is why it should be scored and reported on its own.
Drift over time. A model, a prompt or a scheme that changes will change the coding, and a trend computed across the change is not a trend. The discipline is to keep a small held-out set of hand-coded insights and re-run it after every change to any of the three, before the quarter's numbers are published.
The measurement protocol, once you have chosen the quarter to test on
- At least 30 comparisons, and a few hundred if you can
Thirty is the published minimum for the statistic to carry meaning. Below it the interval is so wide that "no agreement" sits inside it.
- Code blind, and fix the reference first
The hand coding is done, and locked, before anyone sees the system’s output. A reference adjusted after the fact measures nothing.
- One gold standard, written down
Where two human coders disagreed, record which reading is the reference and why. Unresolved human disagreement becomes noise attributed to the model.
- Score each dimension separately
Topic, sentiment and segment produce different figures. A single blended number hides the dimension you should stop using.
- Over-sample the rare codes
A code in 2% of insights contributes almost nothing to a corpus figure and is often the one that matters. Judge it on its own sample.
- Report raw agreement and the chance-corrected figure together
The gap between them is information: it tells you how unbalanced the scheme is, and it stops a flattering percentage standing alone.
- Keep a held-out set, and re-run it after every change
Any change to the model, the prompt or the scheme invalidates the comparison with last quarter unless the held-out set says otherwise.
Invented themes, and the citation rule that prevents them
The failure that costs most is not a miscode; it is a plausible theme that nobody said. The mechanism is well described in the medical affairs literature: a model "learns to generate texts by identifying patterns in the training data, but it lacks the capability to verify the accuracy of the information it generates", so its outputs "may contain inaccuracies, errors, and false information, which is often termed as hallucination" (Ng and Zhang, 2023).
In an insights programme the consequence is specific and severe. A theme statement that attributes a concern to clinicians who never raised it, reaching a medical director in a quarterly report, is worse than a missed theme: it generates evidence work, it is cited in a decision memo, and it is almost impossible to retract gracefully once a leadership team has acted on it.
The structural defence is a citation rule rather than a better model. If every sentence of a theme statement has to point at the insights behind it, and a reader can open any of them in one click to see the passage highlighted, an invented sentence has nowhere to live — it either has citations that do not support it, which a reviewer can see, or it has none and does not get written. That is the rule the report writer in this solution works to, and it is the single feature to insist on in any product in this category.
A theme statement where every sentence opens the notes behind it

The three decisions a model should not be offered
Three decisions do not move, and a product that offers to take them should be refused rather than configured. The first is promotion: whether a theme becomes a key insight is a judgement about what the organisation should pay attention to. The second is merging: combining two themes rewrites the quarter's history and its counts, so it is proposed and never applied silently. The third is release: what leaves medical affairs is a governance decision with a named approver.
Review is the fourth thing a person does, and it is the one that is easy to under-resource. The accurate version is that review work does not vanish, it changes kind — from making every code to confirming the uncertain ones — and that confirming is harder per item than coding, because it requires judging a decision you did not make. Budget for it as a standing part of the week rather than as a transition cost, as set out in who owns each step of a field insight.
What each agent does, and what it is not allowed to do
Agent
Insight coderApplies the coding scheme — topic, product, sentiment, segment — to everything that arrives, and flags a question the clinician wants answered in writing.
Never: resolves an uncertain code on its own; it goes to the insights lead
Agent
Theme clustererGroups the coded insights by meaning, names each theme in plain words, and tracks it week by week and region by region.
Never: applies a merge or a split without the insights lead
A person decides
The insights lead decidesPromotes, keeps watching, merges or dismisses, with a reason that stays attached to the theme.
Agent
Report writerDrafts the quarterly report section by section from the decisions that were made, with a citation on every sentence.
Never: writes a sentence without a citation, or presents an undecided theme as settled — those are marked for discussion
Privacy, the audit trail, and the records you keep of a model's work
Three record-keeping questions come up in every procurement and all three have practical answers. What leaves your own systems: free-text insights contain clinicians' names and sometimes patient detail, so the design that avoids most of the problem is to de-identify at intake and keep identity in the source system, which is what this solution does. What is retained by any model provider, and for how long, is a contractual question to settle before the pilot rather than after it.
What is kept of the model's own work is the question teams forget. The federal risk framework's instruction is unambiguous on both points: test a system before it goes live and at intervals once it runs, and document how it works and how far it can be trusted as part of measuring its risk (NIST AI RMF 1.0). In an insights system that means a record of which scheme version, which model and which prompt produced each code, so that a code can be explained a year later — and so that a trend spanning a model change can be identified as one.
On oversight, European law now states the design requirement for systems it classifies as high-risk: they "shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use" (Regulation (EU) 2024/1689, Article 14(1)). Whether a medical insights system falls in that class depends on the deployment and is a question for your own legal colleagues; what is clear is the direction, and a system built so that a person can see, question and overturn each proposed theme meets the standard whether or not it is obliged to.
The recommendation, its confidence, and the four decisions open to a person

Questions people ask
How do you test an AI coder against human coders?
Take a closed quarter you have already coded by hand, hold the hand coding back as the reference, and let the system code the same insights without tuning on the answers. Score each dimension separately — topic, sentiment, segment — report both raw agreement and agreement corrected for chance, and read the disagreements rather than the totals. Then re-run the same held-out set after any change to the scheme, the model or the prompt, because that is where drift shows up.
What is a good kappa for AI coding?
Treat 0.60 as the floor rather than the target, per dimension and per important code. The methodological reference for health research is blunt about what sits below it: when agreement corrected for chance falls under 0.60, the confidence intervals are wide enough that about half the data may be incorrect. Above 0.80 is strong. Be suspicious of anything above 0.90 unless you know what it was measured against, because a figure that high is often a model compared with its own reruns rather than with a person.
How many insights do you need in a test sample?
At least thirty comparisons for the statistic to mean anything, and in practice a few hundred so the estimate is precise and the rare codes appear at all. If an important code occurs in two per cent of your insights, a random sample of three hundred gives you about six of them, which is not enough to judge — so over-sample the rare codes deliberately and report them separately from the corpus figure.
Can a model replace the second human coder?
Not as the published method is defined: the classical statistic was built for two human raters, and McHugh notes that Cohen specifically discussed two raters. What a model can do is a defensible first pass that a single person reviews, which is a different arrangement from dual human coding and should be described as such in a method section. If you publish from the corpus, say plainly which passages were model-coded and how the coding was verified.
What data can you send to a model?
That is a decision for your privacy and compliance colleagues, and it should be made before a pilot rather than during one. The design that removes most of the question is to de-identify at intake — keep the clinician's name, practice and contact details in the source system and carry only a segment on the insight — and to settle in the contract what the provider retains and for how long. Patient detail that arrives inside a free-text note is the case to plan for explicitly, because it arrives whether or not anyone intended it to.
Does the EU AI Act apply to a medical insights system?
It depends on the deployment, and nobody should tell you otherwise in a sales meeting. What is worth doing regardless is meeting the oversight standard the Act sets for its high-risk class: a system designed so that people can effectively oversee it while it is in use. In practice that means every proposed code, merge and theme is visible, attributable and reversible by a named person — which is a good specification for this work whatever the legal classification turns out to be.