WorkflowClinical Development

Buying guide · what the evidence says

Plain language summary generator: can AI write trial results?

The measured evidence points two ways at once. On reading level, generated summaries beat the published ones decisively. On factuality, a benchmark of 345 generated summaries found that the models that simplified most also added the most content the source never said. This guide reads both sides and sets out the controls that make a generated draft usable.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

Ask whether a plain language summary generator can write trial results and you get two honest answers that point in opposite directions, which is why the question stays open. Generated summaries are dramatically easier to read than the ones trials publish today — one study of twenty urologic oncology trials moved a mean Flesch-Kincaid grade of 18.2 to 7.0. Generated summaries also add things the source never said: in an expert-annotated benchmark of 345 plain language summaries of trial abstracts, 31.3 per cent of GPT-4's outputs contained at least one non-factual addition. Both results are real, and together they describe a tool that makes an excellent first draft and an unacceptable last one. Clinical Trial Lay Summaries is Botminds' software for this work, and it is built around that conclusion: agents propose, and a named person decides every change.

The studies, medicines and people named from the product below are sample data on fictional studies and fictional people. The research is published and linked where it appears.

Can AI write a plain language summary of clinical trial results?

It can write one. Whether it can write one you are willing to post is a different question, and the answer turns on what happens between the draft and the posting rather than on the quality of the draft.

Start with what the generation step is actually good at. Lay summary writing is, mechanically, a compression and register-shift task: take a long technical document, select the facts ten required sections need, and express them in short sentences using common words. Models are good at that, which is why the measured improvement in reading level below is so large; the open question is what they do to the facts on the way.

The realistic comparison is not between a generated summary and a perfect one. It is between a generated summary that a writer checks and signs, and a summary that is late, or thin, or written at a reading level nobody outside the field can use.

How much does AI improve the reading level of a trial summary?

By a lot, and the measurement is unusually clean because reading grade is cheap to compute and the baseline is terrible. In a study published in Current Oncology in December 2025, researchers took twenty actively recruiting trials in prostate, bladder, kidney and testis cancer, generated a replacement for each trial's public brief summary, and compared the two. The Flesch-Kincaid grade level fell from 18.2 (3.8) to 7.0 (0.5), and the Flesch reading ease score rose from 17.0 (13.1) to 73.3 (3.5), both at p<0.0001. The generated summaries also included every element the authors' content checklist asked for.

Reading grade on trial summaries: what is published, what is generated, and the published ideal

Urology trial summaries, 25-year average
18.7
Twenty urologic oncology trials, as published
18.2
The same twenty, generated replacements
7.0
The EU Expert Group’s published ideal
6.0
Flesch-Kincaid grade level, means. The 18.7 (SD 5.1) is Baskin et al., Urology Practice, 7 July 2025; the 18.2 (SD 3.8) and 7.0 (SD 0.5) are Cano et al., Current Oncology, 10 December 2025, at p<0.0001, where Flesch reading ease also rose from 17.0 to 73.3. Both studies measure the brief summary written at registration, not a summary of results. The sixth-grade ideal is the EU Expert Group's, which asks writers to get as close to it as they can.Source: Baskin et al., Urology Practice 12(6):759–768 (7 July 2025); Cano et al., Current Oncology 32(12):696 (10 December 2025); EU Expert Group, Summaries of Clinical Trial Results for Laypersons

That last caveat matters more than it looks. Both of the readability studies on this page measure the brief summary a sponsor writes when it registers a trial — a paragraph describing what the trial will do — not the summary of results it writes when the trial ends. The registration summary has no numbers to get wrong. A results summary is mostly numbers, and the error modes are different.

The baseline is also not improving on its own. A separate study of urology trial summaries found an average Flesch-Kincaid grade of 18.7 (SD: 5.1), and tracked the trend over a quarter of a century: from 2000 to 2016 “FKGL increased by 0.15 points per year (P < .01)”, and only after a 2017 plain language policy did the direction reverse, with “FKGL decreasing by 0.18 points per year (P < .01)”. At 0.18 points a year, closing the gap from an average of 18.7 to an eighth-grade reading level by exhortation alone would take about sixty years.

How accurate are AI plain language summaries?

Accurate enough to be dangerous, which is the uncomfortable middle. The best measurement comes from FactPICO, a benchmark built by researchers at Texas, MIT, Duisburg-Essen, Georgia Tech and Northeastern, who had experts annotate the factuality of 345 plain language summaries of randomised trial abstracts — 115 abstracts put through three models — scoring the population, intervention, comparator and outcome of each trial, and the inferences drawn from them, on a four-point scale.

The headline result is a warning rather than a verdict. The authors conclude that “plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level”. Two halves of that sentence are each worth a section of their own, and get one below.

“But given the inherent risks to personal health, the factual correctness of such outputs is paramount in this domain.”

Joseph et al., FactPICO, ACL 2024

Do AI summaries invent facts?

Yes, and the benchmark puts a number on it — along with a finding that should change how a buyer reads any demo. The authors note that “Table 3 also shows a concerning amount of non-factual additions within the generated simplifications”, and define the measure precisely: “%N: percentage of summaries in FACT PICO with at least one non-factual addition.”

Read the three models together and a trade-off appears that no single figure captures. Alpaca added almost nothing false — 7.0 per cent of its summaries had a non-factual addition — but its change in Flesch-Kincaid grade was −0.61, meaning it barely simplified, and its overlap with the original abstract was by far the highest. It was safe because it was mostly copying. GPT-4 and Llama-2 simplified properly and added false content in 31.3 and 38.3 per cent of summaries respectively.

Three models on 345 generated summaries: how much each simplified, and how much it added

ModelExpert factuality, 1–4Change in grade levelOverlap with the abstractNon-factual additionsSummaries with at least one
Alpaca3.53−0.610.47987.0%
GPT-43.33+2.870.1465331.3%
Llama-23.00+2.920.1365738.3%
Table 3 of FactPICO (Joseph et al., Proceedings of ACL 2024). Expert factuality is the average over population, intervention, comparator, outcome and evidence inference, higher being better. A positive change in grade level means the summary was simpler than the abstract; overlap is ROUGE-L against the abstract, where a higher figure means more of the original wording was kept.Source: Joseph et al., FactPICO, Proceedings of ACL 2024 (Long Papers), pp. 8437–8464

The practical lesson is that simplification and fidelity trade against each other, and that a model which looks safe on a factuality metric may be safe only because it refused to do the job. A vendor demo showing a beautifully plain summary is showing you the end of the table where the additions are.

The second failure mode: omissions that change the result

Invention is the failure everyone expects. Omission is the one that actually destroys a results summary, and the benchmark's authors are blunt about its effect: “These omissions not only render the simplification incomplete, but they also make understanding the actual result impossible.”

The specific omission that matters in trial results is the comparator. A summary that says a medicine helped, without saying what it was compared against, has not simplified the result — it has deleted it. Twenty-nine out of a hundred people relapsing on a dummy treatment and nineteen on the medicine is a finding; nineteen out of a hundred on its own is a number with no meaning. The same applies to the denominator, the time point, and the qualifier that says not every medical problem counted was caused by the medicine.

Two ways a generated summary goes wrong, and what each does to the reader

A non-factual additionAn omitted comparator
What it looks like

A non-factual addition: A sentence that reads plausibly and is not in the source: a mechanism the trial did not test, a cause the data cannot support, a reassurance nobody wrote.

An omitted comparator: A result given for one arm only, or a time point or denominator quietly dropped while the sentence still parses.

How it is found

A non-factual addition: By comparing the sentence with its source passage. It is visible the moment the two are side by side.

An omitted comparator: By checking the sentence against the structure the result needs, not against a passage. Nothing in the text looks wrong.

What it does to the reader

A non-factual addition: Gives them a belief the trial did not produce. Dangerous when the belief is about safety or cause.

An omitted comparator: Makes the result impossible to interpret, while looking clearer than the correct version — which is why it survives review.

Why automatic checks miss it

A non-factual addition: Only partly missed: a strict source-comparison catches it; a fluency or similarity score does not.

An omitted comparator: Missed almost entirely. Shorter, simpler text scores better on most automatic measures, and an omission makes text shorter and simpler.

Drawn from the two error classes FactPICO annotates — added information that is not factual, and omissions of key trial elements — and from what each means for a summary of results rather than of a trial design.Source: Joseph et al., FactPICO, Proceedings of ACL 2024 (Long Papers), pp. 8437–8464

Should a language model check its own output?

No. Not because it will always miss its own errors, but because a check that shares the fallibility of the thing it checks is not independent, and independence is the only property that makes a check worth running.

There is a sharper reason in this setting. The errors that matter here are differences between a sentence and a passage in a specific document. A checker that has the study report open and compares number to number, group to group, time point to time point is doing a comparison. A model reflecting on text it generated is doing an estimate. The first can be wrong in ways you can inspect; the second cannot.

In this solution the separation is structural: the fact checker matches each factual sentence back to its passage and compares every number, unit, group and time point, and it cannot change text — it proposes a fix for a person to accept, edit or dismiss. The plain-language checker is separate again, scores the grade of each sentence and explains every flag in one sentence. And the reader test, which answers eight key questions a participant would ask, has no access to the study report while it answers; that constraint is what makes it a test rather than a score.

A drafting run, step by step, ending where it should end

A screen for study NXL-118, Talmirex in major depression, 174 days to its deadline, with a status of Agents drafting. A drafting-run panel lists six steps with timings: study report reader, which read the synopsis and 14 results tables into 56 facts with page and table, 38 seconds; Annex V drafter, which wrote 10 of 10 required sections as reader questions using glossary words only, 54 seconds; fact checker, 21 statements matched to the study report with every number matching, 31 seconds; plain-language and tone checker, reading grade 7.8 with 1 wording flag, 9 seconds; reader test, 7 of 8 key questions answered from the summary alone, 22 seconds; and a final amber step, Writer decides, waiting for Lena Ortiz. The ten required sections down the left are all still unticked, and the facts panel reads 'Nothing to check yet — draft the summary first.'
Clinical Trial Lay Summaries on fictional sample data. The run ends at a person, not at a finished summary; the checks that follow the draft are separate from the thing that wrote it.

What the automatic quality metrics miss

Almost everything you care about. The FactPICO authors found that existing automatic factuality metrics “correlate poorly with expert judgments on the instance level” — meaning that on any individual summary, the metric's verdict and the expert's verdict are close to unrelated. A metric may be useful for comparing two systems over hundreds of outputs. It is not useful for deciding whether this summary is safe to post.

Readability scores fail in a parallel way, and the regulatory guidance already says so. Good Lay Summary Practice warns that a short sentence of short words can score well while making no sense at all, because an acceptable score and real readability are not directly related. Put the two findings together and the position is stark: the automatic measure of factuality does not track expert judgement, and the automatic measure of readability does not track comprehension. Neither can carry a posting decision.

What survives is slower and better. For factuality, the comparison of each sentence against its source passage, done exhaustively rather than sampled. For comprehension, a test with answers that can be wrong, scored against the study report. Both produce something a person can inspect; a score does not.

Who is the author of an AI-drafted lay summary?

A person, by name, and the question is not rhetorical — the summary is a regulated disclosure attached to a trial, and somebody has to be answerable for it. Under the Clinical Trials Regulation the results posting must come with a summary laypersons can understand, and the obligation sits with the sponsor. Nothing in the rules contemplates an unattributed document.

The useful discipline is to make the signature say what it asserts rather than that something was approved. The electronic-records rule that most sponsors already work to requires exactly that: a signed record must state what the signature means, and authorship is one of the four meanings the rule gives as examples, beside review, approval and responsibility. In the sample data behind this page the writer signs with the meaning “I am the author and confirm every fact matches the final study report”, and the signing window repeats the three measures behind it before she signs.

A sponsor that wants a defensible answer to “who wrote this?” needs three things on the file: a named author who accepted every change, a record of which flags were accepted, edited or dismissed and why, and the source passage behind each factual sentence. A draft's origin then stops being the interesting question, because the answer to every version of it is the same person.

The signature, and what it asserts about the summary beneath it

An Approve and sign dialog over study NXL-125. A green band lists the checks behind the signature: 24 of 24 statements match the study report, reading grade 7.9, reader test 8 of 8. A dropdown headed 'Meaning of this signature' reads 'I am the author and confirm every fact matches the final study report'. The signer is Lena Ortiz, with a password field below, and a note that the patient panel reads it next, then two further signatures, and that translations into 7 languages start from this signed English.
Clinical Trial Lay Summaries on fictional sample data. What is being signed, and what the signature means, are stated at the moment of signing rather than reconstructed afterwards.

The controls that make a generated draft usable

Five, and they are cheap to state and expensive to retrofit. Together they turn a generated draft from something nobody can vouch for into something a named person can sign.

Five controls between a generated draft and a posted summary

  1. A cited passage per factual sentence

    Written at generation time, not reconstructed afterwards, and surviving the writer's edits. Without it, every other check is a re-reading rather than a comparison.

  2. An exhaustive number comparison

    Every number, unit, group and time point against the report, not a sample. Rounding beyond a tolerance you set is a difference, not a style choice.

  3. A separate checker that cannot edit

    The component that finds a problem must not be the one that fixes it. Otherwise a defect and its correction happen in the same unobserved step.

  4. A comprehension test the summary can fail

    Answer the questions a participant would ask, using only the summary, then compare with the report. A score that cannot be wrong proves nothing.

  5. A named signature with its meaning

    Authorship, medical accuracy and the non-promotional check are three different assertions by three different people. One generic approval collapses them.

The controls this guide argues for, and the ones this solution implements: posting is blocked while any fact is unchecked or any flag is open, and the reader test must answer at least 7 of 8 key questions before the patient read-through.

Three things a generator must not be allowed to do on its own

Round a number

64.7 per cent becoming “about 7 out of 10” is a 5.3-point overstatement dressed as friendliness. Rounding is a decision with a tolerance, and the tolerance belongs to the sponsor.

Name a cause

“Side effects caused by the medicine” where the report counted events that merely happened during treatment. The causal claim is almost never in the table, and it is the claim a reader remembers.

Draw a conclusion the trial did not test

“Safe”, “effective”, “works well for people like you”. A trial measures what happened to the people in it; generalising beyond them is a judgement, and a promotional one at that.

Each of these is a small, reasonable-looking liberty that changes what the summary claims. All three are the kind of change a person should have to accept explicitly.

What a human reviewer still has to do

Decide. That is the short answer and it is not a platitude — the specific decisions are enumerable, and enumerating them is how a team works out how much time to budget.

The reviewer accepts, edits or dismisses each flag with a reason, which means reading the proposed sentence against the passage rather than against the draft. They judge the rewrites that are shorter and may have lost something: a plainer sentence that drops a qualifier is the most common quiet failure. They decide what goes in the comments section, which is where one trial's result most easily becomes a claim about a medicine. They carry the limitations, because a model has no stake in whether a result generalises. And they sign, with a meaning.

The research cited here agrees on this point. The Current Oncology authors write that “human oversight is mandatory at this stage not only for detecting possible hallucinations and discrepancies between original sources and generated texts but also for identifying information that is clinically meaningful” — note the second half, which is about judgement rather than error-catching. Good Lay Summary Practice frames the same expectation for reviewers generally: they are to be told that a lay summary must not promote and must avoid biased language, and the content itself is to be factual, objective and not designed to favour the medicine.

Three flags waiting on a person, each with its source passage and a proposed fix

A side-by-side screen for study VLT-302, Veltrimab in plaque psoriasis, 99 days to its deadline, with 3 to decide before signing. The ribbon reads 28 of 29 facts match the report, 8.9 reading grade against target 8, 5 of 8 reader-test questions answered. The adverse-reactions row shows the study report's Table 12.2 on page 63 — any event, veltrimab 249 of 428 (58.2%), placebo 110 of 214 (51.4%) — beside a struck-through English sentence using the phrase treatment-emergent adverse events, marked grade 11 and matching Table 12.2. A proposed replacement below reads 'In the first 16 weeks, 58 out of 100 people on Veltrimab and 51 out of 100 people on placebo had at least one medical problem. Not all of these were caused by the study medicine', with the explanation that the original is study-report language and the glossary term is 'medical problem', and Accept, Edit and Dismiss buttons showing grade 11 becoming 7. The row below shows the overall-results sentence marked as differing from Table 11.4.
Clinical Trial Lay Summaries on fictional sample data. Each flag arrives with the passage it rests on and a proposed sentence; none of them changes the summary until the writer says so.

How do you measure whether an AI lay summary is good?

With four measures, none of which is a quality score, and all of which can come out badly. A single number that always looks healthy is the sign that nothing is being measured.

Four measures that can fail, on one sample summary

24 of 24

statements matching the study report, each with the page or table it came from

7.9grade

across the summary, against a house maximum of 8 and a published ideal of sixth grade

8 of 8

key questions answered correctly from the summary alone, compared afterwards with the report

0open

flags: every one accepted, edited or dismissed with a reason by a named writer

Sample data on the fictional NXL-125 study in Clinical Trial Lay Summaries at the point of signing, not a customer's measured result. Each of the four can be short, and posting is blocked while any fact is unchecked or any flag is open.

Add a fifth that no system can produce for you: what real participants took away at the read-through. A comprehension test run before the panel is a way of arriving at that meeting with the obvious failures already fixed, so the hour with real readers is spent on the problems only real readers can find.

The honest summary of the evidence is narrower than either side of the argument usually allows. A generator is good at the register shift and measurably better at it than what trials publish today. It is unreliable at the part a results summary consists of, which is numbers in context, and it fails in two directions at once — inventing content and dropping comparators. So use it for the draft, check it against the report sentence by sentence, test what a reader takes away, and have a person sign. The worked example in four sentences taken apart shows what that checking actually catches, and what to look for in a tool turns it into requirements you can buy against.

To run a draft and its checks on a study report of your own, start from Clinical Trial Lay Summaries.

Questions people ask

Does a regulator allow an AI-written lay summary?

No published rule prohibits drafting with software, and none authorises it either — the rules address the sponsor's duty and the document's content, not the tools used to produce it. The obligations that bite are the same whatever wrote the draft: the content required of a layperson summary, factual and non-promotional language, and a sponsor answerable for what is posted. Treat the drafting method as an internal procedural question and the output as the regulated thing.

How do you stop a model from inventing a number?

You cannot stop it; you catch it. Extract every number, unit, group and time point from the draft and compare each with the study report passage it claims to come from, exhaustively rather than by sampling, and treat a number without a source as a failure rather than a warning. Posting should be blocked while any number is unmatched, because an unmatched number is indistinguishable from an invented one.

Are AI summaries more accurate than human ones?

There is no study comparing the two on summaries of trial results, so the honest answer is that nobody knows. What is measured is that generated summaries of trial abstracts contain non-factual additions at meaningful rates, and that published summaries are written far above the reading level their audience has. Those are different failures, and a process that checks the draft against the source addresses both.

What is the biggest risk of a generated lay summary?

The omitted comparator, because it is the error that hides. An invented sentence is visible the moment you look at the source passage. A result reported for one arm, or without its time point or denominator, reads more clearly than the correct version and scores better on every automatic measure of simplicity. The benchmark's authors found such omissions make understanding the actual result impossible.

Should we test a generator on our own study reports before buying?

Yes, and on a messy one rather than a clean one. Plant errors before anyone sees the draft — a rounding past your tolerance, a swapped arm, a shifted time point, a wrong denominator and a sentence with no source — and score the candidate on which it returns and what evidence it leaves behind. Judge the pilot on its artefacts rather than on how the prose reads.

Does using a generator change who is responsible for the summary?

Not at all. The duty to post a summary understandable to laypersons sits with the sponsor, and the author is whoever signs. What a generator changes is how much of the writer's day is spent producing a first draft rather than judging one — which is why the controls that preserve the writer's decisions, and record them, are the part worth paying for.

Clinical Trial Lay Summaries.
See it on your own files.

An agentic solution from Botminds. The study report, the plain English and every translation side by side, one row per sentence, each with its reading grade and a tick that its fact matches the report. Book a demo, or explore how it works.