SearchResearch & Discovery

Evaluate · before you buy

AI literature review tools: what to test before you trust one

Every tool in this market will hand you a confident answer with references attached. Measured studies say a large share of those references are wrong or invented — 39.6%, 28.6% and 91.4% for three models in one controlled test. So buy on verification rather than on fluency. This guide sets out five tests you can run in a week, and where Scientific Literature Search sits in them.

Updated · 14 min read

See one case, screen by screen →
Watch with transcript →

AI literature review tools take a research question in plain words and return a written answer with references, instead of a list of hits to read yourself. They are worth buying and they are easy to buy badly, because the thing they are best at — fluent, confident prose with a tidy reference list — is exactly the thing that hides the failure: in a controlled test published in the Journal of Medical Internet Research, the share of invented references stood at 39.6% for one model, 28.6% for another and 91.4% for a third. The defence is not a better model but verification you can see, which is what Botminds’ Scientific Literature Search is built around — every statement is a numbered claim made of evidence rows, every row opens the source at the passage it came from, and a row the citation checker cannot support against that passage is removed before the answer is shown.

The screens on this page come from the solution running on its sample library; the programmes, studies, people, journals and sources named in them are fictional. The measured figures are from peer-reviewed studies and a regulator’s own guidance, linked where they appear.

What the studies measured

39.6%

invented references from one model, and 28.6% from a second, in a 2024 test on systematic-review questions

91.4%

invented references from the third model in the same test

7.13%

of 2,736 references fabricated outright across 350 queries, in a 2026 test of a newer model

49.34%

of citations in that test were bibliographically accurate in every field

Chelli and colleagues, Journal of Medical Internet Research 26:e53164, 22 May 2024, where a paper counted as invented if any two of title, first author or year of publication were wrong; and Bernstein and colleagues, Cureus 18(7):e113654, 30 July 2026. The two tests used different models, questions and definitions and are not directly comparable.Source: JMIR 26:e53164; Cureus 18(7):e113654

What do AI literature review tools actually do?

Under the marketing there are only three mechanisms, and which one a tool uses determines how it can fail.

The first is generation from the model’s own training: you ask, it writes, and the references come out of the same process that produced the prose. This is the arrangement that invents references, because nothing in it ever consults a bibliographic index. The second is retrieval first: the tool searches a real corpus, takes the passages it found, and writes only from those. The third is a hybrid in which retrieval supplies some statements and the model supplies the rest, which is the hardest to evaluate, because a reader cannot tell the two kinds of sentence apart.

The question that separates them in a demonstration is not “where do you get your papers from” — every vendor has an answer — but “show me a statement, and open the sentence it came from”. A retrieval-first tool can do that for every statement. A hybrid can do it for some. A generator cannot do it at all, and will instead offer you a reference list.

How accurate are AI literature reviews?

Accurate enough to be dangerous and not accurate enough to be trusted unverified, on the two published measurements available. The 2024 JMIR study asked three assistants questions drawn from systematic reviews and checked each returned paper against reality; a paper counted as invented if “any 2 of the following information were wrong: title, first author, or year of publication”. Invention rates were 39.6%, 28.6% and 91.4%.

The 2026 Cureus study took a different cut: 350 queries produced 2,736 references, of which 7.13% were fabricated outright — but only 49.34% of citations were bibliographically accurate in every field. The two studies used different models, questions and definitions, so they do not measure a trend; read together, they say that outright invention can be low while roughly half of the citations still have something wrong with them. That is the shape of the risk: fewer ghosts than the early tests found, still many misquotations, and a reader who cannot tell which is which without opening the source.

Where citation errors fall when the reference is real

Identifier (PMID)
37.28%
Title
30.70%
Pages
27.49%
Bernstein and colleagues, Cureus 18(7):e113654, 30 July 2026: the fields most often inaccurate across 2,736 references generated from 350 queries. Errors in different fields overlap within a citation, so the shares do not sum to a count of faulty references.Source: Cureus 18(7):e113654

Read that chart as a warning about what verification has to cover. A tool that checks only whether an identifier resolves will pass a citation whose pages are wrong; a tool that checks the title will pass a citation attached to a sentence the paper never said. The only check that catches all of it is the one that reads the cited passage back against the sentence it is supposed to support.

What an invented reference looks like, and why it is convincing

It looks like a reference. The authors are real people who work in the field. The journal is the one that publishes this kind of work. The year is plausible, the volume and page numbers are the right shape, and the title describes a study that genuinely ought to exist. Nothing about it reads as an error, because it was not produced by an error: it was produced by a system that models what a citation in this context looks like, and did that job well.

That is why the usual human defences fail. A reviewer skimming a reference list is checking the shape of the citations, and the shape is correct. A scientist who recognises three of the five authors concludes that the fifth is simply someone they have not read. The failure is only visible at the level of the individual sentence: does this source, opened at this page, say this?

A second, quieter failure matters as much. A reference may be entirely real and still not support the sentence it is attached to — the study exists, but it used a different model system, a different population, or reported the number as a median where the answer quotes it as a mean. Checking that a citation resolves does not catch any of that, which is the same problem as checking a paper before you cite it, one level further in.

What checking one statement should cost: one click

The source viewer with a cited source open at page 18 and the cited sentences highlighted, beside a panel giving why that passage was returned, what the source studied, its model system and n, who wrote it and who may read it. A line notes that the citation checker read the evidence row against this passage and found it supported.
The test any tool in this market should be made to pass in a trial: take a statement, open it, and read the sentence it rests on. On the fictional sample library this is one click from the row.

The five tests to run before you buy

Five tests, all runnable in an afternoon with the vendor in the room, which together tell you more than any benchmark a vendor will show you.

The five tests, and what a good answer looks like

  1. 1. Open every statement at its source

    Take one answer and click through every sentence. A good tool lands you on the page with the sentences highlighted. A bad one shows you a reference list and leaves the search to you. Count the statements you could not open.

  2. 2. Ask something the library cannot answer

    Invent a plausible question about a target nobody has studied. The right answer is “nothing here answers that”. A tool that produces a confident paragraph has told you how it behaves on every hard question.

  3. 3. Ask the same question again tomorrow

    Reproducibility is a property of search systems, not a nicety: methodologists call a search reproducible only when the same query returns the same results. Compare the two answers claim by claim and ask what changed and why.

  4. 4. Check one figure you know by heart

    Pick a number you can recite — a dose, an n, a rate — and see whether the tool quotes it as written or rounds, recalculates or converts it. Silent arithmetic on someone else’s figure is a defect, not a feature.

  5. 5. Ask who may switch verification off

    If the rule that every statement must carry a passage can be disabled, find out by whom, whether the change is visible afterwards, and whether answers written while it was off are marked.

Run all five on your own questions and your own corpus, not on the vendor’s demonstration set. Test 3 paraphrases Gusenbauer and Haddaway’s definition of a reproducible search, Research Synthesis Methods 11(2):181–217.Source: Research Synthesis Methods 11(2):181–217

Verification: can every sentence open at its passage?

This is the whole of test one, and it is worth stating as a design rule rather than a feature request: an answer should be built out of sourced units, not decorated with sources afterwards. The difference shows up in what the tool can do when a source is missing. If sources are attached afterwards, a sentence with no source still exists and simply loses its superscript. If the answer is built from sourced units, a unit with no passage behind it has nothing to be, and disappears.

That is how Scientific Literature Search is arranged. An answer is a short paragraph and numbered claims; each claim is a block of evidence rows, and each row carries its source, year, model system, n, effect and stance together with the passage it came from. A row without a passage is removed before the answer reaches the screen, and the citation checker then reads every surviving row back against its passage — including the n and the model system — removing or flagging anything the source does not actually say, with what it removed written to the audit trail.

Two answers to the same question

A fluent summary with a reference listNumbered claims whose rows open their passages
What you get

A fluent summary with a reference list: Three well-written paragraphs and twelve references at the foot.

Numbered claims whose rows open their passages: A short answer, then numbered claims, each a block of rows with source, year, model system, n, effect and stance.

Checking one sentence

A fluent summary with a reference list: Find the reference, find the paper, find the section, decide whether it says this.

Numbered claims whose rows open their passages: Open the row: the source opens at its page with the cited sentences highlighted.

When a source is missing

A fluent summary with a reference list: The sentence stays; the superscript is absent.

Numbered claims whose rows open their passages: The row is removed before the answer is shown, and the removal is recorded.

When the library has no answer

A fluent summary with a reference list: A paragraph is produced anyway, in the same voice as the rest.

Numbered claims whose rows open their passages: It says nothing here answers that, instead of writing a claim.

When your own work disagrees

A fluent summary with a reference list: Not visible: internal reports were never in the corpus.

Numbered claims whose rows open their passages: The row turns amber with the reason the two may differ, and goes to the owner of your source.

What a reviewer can audit

A fluent summary with a reference list: The reference list.

Numbered claims whose rows open their passages: Every row, every passage, every removal, and who asked what and when.

The left column describes the common arrangement in this market; the right column describes Scientific Literature Search. Both can be checked in a trial by running the five tests above.

Inside Scientific Literature Search: an answer as numbered claims, each row opening its passage

The answer screen: a short paragraph, then five numbered claims. Each claim is a block of evidence rows showing source, year, model system, n, effect and stance, with an agreement meter counting sources that support, contradict or limit the claim. A side panel shows the passage behind the selected row, and a line reads seventeen rows supported, none removed, twelve sources, four of them internal.
Scientific Literature Search on its fictional sample library. The sample run showed 17 rows supported and 0 removed on that question; it is what one run showed, not a measured accuracy rate.

What the tool does when it has no answer

The most informative test in the list is the second, and almost nobody runs it. Ask about a target nobody has studied, a combination nobody has tried, a population nobody has reported. The corpus genuinely has no answer, so whatever comes back is the tool’s behaviour in the absence of evidence — and that behaviour governs every hard question you will ever ask it, because hard questions are the ones where the corpus is thin.

There are three possible answers. “Nothing in the library answers that” is the right one. “Here is the closest related work, which does not answer your question” is acceptable and often useful, provided the caveat is in the answer rather than in a tooltip. A confident paragraph is disqualifying, and no amount of accuracy on easy questions compensates, because you will not be able to tell the two kinds of answer apart in production.

In the solution this is a stated limit of the research agent rather than a tuning parameter: if the library does not answer the question, it says so instead of writing a claim.

Regulated work: credibility for a stated use

If the output will be read outside the company — in a submission, a report to a regulator, a publication — the question stops being “is this tool good” and becomes “is this tool credible for this use”. The United States Food and Drug Administration’s draft guidance on artificial intelligence in regulatory decision-making for drug and biological products frames it exactly that way.

This guidance provides a risk-based credibility assessment framework that may be used for establishing and evaluating the credibility of an AI model for a particular context of use (COU).

Food and Drug Administration, draft guidance, January 2025

Two words in that sentence do the work. Risk-based: the evidence you need scales with what the output decides, so a tool used to triage reading has a lower bar than one whose output supports a submission. Context of use: credibility is established for a stated use, not in general, so “we validated the tool” is not an answer — the answer names the question type, the corpus and the decision. The guidance is a draft with non-binding recommendations, not a rule, and it applies to models supporting regulatory decisions rather than to literature search as such. But the framing is the right one to borrow for a purchase: write down the use first, then ask what evidence would make the tool credible for it.

What that implies for procurement is unglamorous. Specify which questions the tool will answer, whose eyes the answers reach, and what a wrong answer would cost in each case. Then ask for evidence proportionate to the worst of those, and for the ability to show, after the fact, which sources an answer rested on. A tool that cannot reconstruct an answer it gave three months ago cannot support a stated use, however good it is today.

Can AI write a systematic review?

Not on its own, on the evidence available, and the authors of the 2024 test say so themselves: “Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews.”

The deeper obstacle is not accuracy but reporting. A systematic review has to publish the sources consulted, the strategies as they were run, the counts at each step and the reasons for exclusion. An assistant that produces an answer without exposing the search behind it cannot supply any of that, however good the answer is — which is a different failure from getting the science wrong, and the one that stops the review being publishable. That machinery is what systematic review software and what the standard makes you publish is about.

Where these tools do earn their place in a formal review is upstream and downstream of the screening: finding the work that exists, pulling n and model system into a table, and re-checking kept claims when something new arrives. Said plainly, Scientific Literature Search is not a screening platform and does not produce a flow count; a review team will still need a screening tool beside it.

Questions for a literature-review tool vendor, answered in writing

Demonstrations are designed; contracts are not. Ask the questions below in the room, and then ask for the answers in the agreement.

Nine questions, and the answer to listen for

AskA good answerWhat a weak answer sounds like
What share of statements open at a passage?All of them, and here is one we did not pick.“Everything is sourced” — with a reference list as the evidence.
What happens to a statement whose source cannot be verified?It is removed before the answer is shown, and the removal is recorded.“It is flagged for review.”
Do you quote figures or normalise them?Quoted exactly as written; no rounding, no unit conversion, no recalculation.“We standardise units for comparability.”
What does the tool do with no answer in the corpus?It says so.“It gives the best available answer.”
Is the same question answered the same way tomorrow?Yes, and a changed answer says what changed and why.“The model improves continuously.”
Can it search our own reports?Yes, with access decided when the source is indexed.“You can upload documents.”
Who may switch verification off?One named administrator, as a versioned change on the audit trail.“It is configurable.”
What does it do when our data disagrees with a paper?Marks the claim, names the owner of your source, and never decides.“It weighs the evidence.”
Can you reconstruct an answer given three months ago?Yes, with its sources and the rules in force at the time.“Answers are in your history.”
Our checklist for a buyer, written from the failure modes the published tests describe. The right-hand column is paraphrase, not a quotation of any vendor.

The rules a Scientific Literature Search administrator owns, and the ones that are locked

The settings screen showing answer and agreement rules: every row must carry a passage, on; oldest source to use, published 2021 or later; answer length, standard; check every claim against our own work, on; sources needed to call a claim agreed, two; patent text treated as a claim, not a result; who receives a new disagreement, the owner of our source; call out small numbers below ten. Two rules are marked as changeable only by the library administrator, as a new version on the audit trail.
Scientific Literature Search on its fictional sample library. Two rules — every row must carry a passage, and every claim checked against our own work — can be switched off only by the library administrator, as a new version written to the audit trail.

A last point about price. These tools are usually sold per seat, and the per-seat comparison hides the real variable, which is how much of a scientist’s week the tool actually removes. The four kinds of product sold under the neighbouring phrase, and what each one leaves untouched, are mapped in four kinds of literature review software. To run the five tests against a working library, start from Scientific Literature Search.

Questions people ask

Do AI literature review tools make up references?

Some do, measurably. In a 2024 controlled test the share of invented references was 39.6% for one model, 28.6% for another and 91.4% for a third, counting a paper as invented when any two of title, first author or year were wrong. A 2026 test of a newer model, on 350 orthopaedic guideline questions, found 7.13% of 2,736 references fabricated outright, but only 49.34% accurate in every field. The two tests are not directly comparable, but both say the same thing to a buyer: a real-looking citation still has to be opened.

What is an AI hallucination in research?

An output that is fluent, plausible and not true — most often a reference to a paper that does not exist, or a real paper cited for a statement it does not make. It is not a malfunction: the system is producing what a correct answer looks like, and a non-existent citation looks exactly like an existing one. The only reliable detection is opening the cited source at the cited page.

How do you verify a citation from an AI tool?

Resolve the identifier, then read the passage. Resolving alone is not enough: a citation can point at a real paper and still be attached to a sentence the paper never supports, or report a median as a mean. Check the n and the model system too — those are the fields most likely to differ between what a paper measured and what a summary claims it measured.

Are AI literature review tools allowed in regulated work?

There is no blanket permission or ban. The Food and Drug Administration’s January 2025 draft guidance sets out a risk-based credibility assessment framework for establishing the credibility of an AI model for a particular context of use; it contains non-binding recommendations and addresses models supporting regulatory decisions about drug and biological products. The practical reading for a buyer is to define the use first, then gather evidence proportionate to what a wrong answer would cost.

What should you ask a vendor of an AI literature tool?

What share of statements open at a passage; what happens to a statement whose source cannot be verified; whether figures are quoted or normalised; what the tool does when the corpus has no answer; whether the same question gets the same answer tomorrow; whether it can search your own reports and who may read them; who can switch verification off; and whether an answer given three months ago can be reconstructed with its sources.

Can AI write a systematic review?

Not as the primary or exclusive tool — the authors of the 2024 JMIR test, which used questions from 11 systematic reviews, recommend against it. Beyond accuracy, a formal review must publish the sources consulted, the strategies as run and the counts at each step, and an assistant that hides its search cannot supply those. The useful role is upstream and downstream: finding work, building evidence tables, and re-checking kept claims when something new is published.

Scientific Literature Search.
See it on your own files.

An agentic solution from Botminds. Ask in plain words; get numbered claims, each backed by evidence rows and the exact passage, checked against your own reports. Book a demo, or explore how it works.