Evaluate · formal reviews

Systematic review software and what the standard makes you publish

Judge systematic review software by what it lets you publish, not by how fast it screens. The reporting standard asks for every source consulted, every strategy copied exactly as it was run, and a count at each step — and a tool that cannot produce those is a screening tool wearing a review’s name. This guide maps the obligations onto features, and says where Scientific Literature Search helps and where it does not.

Updated · 14 min read

Systematic review software is the set of tools that manage a formal review: importing records from several databases, removing duplicates, splitting titles and abstracts between two screeners, recording include and exclude decisions with reasons, extracting data items, and producing the counts a published review has to report. Choosing one is harder than it looks because the demonstrations all show the screening — the visible, satisfying part — while the thing that decides whether your review is publishable is the search record underneath it, which nobody demonstrates. Botminds’ Scientific Literature Search is not a formal review platform and does not replace one; it sits beside it, answering working questions from published papers, patents and your own research reports and signed notebooks with every statement opening at its passage, and this guide is as much about where it stops as where it helps.

The screens on this page come from the solution running on its sample library; the programmes, studies, people, journals and sources named in them are fictional. The requirements quoted are from the published reporting standards, linked where they appear.

What a formal review costs, and how often the search survives it

67.3wks

mean time from registration to publication, across 195 registered and completed reviews (IQR 42)

64%

of reviews with a librarian among the authors still include a reproducible search

16items

in the reporting checklist for the search alone, over and above the main statement

Borah, Brown, Capers and Kaiser, BMJ Open 7(2):e012545, 27 February 2017, using the PROSPERO registry; and Rethlefsen and colleagues, PRISMA-S, Systematic Reviews 10:39, 26 January 2021.Source: BMJ Open 7(2):e012545; Systematic Reviews 10:39

What does systematic review software do?

Six things, in order. It imports records from each database you searched, keeping them separable by source. It deduplicates. It presents titles and abstracts for screening, usually to two people independently, and records each decision. It manages full-text retrieval and a second round of screening with a reason for every exclusion. It holds the extracted data items. And it produces the numbers that go into the flow diagram: records identified, duplicates removed, screened, assessed in full text, excluded with reasons, included.

What it does not do is search. The search is built and run elsewhere, by a person, usually an information specialist, and handed to the tool as exported records. That division explains the market’s blind spot: the software is excellent at the part that happens after the search, and the part a reviewer will be asked about is the search.

What the reporting standard makes you publish

Two documents govern this, and between them they decide what your tool has to be able to produce. The main statement asks authors to “Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies” and to specify the date each was last searched. It then asks for the strategies themselves: “Present the full search strategies for all databases, registers and websites, including any filters and limits used.”

The word that changed in the 2020 revision is “all”. The statement’s own summary of what is new describes it as a “Modification of the ‘Search’ item to recommend authors present full search strategies for all databases, registers and websites searched, not just at least one database”. Publishing the MEDLINE strategy and summarising the rest in a sentence was acceptable practice; it no longer is.

What the standard asks of the search, and what it means for a tool

  1. Every source named, with the date it was last searched

    Databases, registers, websites, organisations and reference lists — each named individually, with its platform and the date. A tool that merges imports from several databases into one undifferentiated pile cannot produce this afterwards.

  2. The full strategy for every source, not just one

    All databases, registers and websites, with the filters and limits used — the change the 2020 revision made explicit.

  3. The strategies as they were actually run

    Copied and pasted exactly as run, not retyped or tidied; line numbers, operators and field tags intact.

  4. The total records from each source

    A count per database and per other information source, before deduplication, which is what makes the flow diagram reconstructable.

  5. How duplicates were removed

    The processes and any software used to deduplicate records from multiple database searches and other information sources.

  6. Any peer review of the strategy

    Whether the search strategy was peer reviewed, and how. A one-line item that is easy to skip.

Drawn from the PRISMA 2020 statement’s search items and the PRISMA-S reporting extension, which sets out 16 items for the search alone. The right-hand sentences are our reading of what each obligation implies about a tool.Source: PRISMA 2020 statement; PRISMA-S, Systematic Reviews 10:39

How long does a systematic review take?

A mean of 67.3 weeks from registration to publication, with an interquartile range of 42 weeks, measured across 195 registered and completed reviews in the PROSPERO registry. The mean team was five authors, and funded reviews took longer and involved more people — 42 weeks against 26 to complete and publish, and 6.8 people against 4.8 — which is the opposite of what a budget holder expects and probably reflects the scope that funding buys.

Those numbers matter for a purchase in one specific way. Over sixteen months, the people change. Someone leaves, someone joins, the protocol is amended, and the person defending the search at peer review is often not the person who built it. Everything the standard asks you to publish is also what makes the review survivable by the second team, which is the practical argument for recording it as you go rather than reconstructing it at write-up.

A systematic review from question to publication

The review, mean 67.3 weeks to publication195 registered reviews, IQR 42 weeks
Question and protocolprotocol
Search built and runsearch
Deduplicate and screenscreening
Full text, with reasonsfull text
Extraction and biasextraction
Synthesis and writingsynthesis
  • Question and protocolprotocol
  • Search built and runsearch
  • Deduplicate and screenscreening
  • Full text, with reasonsfull text
  • Extraction and biasextraction
  • Synthesis and writingsynthesis
Where the software isScreening onwards
A person, outside the toolprotocol and search
Inside the tooldeduplication, screening, extraction, counts
A person againsynthesis and writing
  • A person, outside the toolprotocol and search
  • Inside the tooldeduplication, screening, extraction, counts
  • A person againsynthesis and writing
The total is the measured mean of 67.3 weeks from Borah and colleagues; the split between phases is illustrative, drawn to show where the software sits rather than measured durations. The second lane is the point: the tool owns the middle, and the obligations that decide publishability sit at both ends.

Screening: where software helps and where it must not decide

Screening is where these tools earn their price. Splitting ten thousand titles between two people, recording both decisions, surfacing the disagreements and keeping an exclusion reason against every excluded record is tedious, error-prone work that software does well and spreadsheets do badly.

Relevance ranking is a genuine improvement too, when it is used honestly: ordering the queue so that likely includes arrive first lets a team stop screening sooner with a stated stopping rule. It becomes a problem the moment the ranking silently excludes. The standard asks you to report how many reviewers screened each record, whether they worked independently and, if applicable, details of automation tools used in the process — so a record a tool excluded without anyone seeing it has to be declared as such, and a flow diagram that implies a human looked at it is wrong. If a tool offers automated exclusion, the only safe configurations are off, or on with a human check of a sample large enough to bound the error, reported as part of the method.

Deduplication, and the records that only look like duplicates

Deduplication has its own reporting item for a reason: it is the one step that can silently change your results. The extension asks authors to describe the processes and any software used to deduplicate records from multiple database searches and other information sources, which means a tool that merges records invisibly on import has taken a methodological decision on your behalf and left you unable to report it.

Two failure modes are worth testing in a trial. Over-merging: a conference abstract and the later full paper share a title and most authors but are different sources with different data, and collapsing them loses the abstract — which matters, because only 46.4% of abstracts are estimated to reach full publication by ten years, 68.7% for randomised and controlled trials and 44.9% for everything else. Under-merging is cheaper but not free: the same record arriving from three databases with three different author formats inflates the “records identified” count and makes the flow diagram wrong in the other direction. Ask any vendor how their matching works, whether a merge is reversible, and whether the merged set can be exported with each original record still identified by the source it came from.

The search record, copied exactly as it was run

This is an obligation that is often broken, and one a peer reviewer can check.

Include the search strategies for each database and information source, copied and pasted exactly as run.

PRISMA-S checklist item 8, Rethlefsen and colleagues, Systematic Reviews 10:39

“Exactly as run” rules out the common shortcut of retyping a strategy into an appendix in a cleaner form, because a retyped strategy is a different strategy. And the state of practice is poor even among the best-resourced reviews: “reviews where librarians are authors still include reproducible searches only 64% of the time”. If the reviews with a professional searcher on the author list fail this one in three times, a team without one should assume the problem is theirs too.

The practical fix is clerical. Save the strategy for each database at the moment it is run, with its platform, its line numbers and its result count, alongside the date. Save the peer review of the strategy, because that has an item too. Keep the export from each source separately so the per-source counts survive. None of this requires software, and none of it can be reconstructed six months later.

Grey sources and registries inside a formal review

The standard’s list of sources runs well past the databases: registers, websites, organisations and reference lists are all named, and the extension gives trial registries and purposefully browsed sources their own items. In practice this is the part of the search a tool helps with least, because registry queries and proceedings browsing do not produce exportable record sets in the same way a database does.

The workable approach is to treat each grey route as a named source with its own written procedure, its own date and its own count — then it reports like any other source, even though it was done by hand. What each grey route is good for, and how to cite something with no journal, volume or page, is the subject of the results that never became papers.

Do you need two reviewers for screening?

The standard does not mandate two; it mandates that you say what you did. The selection-process item asks for the methods used to decide whether a study met the inclusion criteria, including how many reviewers screened each record and each report retrieved, whether they worked independently, and any automation tools used. Much methodological guidance recommends two independent screeners, because a single screener can miss eligible studies, and a review that used one will be read accordingly.

Where a sensible compromise exists, it is also reportable: two independent screeners at title and abstract, or one screener with a second checking a defined sample, or two at full text and one earlier. Each is defensible if stated. What is not defensible is silence, or a flow diagram whose numbers imply a process the methods section does not describe.

What a tool must be able to export, and why

What you must be able to produceWhy it is neededHow to test it in a trial
The strategy for each source, as runRequired item: full strategies for all databases, registers and websites, copied exactly as run.Import from two databases, then export the strategies. If they come back retyped or merged, the tool cannot meet the item.
A count per source, before deduplicationThe flow diagram starts with records identified per source.Check that the per-database counts survive import and deduplication.
The deduplication method and its effectRequired item: the processes and software used to deduplicate.Deduplicate a set with a known abstract-plus-paper pair in it, then ask whether the merge can be reversed.
Every exclusion reason at full textRequired for the flow diagram and for anyone replicating the selection.Exclude twenty records with reasons and export the list; check the reasons are free text, not only a fixed list.
Who screened each record, and whether independentlyRequired by the selection-process item.Run a double-screened batch and export the per-reviewer decisions and the disagreements.
The data items as extracted, with their sourcesNeeded for the evidence table and for any re-analysis.Export the extraction and check each value carries which record and which reviewer it came from.
Our mapping of the reporting obligations onto things a tool has to be able to hand you. Every row is a test you can run inside a trial account in under an hour.

A buying checklist that maps to the standard

What the standard asks, and what to ask the vendor

The obligationThe question to put to the vendor
Sources

The obligation: Name every database, register, website and organisation searched, with the date each was last searched.

The question to put to the vendor: Show me an export that lists every source with its platform and date, from a project with four sources in it.

Strategies

The obligation: Publish the full strategy for all of them, with filters and limits, copied exactly as run.

The question to put to the vendor: Where is a strategy stored, and does the export preserve line numbers and field tags byte for byte?

Counts

The obligation: Report records identified from each source, and what happened at each step.

The question to put to the vendor: Can I reconstruct the flow numbers six months later if the team has changed?

Deduplication

The obligation: Describe the processes and software used.

The question to put to the vendor: What is your matching rule, is a merge reversible, and does an abstract survive beside its later paper?

Screening

The obligation: Report how many reviewers screened each record and whether independently.

The question to put to the vendor: Can any record reach an exclusion without a person seeing it, in any setting?

Peer review

The obligation: Describe any search peer review process.

The question to put to the vendor: Where does a peer review of the strategy live, and does it export with the project?

Portability

The obligation: Nothing in the standard — but everything above must outlive the tool.

The question to put to the vendor: Export the whole project today. What do I lose if we leave?

Left column paraphrases the PRISMA 2020 search and selection items and the PRISMA-S extension; right column is ours. Ask the last question first — the answer tells you how the others were designed.Source: PRISMA 2020 statement; PRISMA-S, Systematic Reviews 10:39

Where does a question-answering tool fit against this list? Honestly: not on it. Scientific Literature Search does not screen, does not deduplicate a multi-database import and does not produce a flow count, so it cannot satisfy a single obligation above, and a review team still needs a screening tool. What it does is the work either side of the review — finding what exists including your own reports, pulling n, model system and effect into comparable rows, and re-checking kept claims when something new arrives — and the recurring question a review cannot be run for. The four kinds of product in that neighbourhood, and which one you are short of, are mapped in four kinds of literature review software.

Kept answers in Scientific Literature Search, with their claims, rows and sources

The programme notebook: kept answers listed with the question that produced each one, its claims, how many evidence rows and sources each carries, how many of those sources were internal, and whether a disagreement is still attached. Underneath, the questions the team asked this week with who asked them.
Scientific Literature Search on its fictional sample library. Keeping an answer with its claims, rows and passages is not a flow diagram and does not pretend to be one; it is how the next person starts from the rows rather than from a search box.

Two settings on that screen matter to a review team even though the tool is not a review platform. The oldest source to use can be set — published 2021 or later, 2016 or later, or any year — which is a date limit of exactly the kind the standard asks you to report, so write down which one was in force. And how many sources are needed before a claim counts as agreed is a number someone chose; it belongs in your methods if any of this reaches a publication.

What Scientific Literature Search measures when the questions are not a formal review

The dashboard: questions answered in the period, median time to a sourced answer, evidence rows checked against their passages, disagreements found between internal work and the published literature, and where cited passages came from across papers, patents, internal reports and lab notebooks.
Scientific Literature Search on its fictional sample library. None of these are review metrics; they are the measures of a working library, which is the other half of a research group’s literature life.

If your next piece of work is a formal review, buy a screening tool and build the search record from day one. If it is the twenty questions a programme asks between reviews, that is a different purchase — start from Scientific Literature Search and judge it on whether a colleague can check an answer without redoing it.

Questions people ask

What software is used for systematic reviews?

Three kinds together, usually. A reference manager holds the library and formats citations. A screening tool handles deduplication, double screening with recorded decisions and exclusion reasons, and produces the flow counts. A bibliographic database is where the search is actually run, by a person. The search itself is built outside all three, which is why the reporting items about it are easy to miss.

What does PRISMA require you to report about the search?

Every database, register, website, organisation and reference list searched or consulted, with the date each was last searched; and the full search strategies for all of them, including filters and limits. The 2020 revision made “all” explicit, replacing the older practice of publishing one database’s strategy. The reporting extension adds items for total records per source, deduplication method and any peer review of the strategy.

Can software do the screening for a systematic review?

It can order and manage the screening; it should not make the exclusions unseen. The selection-process item asks you to report how many reviewers screened each record, whether they worked independently and, if applicable, which automation tools were used — so any exclusion a tool made on its own has to be declared in the method. Relevance ranking with a stated stopping rule is defensible; silent automated exclusion is not.

Do you need two reviewers for screening?

The standard requires you to report how many reviewers screened each record and whether independently, rather than requiring two. Much methodological guidance recommends two independent screeners, because a single screener can miss eligible studies. Compromises — two at title and abstract, or one with a second checking a defined sample — are defensible when stated; silence is not.

Is there free systematic review software?

Yes, including free tiers adequate for a review with a few thousand records, and the major bibliographic databases are free to search. Judge a free tool by the same export tests as a paid one: can it give you the strategy per source as run, the counts before deduplication, the exclusion reasons and the per-reviewer decisions? If it can, the price difference is about scale and support, not about whether the review is publishable.

How many of the records found actually make it into a review?

Very few, and the range is wide. In the PROSPERO-based study of 195 completed reviews, the number of studies found in the searches varied enormously between reviews and the mean yield of included studies was a small single-digit percentage. That asymmetry is the whole argument for screening software: almost all of the effort is spent excluding, and excluding is the part that has to be recorded.

Scientific Literature Search.
See it on your own files.

An agentic solution from Botminds. Ask in plain words; get numbered claims, each backed by evidence rows and the exact passage, checked against your own reports. Book a demo, or explore how it works.