SearchResearch & Discovery

Learn · when evidence disagrees

The reproducibility crisis in medicine, from inside a programme

Told from outside, the reproducibility crisis is a story about other people’s papers. Inside a drug programme it arrives as a smaller, sharper question on an ordinary Tuesday: our four samples disagree with their twenty-three, who is right, and what goes in the study plan on Thursday. This guide is about answering that question well, and about the search problem underneath it that Scientific Literature Search was built for.

Updated · 13 min read

See one case, screen by screen →
Watch with transcript →

The reproducibility crisis in medicine is the finding that a large share of published biomedical results do not hold up when someone tries to repeat them: an analysis of past studies put the cumulative prevalence of irreproducible preclinical research above 50%, at a cost of roughly US$28 billion a year in the United States alone. It is hard to act on because the crisis is usually described at the scale of a literature and met at the scale of one experiment — your cohort, four samples, disagreeing with a published series of twenty-three — and because the evidence that would settle it is scattered between journals you can search and internal reports you cannot. Botminds’ Scientific Literature Search puts both in one answer: it reads published papers, patents and your own research reports and signed notebook entries together, writes numbered claims from evidence rows that each open at their passage, and turns a row that contradicts your own work amber in front of the person who owns it.

The screens on this page come from the solution running on its sample library; the programmes, studies, people, journals and sources named in them are fictional. The measured figures are from peer-reviewed research and from the National Institutes of Health, linked where they appear.

The size of the problem, as measured

>50%

estimated cumulative prevalence of irreproducible preclinical research

$28B/yr

United States preclinical spend on research that is not reproducible, on the same estimate

4causes

study design; biological reagents and reference materials; laboratory protocols; data analysis and reporting

Freedman, Cockburn and Simcoe, ‘The Economics of Reproducibility in Preclinical Research’, PLOS Biology 13(6):e1002165, 9 June 2015. The prevalence is an estimate built up from error rates in the four categories, not a count of failed replications; the authors’ probability bounds put the rate anywhere between 18% and 88.5%, and they offer 50% as a starting point for debate.Source: PLOS Biology 13(6):e1002165

What is the reproducibility crisis in medicine?

It is the gap between what the published literature says is true and what holds when the experiment is run again. The most-cited estimate comes from an economic analysis of preclinical research, which defines irreproducibility inclusively — “the existence and propagation of one or more errors, flaws, inadequacies, or omissions … that prevent replication of results” — and attributes it to four categories: study design, biological reagents and reference materials, laboratory protocols, and data analysis and reporting. Adding those up gave a cumulative rate above 50% and a dollar figure of about US$28 billion a year.

Two things about that number are worth saying plainly, because they are usually lost. It is an estimate built from error rates in each category, not a tally of failed replication attempts, and the authors’ own bounds are wide — from 18% to 88.5%. And “irreproducible” does not mean “false”: a result that cannot be reproduced from the published methods may still be real, and the missing ingredient may be a cell line’s passage number or a strain of mouse. The crisis is as much a reporting problem as a biology problem.

How much of it is misconduct? Almost none

Almost none, on the view of the leaders of the National Institutes of Health. Writing in 2014, the Director and Principal Deputy Director of the National Institutes of Health were categorical: “Let’s be clear: with rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct.” That sentence matters operationally, not just rhetorically. If a disagreement between your data and a paper is presumed to be someone’s bad faith, nobody raises it; if it is presumed to be an ordinary difference between two careful labs, it gets raised early and investigated cheaply.

Still, there remains a troubling frequency of published reports that claim a significant result, but fail to be reproducible.

Collins and Tabak, National Institutes of Health, Nature 505:612–613

The same authors named the reason a disagreement often stays invisible: “Further compounding the problem is the difficulty of accessing unpublished data”. Inside a company that difficulty is self-inflicted and fixable. The unpublished data is yours. It is in a translational interim report, a pharmacology study report and a signed notebook entry, and the only reason it does not meet the published series is that nothing searches both.

The ordinary reasons two good labs disagree

Before anyone asks who is right, it is worth listing what usually differs. The NIH piece puts it in one line: “Many of these failures have simple and practical explanations: different animal strains, different lab environments or subtle changes in protocol.”

In practice that list is longer and duller. The model system is not the same — a patient-derived xenograft against a cell line, plasma against tissue. The assay is not the same, or is the same assay read at a different time point. The population is not the same: one series is lung, another is colorectal, and the mechanism behaves differently in each. The reagent is not the same, which the economics analysis singles out as one of its four categories. And the analysis is not the same: a threshold set at one copy number in one paper and at another elsewhere will produce different rates from identical data.

Every item on that list is a question with an answer somewhere in a methods section or an internal report. That is why the first move after a disagreement is not statistical. It is to read both passages side by side and write down which of these differs — which is exactly what the agreement agent in the solution produces as a note, and what a scientist then checks.

What should you do when your own data contradicts a paper?

Treat it as a finding with an owner, not as a verdict on either side. Four steps, in order, keep it cheap:

First, put the two passages beside each other with their n and their model system visible, so you are comparing claims rather than summaries. Second, name what differs — population, model, assay, threshold, analysis. Third, give the disagreement to the person who owns the internal source, because they are the only one who knows whether the number came out of a run worth standing behind. Fourth, decide what it changes about the next study, and write the decision where the next person will find it, with both passages attached.

What not to do is equally definite. Do not quietly drop the published claim from the plan; do not quietly drop your own result either; and do not let a model, an agent or a search tool pick a winner. The sample answer in the solution carries a contradicting internal row as an amber row with a written reason the two may differ, and the owner of that source is the only person who may explain or settle it.

One disagreement, line by line: the published series and our own cohort

Published plasma seriesOur interim analysis
What it foundMET copy gain was the only new alteration in 12 of 23 samples that had itMET copy gain stood alone in 1 of 4; three carried a second alteration
n23 samples with MET gain, from 212 tested4 samples with MET gain, from 38 tested
MaterialPlasma, with matched biopsy in a subsetPlasma, with matched biopsy in two of the four
What it cannot rule outThat a second alteration was present below the detection threshold of the panel usedThat four samples are simply too few to estimate a proportion
What it licensesA working assumption that single-route escape is commonA check, and a wider entry panel for the next cohort
From the fictional sample answer in Scientific Literature Search. Both rows open at their passage: the published one in the paper, ours at page 18 of the translational interim report. The last line is the scientist’s reading, not the agent’s — an agent never decides which side is right.

Both passages on one screen, with what each one studied

The compare screen: the published passage on the left and our own interim report's passage on the right, each with its n, material and model system, and underneath a written note on why the two may differ and what follows for the next cohort if the internal result holds.
Comparing claims rather than summaries, on the fictional sample library. The note underneath is the agreement agent’s reading of why two results may differ; the decision it leads to stays with the scientist.

Small numbers, on both sides

The sharpest disagreements are often between two small numbers, and small numbers behave badly. Twelve of twenty-three is 52%; one of four is 25%. Those look far apart until you remember that four samples cannot distinguish 25% from 52% at any useful confidence. The honest reading is that our cohort is a signal to check, not a refutation — and the solution says so on the row, because calling out small numbers is one of the agreement agent’s standing limits rather than a judgement it makes case by case.

Alone, or with company: the same question in two cohorts

Published series · 23 with MET gain

52%
48%
  • MET gain alone · 12 · 52%
  • A second alteration · 11 · 48%

Our interim · 4 with MET gain

25%
75%
  • Alone · 1 · 25%
  • A second alteration · 3 · 75%
Fictional sample data from the solution’s sample library. The bars are drawn to the counts in each passage. Four samples cannot settle a proportion; the useful output of this comparison is a question for the next cohort, not a percentage.

The symmetrical point is rarely made: the published number is often small too. Twenty-three samples is a thin basis for a claim about how a mechanism usually behaves, and a review that cites that series without its n passes a thin number on as a fact. Every evidence row in the solution carries the n and the model system on the same line, which is the cheapest available defence against that.

Giving the disagreement an owner

A disagreement with no owner becomes a corridor conversation and then nothing. Giving it one changes three things: somebody is accountable for resolving it, the resolution is written where the next person will find it, and the claim stays marked until then.

Ownership has to follow the source, not the question. The person who ran the internal cohort knows whether the assay was short that week, whether two of the four samples came from the same participant, and whether a draft version was superseded. In the solution, a contradicting row goes to the owner of the internal source — and only that person may explain or settle it. Until they do, the claim shows an amber row on every answer that uses it, including answers other people ask later.

The amber row: our own interim against a published claim

An evidence row turned amber inside claim 2 of an answer: our VLT-302 interim 2 found MET copy gain alone in 1 of 4 samples where the published series found it alone in 12 of 23. The claim's agreement meter reads two support, one contradict. Beside the row, the agreement agent's note says small numbers on our side — four samples — so this is a signal to check, not a verdict, with buttons to compare the two passages side by side and send the disagreement to the owner of our source.
Scientific Literature Search on its fictional sample library. The row is amber, not removed: the claim keeps its published support and carries the contradiction visibly, with the reason the two may differ written underneath.

A finding, not a verdict: what a disagreement does and does not license

What it licensesWhat it does not license
The published claim

What it licenses: Marking it as disputed wherever it is used, with your passage attached.

What it does not license: Deleting it from the evidence base, or treating the authors as careless.

Your own result

What it licenses: Checking it: the run, the samples, the threshold, the version of the report.

What it does not license: Promoting it to the answer because it is yours and it is recent.

The next study

What it licenses: Changing an entry criterion, widening a panel, or adding a confirmatory arm.

What it does not license: Abandoning a mechanism on four samples.

The write-up

What it licenses: Stating both results with their n and saying which differences are known.

What it does not license: Choosing the number that makes the story cleaner.

The decision

What it licenses: A named person settling it, on the record, with a reason.

What it does not license: A tool, a model or an agent settling it.

Our reading of how a contradiction between internal and published work should be handled, written as the rules the solution enforces: an agent never decides which side is right, and never closes a disagreement.

What an unseen disagreement costs

The cost is not the wrong answer; it is the time between the disagreement existing and someone seeing it. A contradiction found before the study plan is written changes a sentence. The same contradiction found after the first cohort has enrolled changes a protocol, and may have cost a cohort.

Two published findings bound the exposure. One is that the literature keeps citing work after it has been shown to be unreliable, and nobody checks what that does to the conclusions: Bolland and colleagues, studying citations to publications with integrity concerns, found that hardly any citing papers go back to assess what the withdrawn source did to their own findings, even though many of those findings might change without it (Bolland et al., Accountability in Research, 2022). The other is that a large part of the evidence never reaches a journal at all, which is why the results that never became papers belong in the same answer as the ones that did. A disagreement you cannot see is a disagreement you are paying for.

One disagreement in Scientific Literature Search, from amber row to settled

Found during an ordinary questionDay 0
Answer written, row turns amberminutes
Both passages comparedminutes
Sent to the owner of our sourceone click
  • Answer written, row turns amberminutes
  • Both passages comparedminutes
  • Sent to the owner of our sourceone click
With the ownerNew · open · explained · settled
New: nobody has looked yetnew
Open: the owner is checking the runopen
Explained: why the two differ, written downexplained
Settled: the claim stands, stands with a limit, or fallssettled
  • New: nobody has looked yetnew
  • Open: the owner is checking the runopen
  • Explained: why the two differ, written downexplained
  • Settled: the claim stands, stands with a limit, or fallssettled
The four states a disagreement moves through in Scientific Literature Search, on fictional sample data: four were raised in the last 30 days — one new, one open, one explained, one settled. Only the owner of the internal source may move it; the lengths of the bars show the sequence, not measured durations.

Writing it down so it outlives the people who found it

The reproducibility literature’s practical recommendations all come down to writing things down: the strain, the passage number, the reagent lot, the threshold, the analysis as run. The same discipline applies one level up, to the disagreement itself. An explanation that lives in a scientist’s head is lost at the next reorganisation; an explanation attached to the claim is found by the next person who asks the question.

That is why a kept answer in the solution carries its claims, its evidence rows and the passages behind them, with any disagreement attached until its owner settles it — and why the programme notebook also lists the questions the team asked this week. A colleague who asks the same question next quarter starts from the rows, and from the amber one, rather than from an empty search box.

Where Scientific Literature Search keeps every disagreement between our work and the literature

The disagreements screen listing four disagreements raised in the last 30 days, each with the claim it touches, the published source, the internal source that contradicts it, the owner and a state: new, open, explained or settled. One is new from this morning on MET copy gain, one open with the pharmacology lead on twice-daily dosing, one explained on assay length and one settled on a panel threshold.
Scientific Literature Search on its fictional sample library. The list is the audit of a programme’s own contradictions — which is a strange thing to want until the first time a reviewer asks whether anyone checked.

When to repeat, when to proceed, when to stop

Not every disagreement deserves a replication study. The decision turns on three things: how much the next step costs if the published claim is wrong, how small the numbers are on each side, and whether the difference is already explained by something in the two methods sections.

Repeat when the published claim is load-bearing for an expensive decision and the difference is not explained by model, population or assay. Proceed, with the limit written into the plan, when the disagreement is explained — a different model system, a different threshold — and the plan can be designed to be robust to it; that is what a wider entry panel buys. Stop and escalate when the two results cannot both be true and the decision cannot be made robust to either; that is a programme question, not a literature question, and it needs the people who own both the data and the plan in the same room.

Underneath all three is a search problem. None of these decisions is available to a team that cannot see its own evidence beside the published evidence on the day the question is asked. Tools that promise to read the literature for you should be judged on exactly that, which is what the tests to run before you trust one are for. To see a published series and an internal cohort answered together, start from Scientific Literature Search.

Questions people ask

How much research is irreproducible?

The most-cited estimate for preclinical work puts the cumulative prevalence above 50%, with about US$28 billion a year spent in the United States on preclinical research that is not reproducible. It is an estimate assembled from error rates across four categories — study design, reagents and reference materials, laboratory protocols, and data analysis and reporting — not a count of failed replications. The authors’ own bounds run from 18% to 88.5%, and they present the 50% midpoint as a reasonable starting point for further debate rather than a measurement.

Is irreproducible research fraud?

Almost never. The National Institutes of Health stated that “with rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct”, and listed the usual causes instead: different animal strains, different lab environments, subtle changes in protocol. Treating a disagreement as misconduct is both usually wrong and practically costly, because it stops people raising disagreements at all.

Why can’t studies be replicated?

Most often because the published methods do not contain everything the result depended on — the strain, the reagent lot, the passage number, the exact threshold, the analysis as run. Add publication bias on top: results that reached a journal are not a random sample of results that were produced, so the published effect tends to be the larger one.

How do you report a failed replication?

Report it as a result with its own methods, not as a comment on the original: state your n, model system, protocol and analysis, say which of these differ from the original, and say what your data can and cannot rule out. Inside a company the same shape works, with one addition — name the person who owns the internal data, because they are the one who can say whether the run is sound.

What makes a disagreement worth acting on?

Three things together: the published claim is load-bearing for an expensive decision; the numbers on at least one side are small enough that the two results could both be sampling noise; and no difference in population, model system, assay or threshold already explains the gap. If a known difference explains it, the right action is usually to design the next study to be robust to it rather than to repeat the experiment.

Does publication bias make the crisis worse?

Yes, and measurably. Across 425 methodology reports covering 307,028 conference abstracts, only 37.3% had been published in full, and abstracts with a significant result were more likely to make it — a risk ratio of 1.31 across 45 reports. So the literature a replication is judged against is itself tilted towards the stronger findings, which raises the chance that a faithful repeat looks like a failure.

Scientific Literature Search.
See it on your own files.

An agentic solution from Botminds. Ask in plain words; get numbered claims, each backed by evidence rows and the exact passage, checked against your own reports. Book a demo, or explore how it works.