The reproducibility crisis in medicine is the finding that a large share of published biomedical results do not hold up when someone tries to repeat them: an analysis of past studies put the cumulative prevalence of irreproducible preclinical research above 50%, at a cost of roughly US$28 billion a year in the United States alone. It is hard to act on because the crisis is usually described at the scale of a literature and met at the scale of one experiment — your cohort, four samples, disagreeing with a published series of twenty-three — and because the evidence that would settle it is scattered between journals you can search and internal reports you cannot. Botminds’ Scientific Literature Search puts both in one answer: it reads published papers, patents and your own research reports and signed notebook entries together, writes numbered claims from evidence rows that each open at their passage, and turns a row that contradicts your own work amber in front of the person who owns it.
The screens on this page come from the solution running on its sample library; the programmes, studies, people, journals and sources named in them are fictional. The measured figures are from peer-reviewed research and from the National Institutes of Health, linked where they appear.
The size of the problem, as measured
>50%
estimated cumulative prevalence of irreproducible preclinical research
$28B/yr
United States preclinical spend on research that is not reproducible, on the same estimate
4causes
study design; biological reagents and reference materials; laboratory protocols; data analysis and reporting
What is the reproducibility crisis in medicine?
It is the gap between what the published literature says is true and what holds when the experiment is run again. The most-cited estimate comes from an economic analysis of preclinical research, which defines irreproducibility inclusively — “the existence and propagation of one or more errors, flaws, inadequacies, or omissions … that prevent replication of results” — and attributes it to four categories: study design, biological reagents and reference materials, laboratory protocols, and data analysis and reporting. Adding those up gave a cumulative rate above 50% and a dollar figure of about US$28 billion a year.
Two things about that number are worth saying plainly, because they are usually lost. It is an estimate built from error rates in each category, not a tally of failed replication attempts, and the authors’ own bounds are wide — from 18% to 88.5%. And “irreproducible” does not mean “false”: a result that cannot be reproduced from the published methods may still be real, and the missing ingredient may be a cell line’s passage number or a strain of mouse. The crisis is as much a reporting problem as a biology problem.
How much of it is misconduct? Almost none
Almost none, on the view of the leaders of the National Institutes of Health. Writing in 2014, the Director and Principal Deputy Director of the National Institutes of Health were categorical: “Let’s be clear: with rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct.” That sentence matters operationally, not just rhetorically. If a disagreement between your data and a paper is presumed to be someone’s bad faith, nobody raises it; if it is presumed to be an ordinary difference between two careful labs, it gets raised early and investigated cheaply.
Still, there remains a troubling frequency of published reports that claim a significant result, but fail to be reproducible.
Collins and Tabak, National Institutes of Health, Nature 505:612–613
The same authors named the reason a disagreement often stays invisible: “Further compounding the problem is the difficulty of accessing unpublished data”. Inside a company that difficulty is self-inflicted and fixable. The unpublished data is yours. It is in a translational interim report, a pharmacology study report and a signed notebook entry, and the only reason it does not meet the published series is that nothing searches both.
The ordinary reasons two good labs disagree
Before anyone asks who is right, it is worth listing what usually differs. The NIH piece puts it in one line: “Many of these failures have simple and practical explanations: different animal strains, different lab environments or subtle changes in protocol.”
In practice that list is longer and duller. The model system is not the same — a patient-derived xenograft against a cell line, plasma against tissue. The assay is not the same, or is the same assay read at a different time point. The population is not the same: one series is lung, another is colorectal, and the mechanism behaves differently in each. The reagent is not the same, which the economics analysis singles out as one of its four categories. And the analysis is not the same: a threshold set at one copy number in one paper and at another elsewhere will produce different rates from identical data.
Every item on that list is a question with an answer somewhere in a methods section or an internal report. That is why the first move after a disagreement is not statistical. It is to read both passages side by side and write down which of these differs — which is exactly what the agreement agent in the solution produces as a note, and what a scientist then checks.
What should you do when your own data contradicts a paper?
Treat it as a finding with an owner, not as a verdict on either side. Four steps, in order, keep it cheap:
First, put the two passages beside each other with their n and their model system visible, so you are comparing claims rather than summaries. Second, name what differs — population, model, assay, threshold, analysis. Third, give the disagreement to the person who owns the internal source, because they are the only one who knows whether the number came out of a run worth standing behind. Fourth, decide what it changes about the next study, and write the decision where the next person will find it, with both passages attached.
What not to do is equally definite. Do not quietly drop the published claim from the plan; do not quietly drop your own result either; and do not let a model, an agent or a search tool pick a winner. The sample answer in the solution carries a contradicting internal row as an amber row with a written reason the two may differ, and the owner of that source is the only person who may explain or settle it.
One disagreement, line by line: the published series and our own cohort
| Published plasma series | Our interim analysis | |
|---|---|---|
| What it found | MET copy gain was the only new alteration in 12 of 23 samples that had it | MET copy gain stood alone in 1 of 4; three carried a second alteration |
| n | 23 samples with MET gain, from 212 tested | 4 samples with MET gain, from 38 tested |
| Material | Plasma, with matched biopsy in a subset | Plasma, with matched biopsy in two of the four |
| What it cannot rule out | That a second alteration was present below the detection threshold of the panel used | That four samples are simply too few to estimate a proportion |
| What it licenses | A working assumption that single-route escape is common | A check, and a wider entry panel for the next cohort |
Both passages on one screen, with what each one studied

Small numbers, on both sides
The sharpest disagreements are often between two small numbers, and small numbers behave badly. Twelve of twenty-three is 52%; one of four is 25%. Those look far apart until you remember that four samples cannot distinguish 25% from 52% at any useful confidence. The honest reading is that our cohort is a signal to check, not a refutation — and the solution says so on the row, because calling out small numbers is one of the agreement agent’s standing limits rather than a judgement it makes case by case.
Alone, or with company: the same question in two cohorts
Published series · 23 with MET gain
- MET gain alone · 12 · 52%
- A second alteration · 11 · 48%
Our interim · 4 with MET gain
- Alone · 1 · 25%
- A second alteration · 3 · 75%
The symmetrical point is rarely made: the published number is often small too. Twenty-three samples is a thin basis for a claim about how a mechanism usually behaves, and a review that cites that series without its n passes a thin number on as a fact. Every evidence row in the solution carries the n and the model system on the same line, which is the cheapest available defence against that.
Giving the disagreement an owner
A disagreement with no owner becomes a corridor conversation and then nothing. Giving it one changes three things: somebody is accountable for resolving it, the resolution is written where the next person will find it, and the claim stays marked until then.
Ownership has to follow the source, not the question. The person who ran the internal cohort knows whether the assay was short that week, whether two of the four samples came from the same participant, and whether a draft version was superseded. In the solution, a contradicting row goes to the owner of the internal source — and only that person may explain or settle it. Until they do, the claim shows an amber row on every answer that uses it, including answers other people ask later.
The amber row: our own interim against a published claim

A finding, not a verdict: what a disagreement does and does not license
What it licenses: Marking it as disputed wherever it is used, with your passage attached.
What it does not license: Deleting it from the evidence base, or treating the authors as careless.
What it licenses: Checking it: the run, the samples, the threshold, the version of the report.
What it does not license: Promoting it to the answer because it is yours and it is recent.
What it licenses: Changing an entry criterion, widening a panel, or adding a confirmatory arm.
What it does not license: Abandoning a mechanism on four samples.
What it licenses: Stating both results with their n and saying which differences are known.
What it does not license: Choosing the number that makes the story cleaner.
What it licenses: A named person settling it, on the record, with a reason.
What it does not license: A tool, a model or an agent settling it.
What an unseen disagreement costs
The cost is not the wrong answer; it is the time between the disagreement existing and someone seeing it. A contradiction found before the study plan is written changes a sentence. The same contradiction found after the first cohort has enrolled changes a protocol, and may have cost a cohort.
Two published findings bound the exposure. One is that the literature keeps citing work after it has been shown to be unreliable, and nobody checks what that does to the conclusions: Bolland and colleagues, studying citations to publications with integrity concerns, found that hardly any citing papers go back to assess what the withdrawn source did to their own findings, even though many of those findings might change without it (Bolland et al., Accountability in Research, 2022). The other is that a large part of the evidence never reaches a journal at all, which is why the results that never became papers belong in the same answer as the ones that did. A disagreement you cannot see is a disagreement you are paying for.
One disagreement in Scientific Literature Search, from amber row to settled
- Answer written, row turns amberminutes
- Both passages comparedminutes
- Sent to the owner of our sourceone click
- New: nobody has looked yetnew
- Open: the owner is checking the runopen
- Explained: why the two differ, written downexplained
- Settled: the claim stands, stands with a limit, or fallssettled
Writing it down so it outlives the people who found it
The reproducibility literature’s practical recommendations all come down to writing things down: the strain, the passage number, the reagent lot, the threshold, the analysis as run. The same discipline applies one level up, to the disagreement itself. An explanation that lives in a scientist’s head is lost at the next reorganisation; an explanation attached to the claim is found by the next person who asks the question.
That is why a kept answer in the solution carries its claims, its evidence rows and the passages behind them, with any disagreement attached until its owner settles it — and why the programme notebook also lists the questions the team asked this week. A colleague who asks the same question next quarter starts from the rows, and from the amber one, rather than from an empty search box.
Where Scientific Literature Search keeps every disagreement between our work and the literature

When to repeat, when to proceed, when to stop
Not every disagreement deserves a replication study. The decision turns on three things: how much the next step costs if the published claim is wrong, how small the numbers are on each side, and whether the difference is already explained by something in the two methods sections.
Repeat when the published claim is load-bearing for an expensive decision and the difference is not explained by model, population or assay. Proceed, with the limit written into the plan, when the disagreement is explained — a different model system, a different threshold — and the plan can be designed to be robust to it; that is what a wider entry panel buys. Stop and escalate when the two results cannot both be true and the decision cannot be made robust to either; that is a programme question, not a literature question, and it needs the people who own both the data and the plan in the same room.
Underneath all three is a search problem. None of these decisions is available to a team that cannot see its own evidence beside the published evidence on the day the question is asked. Tools that promise to read the literature for you should be judged on exactly that, which is what the tests to run before you trust one are for. To see a published series and an internal cohort answered together, start from Scientific Literature Search.
Questions people ask
How much research is irreproducible?
The most-cited estimate for preclinical work puts the cumulative prevalence above 50%, with about US$28 billion a year spent in the United States on preclinical research that is not reproducible. It is an estimate assembled from error rates across four categories — study design, reagents and reference materials, laboratory protocols, and data analysis and reporting — not a count of failed replications. The authors’ own bounds run from 18% to 88.5%, and they present the 50% midpoint as a reasonable starting point for further debate rather than a measurement.
Is irreproducible research fraud?
Almost never. The National Institutes of Health stated that “with rare exceptions, we have no evidence to suggest that irreproducibility is caused by scientific misconduct”, and listed the usual causes instead: different animal strains, different lab environments, subtle changes in protocol. Treating a disagreement as misconduct is both usually wrong and practically costly, because it stops people raising disagreements at all.
Why can’t studies be replicated?
Most often because the published methods do not contain everything the result depended on — the strain, the reagent lot, the passage number, the exact threshold, the analysis as run. Add publication bias on top: results that reached a journal are not a random sample of results that were produced, so the published effect tends to be the larger one.
How do you report a failed replication?
Report it as a result with its own methods, not as a comment on the original: state your n, model system, protocol and analysis, say which of these differ from the original, and say what your data can and cannot rule out. Inside a company the same shape works, with one addition — name the person who owns the internal data, because they are the one who can say whether the run is sound.
What makes a disagreement worth acting on?
Three things together: the published claim is load-bearing for an expensive decision; the numbers on at least one side are small enough that the two results could both be sampling noise; and no difference in population, model system, assay or threshold already explains the gap. If a known difference explains it, the right action is usually to design the next study to be robust to it rather than to repeat the experiment.
Does publication bias make the crisis worse?
Yes, and measurably. Across 425 methodology reports covering 307,028 conference abstracts, only 37.3% had been published in full, and abstracts with a significant result were more likely to make it — a risk ratio of 1.31 across 45 reports. So the literature a replication is judged against is itself tilted towards the stronger findings, which raises the chance that a faithful repeat looks like a failure.