Reproducible research: A hunt for the truth

Researchers write that “reproducibility,” “replicability” and several other terms are not used consistently in scientific communication.

Steven Goodman

The journal Nature recently published the results of a survey that asked scientists if they thought the published scientific literature is mostly correct.

The exact question they asked nearly 1,600 scientists in fields ranging from physics to biomedicine was, “How much published work in your field is reproducible?” Many scientists who answered the survey tended to be quite confident in their field’s literature even though numerous studies have shown reproducibility as low as 11 percent in some fields. Three-quarters of the researchers thought that at least half of the papers published in their field would be reproducible.

But it’s not just Pollyannaish optimism that is the problem, say three researchers from the Meta-Research Innovation Center at Stanford, known as METRICS. It turns out that “reproducibility,” “replicability” and several other terms are not used consistently in scientific communication. To fix the flaws of science, everyone needs to use such terms more thoughtfully and with precision, the researchers wrote in a paper titled “What does research reproducibility mean?” that was published June 2 in Science Translational Medicine.

John Ioannidis

The three authors of the paper are Steven Goodman, MD, PhD, professor of medicine and of health research and policy at Stanford; Daniele Fanelli, PhD, a senior research scientist at METRICS; and John Ioannidis, MD, DSc, professor of medicine and of health research and policy at Stanford. They make the case that even if we define and use terms such as “reproducibility,” “replicability,” “reliability,” “robustness” and “generalizability” consistently and correctly, what researchers are really after is the truth.

The paper said that “treating reproducibility as an end in itself — rather than as an imperfect surrogate for scientific truth — is partly responsible for the current terminological and operational morass, as well as how we can benefit by refocusing on cumulative evidence and truth.”

The paper included an amusing table of terms for misleading practices in science, including torturing, data snooping and P-hacking.

“We need,” the authors wrote, “to move toward a better understanding of the relationship between reproducibility, cumulative evidence and the truth of scientific claims.”



Stanford Medicine integrates research, medical education and health care at its three institutions - Stanford University School of Medicine, Stanford Health Care (formerly Stanford Hospital & Clinics), and Lucile Packard Children's Hospital Stanford. For more information, please visit the Office of Communication & Public Affairs site at http://mednews.stanford.edu.

Leading in Precision Health

Stanford Medicine is leading the biomedical revolution in precision health, defining and developing the next generation of care that is proactive, predictive and precise. 

A Legacy of Innovation

Stanford Medicine's unrivaled atmosphere of breakthrough thinking and interdisciplinary collaboration has fueled a long history of achievements.