Reproducible research: A hunt for the truth

The journal Nature recently published the results of a survey that asked scientists if they thought the published scientific literature is mostly correct.

The exact question they asked nearly 1,600 scientists in fields ranging from physics to biomedicine was, “How much published work in your field is reproducible?” Many scientists who answered the survey tended to be quite confident in their field’s literature even though numerous studies have shown reproducibility as low as 11 percent in some fields. Three-quarters of the researchers thought that at least half of the papers published in their field would be reproducible.

But it’s not just Pollyannaish optimism that is the problem, say three researchers from the Meta-Research Innovation Center at Stanford, known as METRICS. It turns out that “reproducibility,” “replicability” and several other terms are not used consistently in scientific communication. To fix the flaws of science, everyone needs to use such terms more thoughtfully and with precision, the researchers wrote in a paper titled “What does research reproducibility mean?” that was published June 2 in Science Translational Medicine.

The three authors of the paper are Steven Goodman, MD, PhD, professor of medicine and of health research and policy at Stanford; Daniele Fanelli, PhD, a senior research scientist at METRICS; and John Ioannidis, MD, DSc, professor of medicine and of health research and policy at Stanford. They make the case that even if we define and use terms such as “reproducibility,” “replicability,” “reliability,” “robustness” and “generalizability” consistently and correctly, what researchers are really after is the truth.

The paper said that “treating reproducibility as an end in itself — rather than as an imperfect surrogate for scientific truth — is partly responsible for the current terminological and operational morass, as well as how we can benefit by refocusing on cumulative evidence and truth.”

The paper included an amusing table of terms for misleading practices in science, including torturing, data snooping and P-hacking.

“We need,” the authors wrote, “to move toward a better understanding of the relationship between reproducibility, cumulative evidence and the truth of scientific claims.”