Reproducibility Rounds
Reproducibility Rounds are a webinar series designed to foster open dialogue and exchange of experiences between institutions, researchers, and other stakeholders on the critical topics of research rigor and reproducibility.
Rounds are organized by the Stanford School of Medicine’s Program on Research Rigor & Reproducibility (SPORR), in collaboration with Columbia, Duke, Harvard, and Indiana, and occur every third Tuesday of the month.
If you have suggestions for a topic that should be covered in the Rounds, or would like to present, please contact us.
Additionally, if you would like to informed about upcoming rounds and SPORR activities, sign up for our Newsletter.
Last Rounds: May 26 - 9 am to 10 am PT time
Next Rounds: TBA
Title: What Biomedicine Can Learn about Reproducibility from Social & Behavioral Research: The SCORE Project - See the Recording
Speaker: Brian Nosek
Abstract: SCORE, a collaboration of 865 researchers, is now released as three papers in Nature, six preprints, and a lot of data (https://cos.io/score/). SCORE examined repeatability of findings from the social-behavioral sciences and tested whether human and automated methods could predict replicability. A representative subset of 600 claims were available for repeatability tests: reproductions (same data, same analysis), robustness tests (same data, different analyses), and replications (same question, different data). For reproducibility, we could obtain data for only 24% of the 600 papers. Of the 143 papers (551 claims) assessed, we precisely reproduced 54%, and approximately reproduced 74%. We were much more likely to succeed if authors shared data and code, versus just data or if we had to reconstruct data from original sources. For robustness, 34% of reanalyses showed the same result within a narrow tolerance (+/- .05 Cohen’s d), and 57% with a wider tolerance (+/- .20). Limiting to statistical conclusions (p<.05?), 74% of reanalyses reached the same conclusion, 24% observed no effect, and 2% observed an opposing effect. For replicability, we tested findings from 164 papers and successfully replicated 49% of them with the common statistical significance criterion. Original studies had an average effect size of r = 0.25, replication studies r = 0.10. Best performing human methods achieved about 75% accuracy predicting replication outcomes. The talk will also discuss implications for researchers and institutions on ways to improve the credibility of research.
Previous Rounds:
2025:
November 18: Niklas Bobrovitz and Stephana Moss: How Reproducible is Health Science Research?
September 16: Prof. Nichole E. Carlson, MD, PhD: Non-Reproducibility of Real-World Data Research: A Case Study (Video Recording).
May 20: Prof. Olavo Amaral, MD, PhD: Estimating Replicability of Lab Biology Research at a National Level: Results and Lessons from the Brazilian Reproducibility Initiative (Video Recording). Slides
March 18: Naomi Schrag and Roger Lefrot: Rigor and Reproducibility Efforts at Columbia University (ReaDI): Successes and Challenges (Video Recording)
February 18: Nafisa Jadavji, Susan McClatchy, Hao Ye, Flavio Azevedo: Reproducibility Training Initiatives. (Video Recording)
January 21: Timothy M. Errington: Replicating 193 Preclinical Cancer Biology Experiments: Lessons, Surprises, and Future Directions (Video Recording)
2024:
November 19: Leah J. Welty and Manisha Desai: How to Improve Reproducibility of Team Science: A CTSA BERD*s Eye View (Video Recording)
October 22: Shai Silberberg and Devon Crawford: Why Talk (and What to Do) About Research Quality? A Perspective from Within the NIH. (Video recording)
2023:
SPORR organized two webinars primarily aimed for the researchers of Stanford School of Medicine:
May 31: Ivan Oransky (Co-Founder of Retraction Watch) - How Bibliometrics & School Rankings Reward Unreliable Science and What Can Be Done About It. (Video Recording). This talk was turned into an opinion piece published in the BMJ journal on 17 August 2023.
Apr 12: Michael Eisen presented Changes at eLife.