Now is a crucial time for fortifying trust in scientific research, and that will require key actions within the institutions that produce it, said an international panel of presenters at the third Stanford Rigor and Reproducibility Colloquium held at Berg Hall on Oct. 20.
Among the most important:
- Prioritizing replication studies, those in which one research team’s findings are repeated by another.
- Standardizing the systems used to gather, collate and share data.
- Standardizing the scientific guidelines and data-gathering protocols within and between institutions.
- Shifting the culture to incentivize transparency and collaboration and to prioritize rigorous methodology ahead of the race to get published.
At Stanford, the university and the medical school have developed a reputation for studying the research processes employed by their thousands of scientists — and creating infrastructures to implement best practices via the Stanford Program on Research Rigor and Reproducibility (SPORR) and a School of Medicine Task Force on Research, Practice and Culture. That made it a logical host for what has grown into a daylong event with panelists from top universities in the U.S. and abroad.
“It may not always seem like the most exciting work, and maybe it’s not the work that’ll win a Nobel prize,” said Ruth O’Hara, PhD, senior associate dean for research, Director of the Stanford CTSA and the Lowell W. and Josephine Q. Berry Professor in the Department of Psychiatry and Behavioral Sciences. “But it is the work that will make sure we can continue to produce scientific excellence and build upon what has already been achieved here at Stanford, and in the United States more broadly, as leaders of the biomedical revolution.”
This year, panelists included participants from Duke University and universities in England and the Netherlands. They were joined by the dean of Stanford Medicine, the Stanford University president and the full gamut of university researchers — principal investigators, post-doctoral scholars, PhD students — whose careers are defined by what they get published in reputable scientific journals.
Leaders from the National Institutes of Health — which funds most of the research conducted at U.S. universities — were scheduled to attend but had to cancel because of the government shutdown. Instead, Steve Goodman, MD, PhD, a professor of epidemiology and population health who founded and is the director of SPORR, showed slides from NIH outlining new initiatives in which it plans to incentivize replication studies. The initiatives, which can involve universities, independent laboratories and the NIH, have proved challenging to complete largely because there is no clean, tidy handoff of data and protocol from one research team to another — and no funding source until now.
Goodman, like the university leaders who spoke before him, emphasized that conducting studies that involve multiple labs is not about a lack of trust in a researcher’s work, but instead unsuspected sources of variation in study results. Understanding those will enable better-designed experiments that instill more confidence in the results.
“Every university has a superstructure around misconduct. They do not have a superstructure around rigor and reproducibility,” Goodman said. “Part of our conversation today is about how do we build something so we don’t inadvertently trip over these potholes?”
Systems building
Without standardized systems for sharing data — and the code used to produce it — the goal of an open source-style transparency to allow for more team science and reproducibility studies remains elusive, panelists said.
Marcus Munafo, PhD, the deputy vice chancellor and provost at the University of Bath, recounted the story of a researcher who was reluctant to share his findings during the race for a COVID-19 vaccine. “Not that there was anything wrong with the code — it was just a mess and he didn’t want anyone to look at it,” he said. “Most people’s code is like that, and if you can create systems that encourage transparency, then you create a nudge toward good practices from the outset.”
Stanford Medicine’s Paul Nuyujukian, MD, PhD, assistant professor of bioengineering and of neurosurgery, discussed how he is trying to help academic institutions make their research data more uniform and shareable. “It’s just a matter of providing the infrastructure to the researchers to help them do this,” Nuyujukian said. “Of bringing over a culture of these bread-and-butter techniques that are used in the [information technology] world every day.”
Goodman called the suite of tools and educational programs Nuyujukian has been building within the Wu Tsai Neuroscience Institute “a model for what we’ve been talking about here,” and said a commitment to giving researchers the systems they need is where the rubber meets the road.
“I mean, we don’t ask them to build their own flasks and beakers …,” Goodman said.
The efficiency factor
While scientific validity is core to the mission, a dual byproduct is that the proper tools promote a more productive process, said Lloyd Minor, MD, the dean of the School of Medicine and vice president for medical affairs at Stanford University.
“By approaching the technology that can promote and facilitate rigor and reproducibility, we do a lot to improve the efficiency and effectiveness of the scientific process,” Minor said. It’s about “how to do the very best science all the time, and in a way that encourages best practices and raises productivity.”
Sai Gourisankar, PhD, a postdoctoral scholar in the Stanford Cancer Institute, was awarded the first Spark/SPORR award, dedicated to Glenn Begley, a pioneer in the rigor and reproducibility field, for his work as co-first author on a study that focused on getting lymphoma cells to self-destruct. After Gourisankar spoke about how the research spanned multiple labs and institutions, with an emphasis on replicating the results before publishing, Goodman heaped praise on the researcher and the process.
“This is collaboration and reproducibility at scale,” he said. “You’ve really accelerated progress in this area.”
A culture of openness
Daria Mochly-Rosen, PhD, the George D. Smith Professor in Translational Medicine, recalled the first time she presented research before a massive conference crowd. When she finished, her first commenter posited: “I don’t believe it.” It shook her, Mochly-Rosen said. And then it motivated her to embrace the challenge and send every bit of data she had to that person’s lab for replication.
Early career researchers should take the lesson to heart, she said. And academic institutions must incentivize the type of transparent mindset that leads to rigor and reproducibility.
“We need to celebrate people who are doing it so it can become part of the culture,” she said. “It’s a good way to say, ‘We all care about it, not just the institution, or the funders. But all of us as scientists.’”
Stanford University President Jonathan Levin, PhD, noted that the institution has become a national leader in rigor and reproducibility.
“It’s incredibly important that we not only have rules and policies around those areas, but that we establish good practices and norms,” he said. “I think the programs here at the medical school have really set a gold standard for how to do that in both cases, and I’m thrilled that we have that enterprise here.”
The Stanford Program on Research Rigor and Reproducibility (SPORR) will fund several pilot projects within individual departments, divisions or research groups. Directions on how to apply for those funds will be announced soon on the SPORR and Stanford Center for Clinical and Translational Research and Education (Spectrum) websites.
