Stanford professor's pivotal role in bringing commenting capability to PubMed

Norbert von der Groeben

Rob Tibshirani helped to establish an online forum for commenting on articles indexed in PubMed. It opened to all authors of such articles on Oct. 22.

What would you do if you saw an error in a medical journal article?

If you’re biostatistician Rob Tibshirani, PhD, you’d try to correct it. And then, because there’s no easy way to do that, you’d get frustrated and think, like many before you, “Wouldn’t it be nice if there were a website where you could let people know about mistakes in journal articles?”

Tibshirani, a professor of health research and policy and of statistics at Stanford University, had this thought most recently after noticing what seemed to be a serious drawback with a new method, published in Science in December 2011, for analyzing large data sets. Wishing once again for a corrections website, he asked Pat Brown, PhD, professor of biochemistry, what he thought: Was it feasible? Brown was a good person to ask. He had co-founded Public Library of Science, an open-access publishing venture, in 2000 and started one of the first open-access journals, PLOS Biology, 10 years ago.

“Pat told me PLOS had tried it, but didn’t have enough traffic to make it work,” Tibshirani said. “He suggested we try it at a place with a lot of traffic: PubMed.”

PubMed, the massive index of biomedical literature run by the National Center for Biotechnology Information, is one of the Internet’s busiest biomedical websites, with millions of users a day.

That wished-for comment site, which became a reality for Tibshirani and about 300 other beta testers in June of this year, opened to a much larger group of users on Oct. 22. The site, PubMed Commons, allows all authors of articles indexed in PubMed — who number in the hundreds of thousands — to make and read comments on any article indexed on PubMed. It’s a step forward for what’s been termed “post-publication peer review.” Whether it’s a big step or a small step depends on whom you ask.

The development team hopes it will become not only a place for criticism but for discussion — for questions about techniques, for suggestions and for praise.

A lonely business

“Science can be lonely,” Tibshirani said. “Just having people talk about your work is nice. Sure it’s nice to have good comments. But it’s nice to have comments at all. At least someone cares enough to read your paper.”

You might wonder: Why is a site for comments on papers even necessary? Journals, after all, print letters to the editor, and some journals’ websites even allow comments. The catch, though, said Tibshirani, is that journals sometimes decline to publish corrections, or when they publish them they limit the length. Explaining a problem in a few hundred words can be very difficult, he said.

A further hitch is the lack of a centralized location for comments. With more than 20,000 biomedical journals in existence, it’s not realistic to expect many people to notice when a correction is made.

In fact, once a correction is made, it’s often ignored. Research by John Ioannidis, MD, PhD, professor of medicine at Stanford, has documented research papers that have been discredited, yet were cited for years afterward. “If the research was published in a high-ranking journal, it’s considered the truth. The culture is that it’s set in stone,” Tibshirani said.

In some ways, it’s surprising that the leaders of the National Institutes of Health have allowed comments to go forward, Tibshirani said. He knew there was considerable concern that negative comments could harm the reputation of NIH-funded research or be used to attack competitors, and that comments both positive or negative could have financial ramifications.

To keep commenters responsible and to make any potential conflicts of interest transparent, no anonymous comments are allowed, and commenting is restricted to the scientific community, at least during the pilot period, Tibshirani said.

The journey

Getting the go-ahead for even this much openness was not simple.

When Tibshirani first brought up the idea with Brown, they went into Brown’s Stanford office and Skyped David Lipman, MD, director of the National Center for Biotechnology Information, to see if he thought it was feasible. In fact, Lipman had been thinking about doing something like this for nearly a decade. “But I was concerned how we could manage it, and until recently I didn’t think it would get the support of NIH leadership,” he said.

But he told them he thought it was worth a try. “It helped that the request was coming from out in the community, not within NIH,” he added.

It also helped that Brown is an old friend of the director of the National Cancer Institute, Harold Varmus, MD. Brown presented the idea to Varmus, who also liked it.

So in September of 2012, Lipman proposed the site to the NIH steering committee, consisting of NIH director Francis Collins, MD, PhD, and 10 other directors from NIH institutes and centers, and got the go-ahead to develop the idea.

“At that point, I realized I’m not going to develop the idea,” Lipman said. “Rob and Pat should organize themselves and get others involved to do that; they should take the lead. I wanted to see if there’s an active group of people who care enough about it.”

Tibshirani became the community organizer and worked with Brown to involve about 300 people to discuss the ground rules and begin using an early version of the site. “We hashed over the basic ideas such as the level of inclusiveness. Do you allow anonymous comments? And we began commenting,” Tibshirani said. They had posted just over 100 comments to the site by early October.

Naming names

Anonymity was the most contentious issue, Tibshirani said. Anonymity’s big draw is that it would allow scientists to criticize others’ work without jeopardizing their own careers. Ivan Oransky, MD, president and global editorial director of MedPage Today and co-founder of the blog RetractionWatch, argued that anonymity would be crucial. Oransky is a proponent of post-publication review but was not involved in the PubMed Commons project.

“It’s sad that it’s a problem to identify yourself. But let scientists be honest about the system we have,” Oransky said. “What do you think it’s like for someone at any point in their career who comes forward publicly and says there’s something wrong with the grand pooh-bah’s paper? It’s terrifying. It’s because science is as political and hierarchical a human enterprise as any other.

“The distinction should not be: Are you willing to name yourself. The distinction should be: Are you providing verifiable information that helps scientific literature be more correct.”

In fact, while the team has been developing and testing the new site, at least one other site has been launched to enable comments on biomedical literature. PubPeer, for example, is a website run by an anonymous group that allows anonymous commenting on any paper in PubMed. The site went live in October 2012, and a comment there on a high-profile paper published in Cell led to a correction in the journal.

But the NIH is not ready for anonymous comments, nor are a good many of the beta testers, Tibshirani said.

“It’s a tricky thing,” he said, “because you want it to be as transparent as possible. But on the other hand, imagine someone publishes that a drug has side effects. If comments are anonymous, someone associated with a company that makes the drug could take potshots at the authors of that study. When money and lives are at stake, the quality of the interchange must be high. So for now, we have to identify ourselves.”

Lipman said, “In the beginning, it’s balancing inclusiveness and relevance. You want to have as many people as possible who have something relevant to say to be in there. That obviously includes more people than who have had papers published in PubMed. But where do you draw the line? The thinking for now is if I want to be able to comment on your paper, you should be able to comment on my paper,” he said.

Public access

Ultimately, the organizers plan to allow the general public to view comments on PubMed Commons and register to post them, as well.

“I would really want nonscientists to be able to comment,” Oransky said. “Many journalists I know read the medical literature better than I can. And some patients know more about individual diseases and conditions than doctors. I would want them to be critiquing the literature in the same place scientists are critiquing it.”

PubMed Commons is starting as a relatively simple site designed to require minimal involvement by NCBI staff. “We’re not moderating comments as they come up, which would take a lot of staff,” Lipman said. “We’ll let registered users report a problem. If they see something slanderous or a conflict of interest, or irrelevant or someone trying to sell something, that can be reported and removed.”

The site’s features include the ability to search for articles that have been commented on, and to search by individuals to see their comments. Registrants can also tag a comment as helpful (or not).

From Lipman’s perspective, positive comments are of even more interest than corrections. “It’s hard to find the good articles that interest me in all the articles that are being published. One way is to find out what people I know thought was good,” he said. “If I could have a growing number people to follow, and they recommend even just one paper a month, I’m going to be so much better off.”

During Tibshirani’s discussions with researchers as the site was being developed, he found that 10 to 20 percent of them worried about its repercussions, or just didn’t see the point. “They say you could trash someone’s reputation,” Tibshirani said.

But is that so bad?

“My thought is if someone publishes something really wrong, maybe their reputation shouldn’t be trashed but it should be at least tarnished. Should the facts not come up? I think yes, they should. It’s in everyone’s interest.”