Integrity Initiatives in the Südhof Lab
As science has evolved over the last decades from a paper-based to a web-based endeavor, we have instituted procedures to enable transparent access to all data and tools of the Südhof lab beyond the documentation included in published papers. The overall lab policies are documented in our lab manual. (Click to view the manual)
Current procedures include:
- Full unrestricted access to all published renewable reagents, including mice (via Jackson lab) and plasmids for non-profit research entities
- Access to all experimental protocols
- Public access to software applications (Apps) and algorithms (https://med.stanford.edu/sudhoflab/science-resources/tools.html)
- Deposition of all raw data for published papers in the publicly accessible Stanford Data Repository (as of 2023) or other appropriate databases
- Conversion of all lab notebooks to digital form (as of 2023)
- Publication of reviews with papers as far as allowed by journals
- Facilitating lab visits to disseminate key technologies
In addition, all experiments in the Südhof lab are required to conform to the following guidelines:
- Experimenters are unaware of the identity of samples or animal subjects, i.e., these are anonymized whenever possible
- All experiments are carried out in at least three independent replicates (pseudo-replicates are invalid)
- All major conclusions are based on at least two experimentally distinct methods
- Biological sex is considered an independent variable whenever appropriate
- All experiments are documented in virtual notebooks stored in Stanford box
- Primary data are reviewed in lab meetings by multiple lab members
We hope that these rules will promote transparency and reproducibility.
PubPeer Comments on Südhof Lab Work
PubPeer provides a valuable service to the scientific community in identifying potential problems with published papers, generally focusing on discrepancies in data presented in papers but not on reproducibility or validity of conclusions. These posts have helped us tremendously to identify problems we overlooked and to correct errors we made.
Regrettably, however, PubPeer is non-transparent, censors responses, and uses anonymous commentators with continuously changing aliases. Moreover, PubPeer posts are communicated to journals, journalists, and university administrators without feedback by the accused, and many PubPeer commentaters maintain commercial websites publicizing their findings.
Since 2023, a large number of often similar PubPeer posts allege problems with papers from the Südhof lab. Based on PubPeer posts, publications on sites such as ‘ForBetterScience’ and ‘Spectrum’ and tweets provide criticism of the Südhof lab’s research and the P.I.’s person and family. Furthermore, critical commentators continuously tweet their PubPeer comments and are invited to speak about science rigor and reproducibility based on their PubPeer and social media activities. In particular, Dr. E. Bik in March 2024 used new artificial intelligence tools to analyze >10,000 images in our published papers and preprints, identifying multiple copy-paste errors in multi-image panels. All of the identified errors would not have been detectable without artificial intelligence tools and escaped our attention.
Some PubPeer comentators proposed that our lab committed more than 20 mistakes reflects a systemic problem in science. We believe the opposite is true: it shows how well science works! Intense artificial intelligence-driven scrutiny of ~600 papers with >10,000 images identified only ~20 mistakes. The rate of mistakes (<0.2%) is astonishingly, even improbably, low.
Given the persistent scrutiny of our work on PubPeer, we here aim to improve the accountability and transparency of our lab's work by openly discussing all PubPeer accusations regarding the Südhof lab and describing our uncensored responses and efforts to resolve the issues raised.
Südhof lab PubPeer accusations are summarized in reverse order of a paper's first posting. Accuser names are omitted since posts use continuously changing pseudonyms except when accusers identify themselves. Accusations are numbered according to PubPeer entries, with gaps in numbers caused by our responses or irrelevant comments. We classify the accusation based on their resolution as ‘unfounded’, ‘minor error identified by PubPeer or by the lab (i.e., a mistake without detectable impact on conclusions), ‘major error' identified by PubPeer or the lab (i.e., a mistake with a possible impact on conclusions), and ‘disputed statistical analysis’ (i.e., allegations that a statistical analysis was wrong which we do not agree with). We understand that as scientists, we are often reluctant to admit mistakes. In realizing this reluctance and the fact that we have committed errors, we have tried to concede errors wherever we might have made a mistake. However, we also realized in our continuous arguments with PubPeer accusers that they, too, are loath to admit mistakes in their allegations and repeatedly try to prove their point, frequently with the same arguments over and over again– hence an often extensive discussion.
Total number of cases
- Unfounded allegations: 30
- Minor errors identified by PubPeer: 15
- Minor errors identified by lab: 1
- Major errors identified by PubPeer: 1
- Disputed statistical analysis: 2
23. Paper: Golf, S,R,, Trotter, J,H,, Nakahara, G., and Südhof, T.C. (2023) Astrocytic Neuroligins Are Not Required for Synapse Formation or a Normal Astrocyte Cytoarchitecture. bioRxiv 10:2023.04.10.536254. doi: 10.1101/2023.04.10.536254. Preprint.
PubPeer Weblink: https://pubpeer.com/publications/98784D9AF9B1E8B5B1818E516B5001
Date of first accusation: March 2024
#1
Accusation: As presumably identified by artificial intelligence-driven searches, Dr. Bik alleges that the gephyrin blots for the cortex in figure 3B and 3G look “unexpectedly similar”, implying duplications.
Resolution: A common mistake in identification of blot duplications is to assume that if blots look similar, this is unexpected. If similar samples are run on the same gel apparatuses at the same relative positions in the gel, they will have the same blotting artifacts. This doesn’t mean they are duplicated, but is expected. Moreover, this is a preprint and it is precisely to identify mistakes that we and other are posting preprints.
Classification: Unfounded
***
22. Paper: Seigneur, E., and Südhof, T. C. (2018) Genetic ablation of all cerebellins reveals synapse organizer functions in multiple regions throughout the brain. J. Neurosci. 38, 4774-4790. PMCID: PMC5956990
PubPeer Weblink: https://pubpeer.com/publications/D8EAA6F915EE4008B654738F66ABE6
Date of first accusation: March 2024
#1
Accusation: Dr. Bik’s artificial intelligence software identified two duplicated blots in Figure 2 that contains 72 individual blots and 36 graphs.
Resolution: The allegation is correct. We copy-pasted the wrong blots! Original data are now posted on PubPeer.
Classification: Minor mistake
#2 & #3
Accusation: Dr. Bik’s software discovered two image duplications in Figure 3 that contains 67 individual images and would not be detectable without artificial intelligence examination.
Resolution: Again the allegation is correct. Once we copy-pasted the wrong images we would have been unable to detect it. Original data are now posted on PubPeer.
Classification: Minor mistake
***
21. Paper: Wang, J., Miao, Y., Wicklein, R., Sun, Z., Wang, J., Jude, K.M., Fernandes, R.A., Merrill, S.A., Wernig, M., Garcia, K.C., and Südhof, T.C. (2021) RTN4/NoGo-Receptor Binding to BAI Adhesion-GPCRs Regulates Neuronal Development. Cell 184, 5869-5885. PMCID: PMC8620742
PubPeer Weblink: https://pubpeer.com/publications/5813077CE8B5C29E479FD50C259F77
Date of first accusation: March 2024
#1
Accusation: Dr. Bik identified an image duplication in a supplementary figure containing 36 panels of illustrative images
Resolution: Dr. Bik’s scrutiny correctly identified a duplication that occurred during figure assembly and could only be detected by artificial intelligence because the images are very similar
Classification: Minor mistake
***
20. Papers: Jiang, X., Sando, R., and Südhof, T.C. (2021) Multiple signaling pathways are essential for synapse formation induced by synaptic adhesion molecule. Proc. Natl. Acad. Sci. U.S.A. 118, e2000173118. PMCID: PMC7826368 and Li, J., Xie, Y., Cornelius, S., Jiang, X., Sando, R., Kordon, S., Pan, M., Leon, K., Südhof, T.C., Zhao, M., and Araç, D. (2020) Alternative splicing controls teneurin-latrophilin interaction and synapse specificity by a shape-shifting mechanism. Nature Comm. 11, 2140. PMCID: PMC7195488
PubPeer Weblink: https://pubpeer.com/publications/027E93962D3C5DB86482283739C67D#10
Date of first accusation: March 2024
#9
Accusation: Dr. Bik noticed that a control image in the two papers cited above is the same even though the stated conditions appear different, suggesting fraud
Resolution: Dr. Bik’s artificial intelligence-powered scrutiny of all of our papers correctly identified the same control image for the same type of experiment in two different studies performed at the same time. Moreover, in the Nature Communications paper the control condition is labeled as 'Ctrl' whereas in the PNAS paper it is labeled as 'NPR-mut', which may seem to indicate that different conditions are attributed to the images in the two papers. However, the Methods section clearly states that the 'Ctrl' of the Nature Communications paper is the 'NPR-Mut' that was used in the PNAS paper and thus the conditions are the same. It would have been better if we had indicated that the same control condition was used for the same experiment in two different projects. Five years ago we lived in a less prosecutorial and censorious environment and the lead author did not think of stating this explicitly, which was an oversight.
Classification: Unfounded
***
19. Paper: Burré, J., Sharma, M., and Südhof, T.C. (2012). Systematic Mutagenesis of a-Synuclein Reveals Distinct Sequence Requirements for Physiological and Pathological Activities. J. Neurosci. 32, 15227-15242.
PubPeer Weblink: https://pubpeer.com/publications/0FECC6D2E9498F9876CFCC24D2E03E#8
Date of first accusation: December 2023
#1-#3
Accusations: Dr. Bik & Co noticed that the blots shown in Figure 5A and 6A are assembled from multiple individual blots and accuses the lab of ‘splicing’ blots together.
Resolution: These blots assemble analyses of 26 samples that cannot be examined on a single gel. Thus the samples were run in parallel on multiple gel electrophoresis apparatuses and blots since there are no standard apparatuses that could fit so many samples. No attempt was made to hide this fact which is not 'splicing'. A decade ago, before the current atmosphere of prosecution, it was assumed and accepted that composite blots like the one cited by the accusers would be assembled from multiple individual blots, each of which is clearly recognizable as a separate blot. Nowadays this is considered deception even though there is no attempt to hide it (how could one - it is physically impossible to run so many samples) and the point is made properly.
Classification: Unfounded
#4 & #6
Accusations: Presumably based on artificial intelligence searches, Dr. Bik identified image duplications in two sets of panels in Figure 7D.
Resolution: The accusation is correct. During copy-pasting of representative images, two images were erroneously duplicated and differentially cropped. There are no implications for the actual science.
Classification: Minor mistake
#5
Accusations: Dr. Bik’s alleges a duplication in the 208 individual panels in Figure 9A.
Resolution: Again, Dr. Bik’s artificial intelligence search correctly discovered a duplication among the 208 panels that represents a mistake which likely happened during copy-pasting of representative images. Needless to say, this error has no bearing on the conclusions.
Classification: Minor mistake
***
18. Paper: Lin, P.Y., Chen, L.Y., Jiang, M., Trotter, J.H., Seigneur, E., and Südhof, T.C. (2023) Neurexin-2: An Inhibitory Neurexin That Restricts Excitatory Synapse Formation in the Hippocampus. Sci. Advances 9, eadd8856.
PubPeer Weblink: https://pubpeer.com/publications/C22E0805CB0B55CB7388F488611145
Date of first accusation: March 2024
#1
Accusation: One set of representative images in Figure 4B shows enlarged views of two synapses that cannot be found next to each other in the low-magnification view shown in the paper, and therefore these images represent image manipulations. In order to illustrate this accusation, the accuser drew new boxes into the published figure that are not in the paper.
Resolution: As explained in the figure legend, these are representative synapse images taken from the same experiment but are not adjacent in low-magnification images shown. The low-magnification image is show specifically to illustrate adjacent synapses whereas the high-magnification images illustrate particular features of synapses.
Classification: Unfounded
#4, #6 & #7
Accusation: These accusations by Dr. Bik & Co make the same point as #1: Because we did not explicitly state in the figure legend that the representative images shown were not adjacent to each other in the section, this is scientifically wrong.
Resolution: Journal restrictions on legend sizes make it impossible to explain every detail but there was clearly no intent of hiding the fact that the representative images are just that, representative images, that were selected from a larger set of images. Otherwise we would have indicated this using customary boxes.
Classification: Unfounded
***
17. Paper: Wang, S., DeLeon, C., Sun, W., Quake, S.R., Roth, B.L., and Südhof, T.C. (2024) Alternative Splicing of Latrophilin-3 Controls Synapse Formation. Nature 626, 128-135.
PubPeer Weblink: https://pubpeer.com/publications/A04E94FAF81B5D7EC9E6B1668085EA
Date of first accusation: February 2024
#1
Accusation: The p values in Figure 5 must be wrong because they are the same for Exon 31 (E31) and Exon 32 (E32) conditions.
Resolution: The P values in Fig5 (and also throughout the paper) are identical for E31 and E32 in the same comparison groups because the splicing of E31 and E32 are mutually exclusive. Therefore, each PSI datapoint in E31 plot always has a corresponding datapoint with the value of 100-PSI in E32 plot. Thus, for the same comparison group (e.g. KCl 0hr vs 6hr), the p value for E31 must be equal to E32.
Classification: Unfounded
#2
Accusation: The p values shown in Figure 5 are different from those of the Supplemental Tables and therefor one of them must be wrong.
Resolution: Figure 5 used a t-test as specified in the legend. The supplemental tables use Tukey’s test to calculate pairwise p values after correcting family-wise error as specified again in the legend. In the “Statistics and reproducibility” section we explicitly stated: “Most statistical tests were performed using two-sided t-tests, as indicated. To control for family-wise error during multiple comparisons, two-sided Tukey’s tests were used in parallel and the adjusted P values are summarized in Supplementary Tables 1 and 2, and do not change the conclusions drawn from t-tests in this work.”
Classification: Unfounded
#3
Accusation: Figures 5E and 5F are duplications because they are the same graph rotated 180 degrees.
Resolution: The distribution of Figure 5e and 5f are expected to be precise mirror-images of each other because E31 and E32 are mutually exclusive.
Classification: Unfounded
***
16. Paper: Woerman AL, Stöhr J, Aoyagi A, Rampersaud R, Krejciova Z, Watts JC, Ohyama T, Patel S, Widjaja K, Oehler A, Sanders DW, Diamond MI, Seeley WW, Middleton LT, Gentleman SM, Mordes DA, Südhof TC, Giles K, Prusiner SB. (2015) Propagation of Prions Causing Synucleinopathies in Cultured Cells. Proc. Natl. Acad. Sci. USA 112, E4949-4958.
PubPeer Weblink: https://pubpeer.com/publications/F80D8161AB29FEF18967FAF9A0D228#2
Date of first accusation: December 2023
#1
Accusation: Based on data reconstructions, the statistical significance of panel A of Figure 6 is p = 0.0679 instead of p<0.05 as stated in the paper.
Resolution: The incriminated data are not from the Südhof lab but small difference in reconstructed data points could easily be responsible for the differences in calculated p values, especially since figures are generally constructed from data points by programs and subsequent shuffling of panels introduces inaccuracies in the representations. Given that the alleged p value difference is small, the conclusion that the stated p value is wrong seems to be unjustified.
Classification: Unfounded
***
15. Paper: Jiang, X., Sando, R., and Südhof, T.C. (2021) Multiple signaling pathways are essential for synapse formation induced by synaptic adhesion molecule. Proc. Natl. Acad. Sci. U.S.A. 118, e2000173118. PMCID: PMC7826368
PubPeer Weblink: https://pubpeer.com/publications/027E93962D3C5DB86482283739C67D
Date of first accusation: November 2023
#1
Accusation: The stated statistical significance of the right graph in Figure 5C must be wrong because the error bars overlap
Resolution: Prism software indicates that the two conditions are significantly different, consistent with the fact that overlap of confidence intervals is not a reliable indicator of statistical significance (Schenker, Nathaniel, and Jane F. Gentleman. 2001. “On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals.” The American Statistician 55 (3): 182–86. http://www.jstor.org/stable/2685796.)
Classification: Unfounded
#3
Accusation: Using the error bars as a guide to reconstruct the statistical significance between the two conditions, the two conditions cannot be significantly different
Resolution: We went back to the original data and confirmed that the error bars are SDs, not SEMs, and that the two conditions are statistically significantly different. An error in the figure legend was identified.
Classification: Minor error identified by PubPeer
***
14. Paper: Ho, A., Morishita, W., Hammer, R.E., Malenka, R.C., and Südhof, T.C. (2003) A role for Mints in transmitter release: Mint 1 knockout mice exhibit impaired GABAergic synaptic transmission. Proc. Natl. Acad. Sci. U.S.A. 100, 1409-1414.
PubPeer Weblink: https://pubpeer.com/publications/866AA1811F89014742E1EAEB2BD25D
Date of first accusation: October 2023
#1
Accusation: The day 60 day data point in Figure 3C cannot be statistically significantly different because of the error bars almost overlap.
Resolution: In the incriminated graph only the day 60 day data point is statistically significantly different. This is not a biologically significant result as discussed in the paper, but the significance was nevertheless reported as mandated by publishing rules. Also see #1 under Paper 15 above.
Classification: Unfounded
#2
Accusation: The fact that neurexin protein levels are lower in heterozygous than in homozygous KO mice in Table 1 is suspicious.
Resolution: Protein measurements are notoriously noisy. As a result, small changes in protein levels are not interpretable, especially if they were statistically not significantly different as in this case.
Classification: Unfounded
***
13. Paper: Biederer, T., Cao, X., Südhof, T.C., and Liu, X. (2002) Regulation of APP-dependent transcription complexes by Mints/X11s: Differential functions of Mint isoforms. J. Neurosci. 22, 7340-7351.
PubPeer Weblink: https://pubpeer.com/publications/34D6A8F36DFC9F19D753DCAF6B96FE
Date of first accusation: October 2023
#1
Accusation: The lack of error bars on panel D of Figure 3 raises concerns about the validity of the data
Resolution: Error bars are not visible in Figure 3D because the error bars are too small
Classification: Unfounded
***
12. Paper: Fernandez-Chacon, R., Shin, O.-H., Königstorfer, A., Matos, M.F., Meyer, A.C., Garcia, J., Gerber, S.H., Rizo, J., Südhof, T.C., and Rosenmund, C. (2002) Structure/function analysis of Ca2+-binding to the C2A-domain of synaptotagmin 1. J. Neurosci. 22, 8438-8446.
Weblink: https://pubpeer.com/publications/1FD9572D003130580F6C598C39D9C6
Date of first accusation: September 2023
Accusation: Unknown since comment was removed by ‘moderator’
Resolution: No criticism recorded
Classification: Unfounded
***
11. Paper: Trotter, J.H., Hao, J., Maxeiner, S., Tsetsenis, T., Liu, Z., Zhuang, X., and Südhof ,T.C. (2019) Synaptic Neurexin-1 Assembles into Dynamically Regulated Active Zone Nanoclusters. J. Cell Biology 218, 2677-2698. PMCID: PMC6683742
PubPeer Weblink: https://pubpeer.com/publications/71F24BE796880C3A39AC29501382AB
Date of first accusation: July 2023
#1
Accusation: Control blots in Figure 7C are duplicated
Resolution: Original blots document that the control blots were likely not duplicated even though the blots show similar blotting artifacts as documented, although given the similarity of the blots one might easily conclude that they might be identical. A common error of PubPeer posts is that blots with similar artifacts are alleged to be duplicated, which neglects the fact that samples analyzed on the same blotting apparatus with the same antibodies exhibit similar artifacts even though they are different.
Classification: Unfounded
Postscript: Upon reviewing the primary data, we discovered a blot mixup in the images shown that we corrected with an Erratum but that was not detected on PubPeer
Classification: Minor error identified by the lab
#4
Accusation: One bar in the Figure 9C graph is not labeled as significantly different but looks like it should have been labeled as significantly different
Resolution: PRISM software analysis of the data suggests that it is not significantly different
Classification: Unfounded
***
10. Paper: Burré, J., Sharma, M., Tsetsenis, T., Buchman, V., Etherton, M., and Südhof, T.C. (2010) a-Synuclein Promotes SNARE-Complex Assembly in Vivo and in Vitro. Science 329, 1664-1668.
PubPeer Weblink: https://pubpeer.com/publications/2452F555579F6D021205B875814D82
Date of first accusation: May 2023
#1
Accusation: The blots in Figure 2B, 4B, and 4C are duplicated
Resolution: Original blots demonstrate that the blots are not duplicated. This is the same common error in PubPeer accusations as in paper 11, which alleges blot duplications based on similar artifacts but neglects the fact that similar samples run on the same blotting apparatuses exhibit similar artifacts
Classification: Unfounded
#3
Accusation: Demands higher resolution images of the blots that we presented in PubPeer
Resolution: We feel the original blots we presented on PubPeer are sufficient to demonstrate that the accusation that the blots were duplicated is mistaken.
Classification: Unfounded
#5 & #6
Accusations: Dr. Bik reinforced/repeated #1 and #3 accusations.
Resolution: The first author of the paper now posted the original blots of the paper to document that there is no conclusive evidence that the control blots were duplicated and that these blots exist. Moreover, it should be noted that this is enormous exercise in determining whether or not the accusers or our lab are rights refers to control blots that are clearly without signal in the original blots.
Classification: Unfounded
#8
Accusations: Dr. Bik alleges that some panels in Figure S1E seem to show overlapping images of the pictures
Resolution: Dr. Bik is correct. As the first author explains on PubPeer in a lengthy response, this is expected given the design of the experiment as described in the methods and figure legend.
Classification: Unfounded
***
9. Paper: Patzke, C., Brockmann, M.M., Dai, J., Gan, K.J., Grauel, M.K., Fenske, P., Liu, Y., Acuna, C., Rosenmund, C., and Südhof, T.C. (2019) Neuromodulator Signaling Bidirectionally Controls Vesicle Numbers in Human Synapses. Cell 179, 498-513. PMCID: PMC7159982
PubPeer Weblink: https://pubpeer.com/publications/4AEAAAE084C8DFE9E26107D350B0B5
Date of first accusation: March 2023
#1 & #2
Accusation: The source data files for Figures 6E, 6F, 6K, S6F, S6G, S6Q, S6R, and S7C contain instances of data duplication
Resolution: The accusation is correct. During assembly of the source data files, multiple copy-paste errors were committed that need to be corrected
Classification: Minor mistake identified by PubPeer
***
8. Paper: Lin, P.Y., Chen, L.Y., Zhou, P., Lee, S.H., Trotter, J.H., and Südhof, T.C. (2023) Neurexin-2 restricts synapse numbers and restrains the presynaptic release probability by an alternative splicing-dependent mechanism. PNAS120, e2300363120
PubPeer Weblink: https://pubpeer.com/publications/DAF32F6DB6C166337E5381F769AE52
Date of first accusation: March 2023
Note: This paper elicited a large number of comments because, as we confirmed in our reanalysis of the raw data deposited here (https://purl.stanford.edu/cp231wr9194), the source data contain major irregularities. Moreover, many comments were censored by the ‘Moderator’ who decides what is allowable on PubPeer.
#1 - #4, #7, #11, and #13-#16
Accusation: The source data files for Figures 2-6 contain extensive data duplications
Resolution: The accusation is correct. The data duplications initially appeared to be a minor mistake due to extensive copy-paste errors, but the paper’s 1st author then posted a corrected set of replacement data that were not previously seen by the lab and did not appear to be from the Südhof lab, which led to a reanalysis of the raw data by the lab.
Classification: Unresolved
#18 - #21, #23-#26, #28, #30, #31, #33-#35, #37, #38, #40, #42, #44, and #46
Accusation: The unpublished replacement data posted by the 1st author contain irregularities suggesting possible falsification
Resolution: The accusation is correct. The replacement data were not reviewed or endorsed by the Südhof lab prior to being posted by the 1st author.
Classification: Unresolved. The fact that questionable replacement data were posted without input from the Südhof lab led to an ongoing re-analysis and replication of the primary raw data that were deposited publicly here: https://purl.stanford.edu/cp231wr9194.
#64-#67 and #69
Accusation: The posted raw data may or may not contain technical issues
Resolution: The discussion here was largely based on an incomplete understanding of electrophysiological methods as one of the comments points out
Classification: Unresolved
#68
Accusation: The review of the paper was flawed because both reviewers are alumni of the Südhof lab
Resolution: The accusation is incorrect since Prof. Josh Huang was never associated with, or collaborated with, the Südhof lab, although Prof. Katsuhiko Tabuchi was in the Südhof lab 20 years ago but has since established a successful large independent research program.
Classification: Unfounded
Postscriptum: The Südhof lab posted as response #70 the following statement:
“My lab has surveyed the many criticisms of this paper and reanalyzed part of the raw data. We observed multiple errors. The ones we know of, both discovered by us and raised by others, I’m listing here. Our preliminary conclusions are:
a. The published source data and the raw data associated with the paper are often inconsistent.
b. There are extensive raw data falling into two categories: raw data that were labeled as used for the paper and raw data that were not used from analogous experiments without a clear difference in quality or rationale.
c. The limited reanalysis of anonymized data confirms the conclusions of the paper, although we have analyzed only a small part of the data and the conclusions are not always statistically significant for all parameters. The absolute values of the mean parameters differ as would be expected when raw data are analyzed independently. Needless to say that does not confirm the paper as we also need to relate the data to the original lab notebook.
d. As regards comment #68, although Prof. Katsuhiko Tabuchi was a fellow in my lab 25 years ago, there has been no connection to my lab since then, and Prof. Josh Huang would indeed be surprised to learn that he was supposed to have been a fellow in my lab.
Overall, we concur that there are major inconsistencies with the source data of this paper compared to the existing raw data, which we are puzzled by. Note that the raw data, unlike blots or images, cannot be manipulated and contain identifiable metadata. Our current preliminary conclusion thus is that, consistent with some of the comments, the source data of this paper contain major flaws that cannot be explained by copy-paste mistakes as I had mistakenly surmised earlier but must have another origin.”
An ’Editorial Expression of Concern’ was published by PNAS that was immediately re-published as comment #71 on PubPeer and in accusatory blogs on ForBetterScience and Spectrum websites.
#73 and #74
Accusation: The mean value in Figure 1B does not fit the datapoints associated with it, suggesting that the mean value is wrong
Resolution: The accusation is correct that the mean control value does not fit the datapoints but incorrect in concluding that the mean value is wrong. Replotting of the raw data in the source file clearly shows that the datapoints were accidentally shifted during construction of the figure, probably because in the adobe illustrator software independent image objects are often linked and movement of an object can inadvertently cause movement of other linked objects.
Classification: Minor mistake identified by PubPeer
Update. We retracted the P.Y. Lin et al. paper after our independent analysis of the raw data in 2024 (https://www.pnas.org/doi/10.1073/pnas.2403021121) with the following statement: “We wish to retract the paper because re-analysis of the original raw data for Figs. 2, 4, and 6 (https://purl.stanford.edu/cp231wr9194) revealed that, although our analyses of the original data are supportive of the conclusions of the paper, unresolvable differences exist between these raw data and the published data source file that cannot be corrected by a simple erratum. In addition, the data source file contained copy-paste errors, and Fig. 1 included shifted data points that occurred during figure drafting. We thank Dr. Daniel Matus of Stanford University for his independent analysis of the primary raw data."
***
7. Paper: Dai, J., Liakath-Ali, K., Golf, S., and Südhof, T.C. (2022) Distinct Neurexin-Cerebellin Complexes Control AMPA- and NMDA-Receptor Responses in a Circuit-Dependent Manner. E-Life 11, e78649. PMCID: PMC9586558
PubPeer Weblink: https://pubpeer.com/publications/68D8490A4754CE00F936214C3931F2
Date of first accusation: March 2023
#1 and #4
Accusation: The source data file of the paper contains duplicated values for several cells
Resolution: The accusation is correct. The duplications are copy-paste errors that were corrected with an Erratum in the journal.
Classification: Minor mistake identified by PubPeer
***
6. Paper: Zhang, X., Lin, P.Y., Liakath-Ali, K, and Südhof, T.C. (2022) Teneurins Assemble into Presynaptic Nanoclusters that Promote Synapse Formation via Postsynaptic Non-Teneurin Ligands. Nature Comm. 13, 2297. PMCID: PMC9050732
PubPeer Weblink: https://pubpeer.com/publications/EC9A138F8BFAAE1F2FB803106703AB
Date of first accusation: March 2023
#1
Accusation: Two columns in the source data file for Figure 8 are duplicated
Resolution: The accusation is correct. The columns were accidentally duplicated in a copy-paste error during assembly of the file for sharing after figures were drafted. The error was corrected with an erratum.
Classification: Minor mistake identified by PubPeer
***
5. Paper: Dai, J., Patzke, C., Liakath-Ali, K., Seigneur, E., and Südhof, T.C. (2021) GluD1, A signal transduction machine disguised as an ionotropic receptor. Nature 595, 261-265. PMCID: PMC8776294
PubPeer Weblink: https://pubpeer.com/publications/0B87E141DC8DFFF4826A9250A94BAD
Date of first accusation: March 2023
#1
Accusation: The PPR data in the source file are missing
Resolution: The corresponding data were mislabeled as belonging to ‘1f’ instead of ‘1k’ in the source data file due to a copy-paste mistake. This mistake is being addressed in a ‘Correction’ in the journal.
Classification: Minor mistake identified by PubPeer
#2 and #4
Accusation: The source file contains duplicated values for multiple isolated cells
Resolution: Correct. We introduced several copy-paste mistakes during transfer of data from experimental logs to the source file, as occurs when the ‘copy’ key is not pressed sufficiently. This mistake is being addressed in a ‘Correction’ in the journal.
Classification: Minor mistake identified by PubPeer
#3, #9, and #11
Accusation: Some of the full-sized blots in Suppl. Figure 1b and 1c do not correspond to the cropped or quantified data in the Extended Data figure.
Resolution: Correct. Reassessment of original blots identified two mistakes. In Suppl. Figure 1b one blot was mislabeled and not all blots were included. In Suppl. Figure 1c a different blot of the same experiment with identical results was shown. Again, these mistakes are being addressed in a ‘Correction’ in the journal.
Classification: Minor mistake identified by PubPeer
#12-#16
Accusation: The data on the effect of a mutation on a function should not have been tested by t-tests
Resolution: T-tests are the standard test for manipulations that contains only a single independent variable and only compare that variable to the control, but not among samples. The graph reports a comparison of multiple single tests to the same shared controls, and could be plotted as multiple graphs consisting of a control and test condition. We agree, however, that the question of the appropriate statistical test can be contentious especially if one focuses less on the biological experiment and more on the graph format.
Classification: Disputed statistical analysis
***
4. Paper: Sclip A, and Südhof, T.C. (2020) LAR receptor phospho-tyrosine phosphatases regulate NMDA-receptor responses. E-Life 9, pii: e53406. PMCID: PMC6984820
PubPeer Weblink: https://pubpeer.com/publications/613AFEC1A22C0725BB6D5A9E5CFE76
Date of first accusation: February 2023
#1
Accusation: Alleged duplication of blots because (i) quantifications of blots were made on the basis of different ‘n’s’ for different proteins but (ii) only a single Tuji control blot is shown for the quantifications.
Resolution: Ad (i), the ‘n’s’ (number of replicates) differ between experiments because each protein quantification is performed separately with a different antibody. Different numbers of repeat experiments are performed for various proteins because the noise levels differ between proteins, with the noise level in turn being determined by a protein’s abundance and the quality of an antibody. Ad (ii), a single sample control blot is shown for each set of experiments because illustrating samples of the Tuji control blots separately for each protein seems superfluous and adds no information
Classification: Unfounded
***
3. Paper: Dai, J., Aoto, J., and Südhof, T.C. (2019) Alternative Splicing of Presynaptic Neurexins Differentially Controls Postsynaptic NMDA- and AMPA-Receptor Responses. Neuron 102, 993-1008. PMCID: PMC6554035
PubPeer Weblink: https://pubpeer.com/publications/568B4CF8B40A979424B7F343F3B061
Date of first accusation: January 2023
#1
Accusation: The y-axis labels of Figure 6B and 6C are incorrect
Resolution: Correct - the y-axes were mislabeled
Classification: Minor mistake identified by PubPeer
#3-#11, #22, #23
Accusation: The source data for Figure 2D, 4A, 5E, 6A, 6B, 8, S2, S4, and S8 contain duplications and possible calculation errors
Resolution: Correct - the source data contain multiple isolated errors produced by copy-paste mistakes that also lead to errors in downstream calculations but have no discernable effect on figures or conclusions.
Classification: Minor mistake identified by PubPeer
#25-#28 and #30
Accusation: The statistical tests used for experiments examining 3 experimental groups – derived from littermate SS4+ and SS4- mice vs. unrelated WT samples – are generally incorrect
Resolution: The accusers express their view that the three experimental groups should be treated as equivalent in pairwise comparisons whereas biologically they are not equivalent. The 3 experimental groups comprise a genetically distinct WT sample expressing both SS4+ and SS4- variants of neurexins and two genetically identical samples that express either only SS4+ or SS4- variants of neurexins. Thus, the statistics are more complex than a simple 2-way ANOVA with a post-hoc correction since the comparison of the two genetically identical samples between each other is inherently different from their comparison with the WT sample. Given this experimental configuration, we believe our statistical approach may be the most appropriate, but it is possible that a non-traditional test that accounts for the experimental non-equivalency might be even better.
Classification: Disputed statistical analysis
#33-#37
Accusation: Dr. Bik states that the Correction we published on this paper to rectify the labeling mistakes and copy-paste errors in our paper represent a ‘Mega Correction’ and another accuser criticized that we did not address the accuser’s concerns about statistics.
Resolution: We feel that our correction of multiple copy-paste errors in supplementary tables and of mislabeled graphs in the supplement is not a ‘Mega Correction’, although we deeply regret these errors and mistakes. These mistakes had no consequences for the study, either its data or its conclusions. We started to publish primary data before artificial intelligence-powered software was available to detect mistakes, such as copy-pasted numbers or images, that we in the pre-artificial intelligence era overlooked, which clearly should not have happened. The impact of our 'Mega Corrections' is purely procedural and has no consequences for the actual science.
Classification: Difficult to classify since we already stated that we believe that this is a minor correction and that we do not agree with the statistical criticism
***
2. Paper: Sclip, A., Bacaj, T., Giam, L., and Südhof, T.C. (2016) Extended synaptotagmin (ESyt) triple knock-out mice are viable and fertile without obvious endoplasmic reticulum dysfunction. PlosONE 11, e0158295.
PubPeer Weblink: https://pubpeer.com/publications/FBC43D21E0E903A65AF81CD8D1CAF1
Date of first accusation: August 2022
#1
Accusation: Alleged duplication of blots in Figure 3 and S1.
Resolution: It is correct that the same tubulin and actin blots were used as loading controls for experiments that were performed at the same time. This is not a duplication of blots but the use of the same controls for an experiment that was performed at the same time, and we should have noted this in the legends. An Erratum to document this detail is being prepared.
Classification: Minor mistake identified by PubPeer
#3
Accusation: A correction was made in the published paper without notification
Resolution: This is incorrect – no correction was made
Classification: Unfounded
#5
Accusation: Claims that the data are no longer in the PlosONE website
Resolution: A screenshot from the PlosONE website shows that the data are clearly still there.
Classification: Unfounded
#7
Accusation: The Munc18 and Syt1 blots are identical, i.e. represent duplications
Resolution: Original blots of the experiments as now shown on PubPeer demonstrate that the Munc18 and Syt1 blots were derived from the same experiments run on the same gels but probed with species-specific distinct antibodies. Since these proteins have similar sizes, the shape of their bands is nearly identical, creating an appearance of duplication. However, based on direct communication of the accusers with PlosONE ‘Ethics’, the PlosONE ‘Ethics’ administrators demand that we amend the blots to show samples from the same original publicly deposited blots that make the bands look different, which will be published as a ‘correction’.
Classification: Unfounded
#10
Accusation: Maintains that he/she/they cannot see the blots properly
Resolution: Original blots were publicly deposited (https://purl.stanford.edu/vq040hz0549)
Classification: Unfounded
#11
Accusation: Dr. Bik states “the blot provided in #2 appears to contain munc18 bands in green and syt1 bands in red, just under the munc18 bands, while the published panels show both bands separately (no double bands) and both in red. So it appears you either did not provide the originals, or the published panels have been color-converted with double bands removed.”
Resolution: We have already published a ‘Correction’ that explains better the various blots with inclusion of full-length original blots and boxed areas, and all original blots have been posted on PubPeer and are also publicly available at the published URL as high-resolution images. However, the accusations that "So it appears you either did not provide the originals, or the published panels have been color-converted with double bands removed" are incorrect. In the incriminated blot, two different species-specific antibody signals were monitored on the same blot (for Munc18 and synaptotagmin that have almost identical molecular weights) in different optical channels. Thus, no double band has been removed, nor are colors converted because every color represents an arbitrary assignment of a digital signal. We chose to show the Munc18 and synaptotagmin blots in the same false color in our paper because we thought that this would make it easier to compare the blots, but this has clearly confused people who are not familiar with this type of experiments.
Classification: Unfounded
***
1. Paper: Yi, F., Danko, T., Botelho SC, Patzke C, Pak C, Wernig, M., and Südhof, T.C. (2016) Autism-associated SHANK3 haploinsufficiency causes Ih channelopathy in human neurons. Science 352, aaf2669. PMCID: PMC4901875
PubPeer Weblink: https://pubpeer.com/publications/CBAA10B0FBF31CC41E9B7D56C8B0C3
Date of first accusation: April 2018
#1
Accusation: Since the resting membrane potential of the human neurons in the experiments is at approximately -40 mV, the neurons must be leaky and sick and it is therefore not possible to draw any conclusions from these neurons
Resolution: All human neurons produced from stem cells are immature and exhibit a decreased resting potential, even though they are not leaky or sick, exhibit robust active and passive membrane properties, and form fully functional synapses
Classification: Unfounded
#3
Accusation: How can different clones in different sets of experiments produce very similar readouts given that evoked EPSC amplitudes are dependent on intensity of stimulation
Resolution: The electrophysiological approaches used here were validated previously in a number of papers and shown to be highly reproducible across experiments (e.g., see Zhang, Y., Pak, C.H., Han, Y., Ahlenius, H., Zhang, Z., Chanda, S., Marro, S., Xu, W., Yang, N., Patzke, C., Chen, L., Wernig, M., and Südhof, T.C. (2013) Rapid Single-Step Induction of Functional Neurons from Human Pluripotent Stem Cells. Neuron 78, 785-798. PMCID: PMC3751803; Pak, C., Danko, T., Mirabella, V.R., Wang, J., Liu, Y., Vangipuram, M., Grieder, S., Zhang, X., Ward, T., Huang, Y.W.A., Jin, K., Dexheimer, P., Bardes, E., Mittelpunkt, A., Ma, J., McMachlan, M., Moore, J.C., Qu, P., Purmann, C., Dage, J.L., Swanson, B.J., Urban A.E., Aronow, B.J., Pang, Z.P., Levinson, D.F., Wernig, M., and Südhof, T.C. (2021) Cross-Platform Validation of Neurotransmitter Release Impairments in Schizophrenia Patient-Derived NRXN1-Mutant Neurons. Proc. Natl. Acad. Sci. U.S.A. 118, e2025598118. PMCID: PMC8179243)
Classification: Unfounded
#4
Accusation: The electrophysiological experiments are unreliable (e.g., “Anyone knows that hyperpolarizing MP from -40 to -70 takes heroic current magnitude” and “Not all cells should be active at rest, even not ~60 %”)
Resolution: The electrophysiological approaches used here are consistent with a large number of previous studies by multiple independent experimenters not only the Südhof lab but also in others, and the data shown in the paper are internally fully consistent
Classification: Unfounded
#5
Accusation: Mean amplitudes and intervals do not correlate with the 'representative traces'
Resolution: Electrophysiological measurements vary between cells and representative traces are only an illustration of the results, they do not depict an average of traces
Classification: Unfounded
******