Guest post by Xi-Nian Zuo, Project Coordinator and Co-Founder of Consortium for Reliability and Reproducibility (CoRR), Professor of Psychology and Director of the Magnetic Resonance Imaging Research Center in the Institute of Psychology at Chinese Academy of Sciences, China.

Xi-Nian Zuo

About a decade ago (2006), as a PhD student graduating from the School of Mathematics at Beijing Normal University, I stepped into the field of neuroimaging of the human brain by way of a short job interview offered by Dr. Yu-Feng Zang, my postdoc mentor in China. The most important thing that I learned and developed during my post doc training was how to question a study, an indication likely of my somewhat different background (mathematics versus brain sciences). Probability and statistics became my major tools in bridging new learning experiences with my existing knowledge, pushing me to further pursue research training offered by Dr. Michael Peter Milham at New York University. Ongoing work in his laboratory really interested me, particularly test-retest reliability of resting-state functional connectivity¹, the first study of test-retest reliability in the nascent field of functional connectivity. However, an obvious limitation existed to that study, and a series of test-retest reliability studies I carried out subsequently²; the small sample size. This directly motivated me to seek and build up a truly big data set for test-retest reliability in connectomics.

The idea of initializing the Consortium for Reliability and Reproducibility (CoRR) came from a conversation between Dr. Milham and I before I went back to China. I started work in the Institute of Psychology, Chinese Academy of Sciences on December 1, 2010, and after 5 years of preparation of the CoRR data release, the dream of CoRR came true and all the data were released openly to the public and published as a data descriptor in Nature Research’s Scientific Data³. During the publication process Dr. Andrew L. Hufton, the managing editor, did excellent work in making CoRR significantly recognized and improved through the peer-review process, and put CoRR at the center of the ‘Human Brain MRI Reproducibility’ collection. This international effort aims to advance reproducible and standardized human connectomics by sharing longitudinal MRI data, a goal yet to be achieved by modern connectomics^4,5.

Beyond underscoring the importance of sharing CoRR data (see ‘Data Matters’ from Dr. Milham), I summarize here the influences of CoRR on the community based upon the 43 citations it has received since publication as of August 2016, according to Google Scholar. First, CoRR has increased the recognition of a cultural shift toward sharing big data in human neuroscience and clinical applications^6-8. Second, CoRR has highlighted to both research and clinical communities the need for large test-retest datasets to achieve reliable imaging measures for the production of replicable associations between genetic variation, environmental experience, and psychopathological symptomatology, reinforcing the fact that the validity of a measure can never be greater than its reliability^9-11. Finally, the role of CoRR as a testbed in the development of novel connectomic methods is emerging. One popular CoRR dataset for such a practice is the HNU1 dataset¹², which has been applied to establish a continuous model of cortical connectivity¹³ and a 4D filter of spatial-temporal image segmentation¹⁴ as well as to optimize the brain registration for multi-site meta-analysis¹⁵ and scan duration of resting-state brain imaging for functional connectivity density¹⁶. To further call on the community to utilize CoRR datasets, two efforts have been made recently, including the launch of a Research Topic ‘Reliability and Reproducibility in Functional Connectomics’ in Frontiers journals, and a similar CoRR-theme BrainHack conference.

Of great importance is the transformation of the neuroimaging and connectomics fields with standardized processes catalyzed by open access and reproducibility. For example, the Organization for Human Brain Mapping (OHBM) has announced the ‘Replication Award’ and published a report on Best Practice in Data Analysis and Sharing. One of the leading forces in improving transparency and reproducibility in neuroimaging research is the Center for Reproducible Neuroscience at Stanford University led by Dr. Russ Poldrack¹⁷. Meanwhile, Scientific Data has already demonstrated its important role in publishing big benchmark data for reliability and reproducibility such as CoRR and field standards for reproducible sciences, such as BIDS (Brain Imaging Data Structure)¹⁸.

The strong engagement of the human connectomics community represents a key component in developing reproducible and standardized human connectomics. Replication studies should be encouraged by journal editors and publishers. I suggest that the use of CoRR is an important part of the ongoing quest for reproducibility and standardization.

References

1. Shehzad, Z., et al. The resting brain: unconstrained yet reliable. Cereb. Cortex 19, 2209-2229 (2009).
2. Zuo, X. N. & Xing, X. X. Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: a systems neuroscience perspective. Neurosci. Biobehav. Rev. 45, 100-118 (2014).
3. Zuo, X. N., et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1:140049 (2014).
4. Sporns O, Tononi G & Kötter R. The human connectome: A structural description of the human brain. PLoS Comput. Biol. 1:e42 (2005).
5. Pestilli, F. Test-retest measurements and digital validation for in vivo neuroscience. Sci. Data 2:140057 (2015).
6. Holmes, A. J. & Yeo, B. T. From phenotypic chaos to neurobiological order. Nat. Neurosci. 18, 1532-1534 (2015).
7. Sporns, O. Enabling discovery science in human connectomics. Sci. Bull. 60, 139 (2015).
8. Mišić, B. & Sporns, O. From regions to connections and networks: new bridges between brain and behavior. Curr. Opin. Neurobiol. 40, 1-7 (2016).
9. Kraemer, H. C. The reliability of clinical diagnoses: state of the art. Annu. Rev. Clin. Psychol. 10, 111-130 (2014).
10. Matthews, P. M. & Hampshire, A. Clinical concepts emerging from fMRI functional connectomics. Neuron 91, 511-528 (2016).
11. Bogdan, R., Pagliaccio, D., Baranger, D. A. & Hariri, A. R. Genetic moderation of stress effects on corticolimbic circuitry. Neuropsychopharmacology 41, 275-296 (2016).
12. Chen, B., et al. Individual variability and test-retest reliability revealed by ten repeated resting-state brain scans over one month. PLoS One 10:e0144963 (2015).
13. Moyer, D., Gutman, B., Jahanshad, N., Faskowitz, J. & Thompson, P. M. A continuous model of cortical connectivity. The 19th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2016), October 17th to 21st, 2016 in Athens, Greece.
14. Roy, S., Carass, A., Pacheco, J., Bilgel, M., Resnick, S. M., Prince, J. L., Pham, D. L. Temporal filtering of longitudinal brain magnetic resonance images for consistent segmentation. Neuroimage Clin. 11, 264-275 (2016).
15. Faskowitz, J., et al. Comparison of template registration methods for multi-site meta-analysis of brain morphometry. Proc. SPIE 9788, Medical Imaging 2016: Biomedical Applications in Molecular, Structural, and Functional Imaging, 978822 (March 29, 2016); doi:10.1117/12.2217370.
16. Thomasi, D. G., Shokri-Kojori, E. & Volkow, N. D. Temporal evolution of brain functional connectivity metrics: Could 7 min of rest be enough? Cereb. Cortex. in press, doi: 10.1093/cercor/bhw227 (2016).
17. Gorgolewski, K. J. & Poldrack, R. A. A practical guide for improving transparency and reproducibility in neuroimaging research. PLoS Biol. 14:e1002506 (2016).
18. Gorgolewski KJ, et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data 3:160044 (2016).