Ritchie, S. J. et al. Sex differences in the adult human brain: evidence from 5216 UK biobank participants. Cereb. Cortex 28, 2959–2975 (2018).Article
PubMed
PubMed Central
Google Scholar
Aycheh, H. M. et al. Biological brain age prediction using cortical thickness data: a large scale cohort study. Front. Aging Neurosci. 10, 252 (2018).Article
PubMed
PubMed Central
Google Scholar
Hidese, S. et al. Association of obesity with cognitive function and brain structure in patients with major depressive disorder. J. Affect. Disord. 225, 188–194 (2018).Article
PubMed
Google Scholar
Ledig, C., Schuh, A., Guerrero, R., Heckemann, R. A. & Rueckert, D. Structural brain imaging in Alzheimer’s disease and mild cognitive impairment: biomarker analysis and shared morphometry database. Sci. Rep. 8, 11258 (2018).Article
ADS
PubMed
PubMed Central
Google Scholar
Tolan, E. & Isik, Z. Graph theory based classification of brain connectivity network for autism spectrum disorder. in Bioinformatics and Biomedical Engineering 520–530 (Springer International Publishing, 2018).Zhu, X., Du, X., Kerich, M., Lohoff, F. W. & Momenan, R. Random forest based classification of alcohol dependence patients and healthy controls using resting state MRI. Neurosci. Lett. 676, 27–33 (2018).Article
CAS
PubMed
PubMed Central
Google Scholar
Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).Article
Google Scholar
Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).Article
CAS
PubMed
Google Scholar
Glasser, M. F. et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage 80, 105–124 (2013).Article
PubMed
Google Scholar
Volkow, N. D. et al. The conception of the ABCD study: from substance use to a broad NIH collaboration. Dev. Cogn. Neurosci. 32, 4–7 (2018).Article
PubMed
Google Scholar
Markiewicz, C. J. et al. OpenNeuro: an open resource for sharing of neuroimaging data. bioRxiv 2021.06.28.450168 https://doi.org/10.1101/2021.06.28.450168 (2021).Rougier, N. P. et al. Sustainable computational science: the ReScience initiative. PeerJ Comput Sci. 3, e142 (2017).Article
PubMed
PubMed Central
Google Scholar
Plesser, H. E. Reproducibility vs. replicability: a brief history of a confused terminology. Front. Neuroinform. 11, 76 (2017).Article
PubMed
Google Scholar
Charter, R. A. Methodological commentary: effect of measurement error on tests of statistical significance. J. Clin. Exp. Neuropsychol. 19, 458–462 (1997).Article
CAS
PubMed
Google Scholar
Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638 (1979).Article
Google Scholar
Thompson, W. H., Wright, J., Bissett, P. G. & Poldrack, R. A. Meta-research: dataset decay and the problem of sequential analyses on open datasets. Elife 9, e53498 (2020).Article
PubMed
PubMed Central
Google Scholar
Paus, T. Population neuroscience: why and how. Hum. Brain Mapp. 31, 891–903 (2010).Article
PubMed
PubMed Central
Google Scholar
Nielsen, M., Haun, D., Kärtner, J. & Legare, C. H. The persistent sampling bias in developmental psychology: a call to action. J. Exp. Child Psychol. 162, 31–38 (2017).Article
PubMed
PubMed Central
Google Scholar
Charpentier, C. J. et al. How representative are neuroimaging samples? Large-scale evidence for trait anxiety differences between fMRI and behaviour-only research participants. Soc. Cogn. Affect. Neurosci. 16, 1057–1070 (2021).Article
PubMed
PubMed Central
Google Scholar
Li, J. et al. Cross-Ethnicity/race Generalization Failure of RSFC-Based Behavioral Prediction and Potential Consequences. https://juser.fz-juelich.de/record/910372 (2022).Guyatt, G. H. et al. Users’ guides to the medical literature: IX. A method for grading health care recommendations. JAMA 274, 1800–1804 (1995).Article
CAS
PubMed
Google Scholar
Jennings, R. G. & Van Horn, J. D. Publication bias in neuroimaging research: implications for meta-analyses. Neuroinformatics 10, 67–80 (2012).Article
PubMed
Google Scholar
Traut, N. et al. Cerebellar volume in autism: literature meta-analysis and analysis of the autism brain imaging data exchange cohort. Biol. Psychiatry 83, 579–588 (2018).Article
PubMed
Google Scholar
Steegen, S., Tuerlinckx, F., Gelman, A. & Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11, 702–712 (2016).Article
PubMed
Google Scholar
Guest, O. & Martin, A. E. How computational modeling can force theory building in psychological science. Perspect. Psychol. Sci. 16, 789–802 (2021).Glatard, T. et al. Reproducibility of neuroimaging analyses across operating systems. Front. Neuroinform. 9, 12 (2015).Article
PubMed
Google Scholar
Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).Article
CAS
PubMed
PubMed Central
Google Scholar
Van Essen, D. C. et al. The WU-Minn human connectome project: an overview. Neuroimage 80, 62–79 (2013).Article
PubMed
Google Scholar
Jack, C. R. Jr et al. The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27, 685–691 (2008).Article
PubMed
PubMed Central
Google Scholar
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).Article
CAS
PubMed
Google Scholar
Regier, D. A. et al. DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses. Am. J. Psychiatry 170, 59–70 (2013).Article
PubMed
Google Scholar
Nikolaidis, A. et al. Suboptimal phenotypic reliability impedes reproducible human neuroscience. bioRxiv 2022.07.22.501193 https://doi.org/10.1101/2022.07.22.501193 (2022).Gell, M. et al. The burden of reliability: how measurement noise limits brain-behaviour predictions. bioRxiv 2023.02.09.527898 https://doi.org/10.1101/2023.02.09.527898 (2023).Falk, E. B. et al. What is a representative brain? Neuroscience meets population science. Proc. Natl. Acad. Sci. USA 110, 17615–17622 (2013).Article
ADS
CAS
PubMed
PubMed Central
Google Scholar
Varoquaux, G. Cross-validation failure: small sample sizes lead to large error bars. Neuroimage 180, 68–77 (2018).Article
PubMed
Google Scholar
Silberzahn, R. et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. 1, 337–356 (2018).Article
Google Scholar
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).Article
ADS
CAS
PubMed
PubMed Central
Google Scholar
Schilling, K. G. et al. Tractography dissection variability: what happens when 42 groups dissect 14 white matter bundles on the same dataset? Neuroimage 243, 118502 (2021).Article
PubMed
Google Scholar
Fornito, A., Zalesky, A. & Bullmore, E. T. Network scaling effects in graph analytic studies of human resting-state FMRI data. Front. Syst. Neurosci. 4, 22 (2010).PubMed
PubMed Central
Google Scholar
Li, X. et al. Moving beyond processing and analysis-related variation in neuroscience. bioRxiv 2021.12.01.470790 https://doi.org/10.1101/2021.12.01.470790 (2021).Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).Article
CAS
PubMed
PubMed Central
Google Scholar
Bowring, A., Maumet, C. & Nichols, T. E. Exploring the impact of analysis software on task fMRI results. Hum. Brain Mapp. 40, 3362–3384 (2019).Article
PubMed
PubMed Central
Google Scholar
Bowring, A., Nichols, T. E. & Maumet, C. Isolating the sources of pipeline-variability in group-level task-fMRI results. Hum. Brain Mapp. https://doi.org/10.1002/hbm.25713 (2021).Bhagwat, N. et al. Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses. Gigascience 10, giaa155 (2021).Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R. & Collins, D. L. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage Suppl. 1, S102 (2009).Article
Google Scholar
Lancaster, J. L. et al. Bias between MNI and Talairach coordinates analyzed using the ICBM-152 brain template. Hum. Brain Mapp. 28, 1194–1205 (2007).Article
PubMed
PubMed Central
Google Scholar
Bouthillier, X. et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems 3, (2021).Parker, D. S. Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic (University of California (Los Angeles). Computer Science Department, 1997).Skare, S., Hedehus, M., Moseley, M. E. & Li, T. Q. Condition number as a measure of noise performance of diffusion tensor data acquisition schemes with MRI. J. Magn. Reson. 147, 340–352 (2000).Article
ADS
CAS
PubMed
Google Scholar
Kiar, G. et al. Comparing perturbation models for evaluating stability of neuroimaging pipelines. Int. J. High Perform. Comput. Appl. 34, 491–501 (2020).Kiar, G. et al. Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. bioRxiv 2020.10.15.341495 https://doi.org/10.1101/2020.10.15.341495 (2021).Di Martino, A. et al. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol. Psychiatry 19, 659–667 (2014).Article
PubMed
Google Scholar
Gorgolewski, K. J. et al. NeuroVault.org: a web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Front. Neuroinform. 9, 8 (2015).Article
PubMed
PubMed Central
Google Scholar
Dockès, J. et al. NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife 9, e53385 (2020).Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8, 665–670 (2011).Krstajic, D., Buturovic, L. J., Leahy, D. E. & Thomas, S. Cross-validation pitfalls when selecting and assessing regression and classification models. J. Cheminform. 6, 10 (2014).Article
PubMed
PubMed Central
Google Scholar
Traut, N. et al. Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery. Neuroimage 255, 119171 (2022).Article
PubMed
Google Scholar
Olivetti, E., Greiner, S. & Avesani, P. ADHD diagnosis from multiple data sources with batch effects. Front. Syst. Neurosci. 6, 70 (2012).Article
PubMed
PubMed Central
Google Scholar
Gau, R. et al. Brainhack: developing a culture of open, inclusive, community-driven neuroscience. Neuron 109, 1769–1775 (2021).Article
CAS
PubMed
PubMed Central
Google Scholar
Craddock, C. et al. Towards automated analysis of connectomes: the configurable pipeline for the analysis of connectomes (c-pac). Front. Neuroinform. 42, 10–3389 (2013).
Google Scholar
Glatard, T. et al. Boutiques: a flexible framework to integrate command-line applications in computing platforms. Gigascience 7, giy016 (2018).Amstutz, P. et al. Common Workflow Language, draft 3. https://doi.org/10.6084/m9.figshare.3115156.v1 (2016).Halchenko, Y. et al. DataLad: distributed system for joint management of code, data, and their relationship. J. Open Source Softw. 6, 3262 (2021).Article
ADS
PubMed
Google Scholar
Kiar, G. et al. Verificarlo/fuzzy: Fuzzy v0.5.0. https://doi.org/10.5281/zenodo.5027708 (2021).Chatelain, Y. et al. A numerical variability approach to results stability tests and its application to neuroimaging. arXiv [physics.med-ph] (2023).Gau, R. et al. COBIDAS checklist. Preprint at https://doi.org/10.17605/OSF.IO/ANVQY (2019).Salkind, N. J. (Ed.) Encyclopedia of research design. SAGE Publications, Inc., https://doi.org/10.4135/9781412961288 (2010).Gueorguieva, R. & Krystal, J. H. Move over ANOVA: progress in analyzing repeated-measures data and its reflection in papers published in the archives of general psychiatry. Arch. Gen. Psychiatry 61, 310–317 (2004).Article
PubMed
Google Scholar
Simmons, J., Nelson, L. & Simonsohn, U. Pre‐registration: why and how. J. Consum. Psychol. 31, 151–162 (2021).Article
Google Scholar
Baribault, B. et al. Metastudies for robust tests of theory. Proc. Natl. Acad. Sci. USA 115, 2607–2612 (2018).Article
ADS
CAS
PubMed
PubMed Central
Google Scholar
Brandmaier, A. M. et al. Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED). Elife 7, e35718 (2018).Kiar, G., Chatelain, Y., Salari, A., Evans, A. C. & Glatard, T. Data augmentation through Monte Carlo arithmetic leads to more generalizable classification in connectomics. Neurons Behav. Data Theory https://doi.org/10.51628/001c.28328 (2021).Bellec, P., Rosa-Neto, P., Benali, H. & Evans, A. C. Multi-level bootstrap analysis of stable clusters (BASC) in resting-state fMRI. NeuroImage 47, S123 (2009).Nikolaidis, A. et al. Bagging improves reproducibility of functional parcellation of the human brain. Neuroimage 214, 116678 (2020).Article
PubMed
Google Scholar
Doshi, J. et al. MUSE: MUlti-atlas region Segmentation utilizing Ensembles of registration algorithms and parameters, and locally optimal atlas selection. Neuroimage 127, 186–195 (2016).Article
PubMed
Google Scholar