Moving beyond processing- and analysis-related variation in resting-state functional brain imaging

Shehzad, Z. et al. The resting brain: unconstrained yet reliable. Cereb. Cortex 19, 2209–2229 (2009).Article 
PubMed 
PubMed Central 

Google Scholar 
Zuo, X.-N. et al. The oscillating brain: complex and reliable. Neuroimage 49, 1432–1445 (2010).Article 
PubMed 

Google Scholar 
Bennett, C. M. & Miller, M. B. How reliable are the results from functional magnetic resonance imaging? Ann. N. Y. Acad. Sci. 1191, 133–155 (2010).Article 
PubMed 

Google Scholar 
Zuo, X.-N., Xu, T. & Milham, M. P. Harnessing reliability for neuroscience research. Nat. Hum. Behav. 3, 768–771 (2019).Article 
PubMed 

Google Scholar 
Kraemer, H. C. The reliability of clinical diagnoses: state of the art. Annu. Rev. Clin. Psychol. 10, 111–130 (2014).Article 
PubMed 

Google Scholar 
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).Article 
CAS 
PubMed 

Google Scholar 
Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).Article 
PubMed 
PubMed Central 

Google Scholar 
Noble, S., Scheinost, D. & Constable, R. T. A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage 203, 116157 (2019).Article 
PubMed 

Google Scholar 
Zuo, X.-N. & Xing, X.-X. Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: a systems neuroscience perspective. Neurosci. Biobehav. Rev. 45, 100–118 (2014).Article 
PubMed 

Google Scholar 
Cho, J. W., Korchmaros, A., Vogelstein, J. T., Milham, M. P. & Xu, T. Impact of concatenating fMRI data on reliability for functional connectomics. Neuroimage 226, 117549 (2021).Article 
PubMed 

Google Scholar 
Lynch, C. J. et al. Rapid precision functional mapping of individuals using multi-echo fMRI. Cell Rep. 33, 108540 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Nikolaidis, A. et al. Bagging improves reproducibility of functional parcellation of the human brain. Neuroimage 214, 116678 (2020).Article 
PubMed 

Google Scholar 
Yoo, K. et al. Multivariate approaches improve the reliability and validity of functional connectivity and prediction of individual behaviors. Neuroimage 197, 212–223 (2019).Article 
PubMed 

Google Scholar 
Elliott, M. L. et al. What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol. Sci. 31, 792–806 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Palumbo, L. et al. Evaluation of the intra- and inter-method agreement of brain MRI segmentation software packages: a comparison between SPM12 and FreeSurfer v6.0. Phys. Med. 64, 261–272 (2019).Article 
CAS 
PubMed 

Google Scholar 
Oakes, T. R. et al. Comparison of fMRI motion correction software tools. Neuroimage 28, 529–543 (2005).Article 
CAS 
PubMed 

Google Scholar 
Klein, A. et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46, 786–802 (2009).Article 
PubMed 

Google Scholar 
Dickie, E., Hodge, S., Craddock, R., Poline, J.-B. & Kennedy, D. Tools matter: comparison of two surface analysis tools applied to the ABIDE dataset. Res. Ideas Outcomes 3, e13726 (2017).Article 

Google Scholar 
Bhagwat, N. et al. Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses. Gigascience 10, giaa155 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Carp, J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012).Article 
PubMed 
PubMed Central 

Google Scholar 
Pauli, R. et al. Exploring fMRI results space: 31 variants of an fMRI analysis in AFNI, FSL, and SPM. Front. Neuroinform. 10, 24 (2016).Article 
PubMed 
PubMed Central 

Google Scholar 
Bowring, A., Maumet, C. & Nichols, T. E. Exploring the impact of analysis software on task fMRI results. Hum. Brain Mapp. 40, 3362–3384 (2019).Article 
PubMed 
PubMed Central 

Google Scholar 
Bowring, A., Nichols, T. E. & Maumet, C. Isolating the sources of pipeline-variability in group-level task-fMRI results. Hum. Brain Mapp. 43, 1112–1128 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Feczko, E., Conan, G., Marek, S. & Tervo-Clemmens, B. Adolescent brain cognitive development (ABCD) community MRI collection and utilities. Preprint at bioRxiv https://doi.org/10.1101/2021.07.09.451638 (2021).Xu, T., Yang, Z., Jiang, L., Xing, X.-X. & Zuo, X.-N. A connectome computation system for discovery science of brain. Sci. Bull. (Beijing) 60, 86–95 (2015).Article 

Google Scholar 
Craddock, C. et al. Towards automated analysis of connectomes: the configurable pipeline for the analysis of connectomes (C-PAC). Front. Neuroinform. 42, 10-3389 (2013).
Google Scholar 
Chao-Gan, Y. & Yu-Feng, Z. DPARSF: a MATLAB toolbox for ‘pipeline’ data analysis of resting-State fMRI. Front. Syst. Neurosci. 4, 13 (2010).PubMed 
PubMed Central 

Google Scholar 
Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).Article 
CAS 
PubMed 

Google Scholar 
Murphy, K. & Fox, M. D. Towards a consensus regarding global signal regression for resting state functional connectivity MRI. Neuroimage 154, 169–173 (2017).Article 
PubMed 

Google Scholar 
Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Shou, H. et al. Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2). Cogn. Affect. Behav. Neurosci. 13, 714–724 (2013).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Schaefer, A. et al. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb. Cortex 28, 3095–3114 (2018).Article 
PubMed 

Google Scholar 
Glasser, M. F. et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage 80, 105–124 (2013).Article 
PubMed 

Google Scholar 
Greve, D. N. & Fischl, B. Accurate and robust brain image alignment using boundary-based registration. Neuroimage 48, 63–72 (2009).Article 
PubMed 

Google Scholar 
Alexander, L. M. et al. An open resource for transdiagnostic research in pediatric mental health and learning disorders. Sci. Data 4, 170181 (2017).Article 
PubMed 
PubMed Central 

Google Scholar 
Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155–163 (2016).Dong, Y., Ifrim, G., Mladenić, D., Saunders, C. & Van Hoecke, S. Machine learning and knowledge discovery in databases. Applied data science and demo track. In Proc. European Conference, ECML PKDD 2020 Part V (eds Dong, Y. et al.) 3–18 (Springer Nature, 2021).Birn, R. M. et al. The effect of scan length on the reliability of resting-state fMRI connectivity estimates. Neuroimage 83, 550–558 (2013).Article 
PubMed 

Google Scholar 
Liu, T. T., Nalci, A. & Falahpour, M. The global signal in fMRI: nuisance or information? Neuroimage 150, 213–229 (2017).Article 
PubMed 

Google Scholar 
Ciric, R. et al. Benchmarking of participant-level confound regression strategies for the control of motion artifact in studies of functional connectivity. Neuroimage 154, 174–187 (2017).Article 
PubMed 

Google Scholar 
Buades, A., Coll, B. & Morel, J.-M. Non-local means denoising. IPOL J. 1, 208–212 (2011).Article 

Google Scholar 
Tustison, N. J. et al. N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29, 1310–1320 (2010).Article 
PubMed 
PubMed Central 

Google Scholar 
Ciric, R. et al. TemplateFlow: FAIR-sharing of multi-scale, multi-species brain models. Nat. Methods 19, 1568–1571 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R. & Collins, D. L. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage 47, S102 (2009).Article 

Google Scholar 
Grabner, G. et al. Symmetric atlasing and model based segmentation: an application to the hippocampus in older adults. Med. Image Comput. Comput. Assist. Interv. 9, 58–66 (2006).PubMed 

Google Scholar 
Mazziotta, J. et al. A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). Philos. Trans. R. Soc. Lond. B Biol. Sci. 356, 1293–1322 (2001).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Wu, J. et al. Accurate nonlinear mapping between MNI volumetric and FreeSurfer surface coordinate systems. Hum. Brain Mapp. 39, 3793–3808 (2018).Article 
PubMed 
PubMed Central 

Google Scholar 
Uddin, L. Q. Mixed signals: on separating brain signal from noise. Trends Cogn. Sci. 21, 405–406 (2017).Article 
PubMed 
PubMed Central 

Google Scholar 
Murphy, K., Birn, R. M., Handwerker, D. A., Jones, T. B. & Bandettini, P. A. The impact of global signal regression on resting state correlations: are anti-correlated networks introduced? Neuroimage 44, 893–905 (2009).Article 
PubMed 

Google Scholar 
Xu, H. et al. Impact of global signal regression on characterizing dynamic functional connectivity and brain states. Neuroimage 173, 127–145 (2018).Article 
PubMed 

Google Scholar 
Gordon, E. M. et al. Precision functional mapping of individual human brains. Neuron 95, 791–807 (2017).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Di Martino, A. et al. Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci. Data 4, 170010 (2017).Article 
PubMed 
PubMed Central 

Google Scholar 
Casey, B. J. et al. The Adolescent Brain Cognitive Development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Richie-Halford, A. et al. Author correction: an analysis-ready and quality controlled resource for pediatric brain white-matter research. Sci. Data 10, 247 (2023).Article 
PubMed 
PubMed Central 

Google Scholar 
Doshi, J. et al. MUSE: Multi-atlas region segmentation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. NeuroImage 127, 186–195 (2016).Wu, D. et al. Resource atlases for multi-atlas brain segmentations with multiple ontology levels based on T1-weighted MRI. Neuroimage 125, 120–130 (2016).Article 
PubMed 

Google Scholar 
Kiar, G., Chatelain, Y., Salari, A., Evans, A. C. & Glatard, T. Data augmentation through Monte Carlo arithmetic leads to more generalizable classification in connectomics. Neurons Behav. Data Anal. Theory 1, 1–20 (2021).
Google Scholar 
Kiar, G. et al. Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. PLoS ONE 16, e0250755 (2021).Mehta, K. et al. XCP-D: a robust pipeline for the post-processing of fMRI data. Preprint at bioRxiv 10.1101/2023.11.20.567926 (2023).Bujang, M. & Baharum, N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Arch. Orofac. Sci. 12, 1–11 (2017).
Google Scholar 
Smith, S. M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23, S208–S219 (2004).Article 
PubMed 

Google Scholar 
Jenkinson, M., Bannister, P., Brady, M. & Smith, S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17, 825–841 (2002).Article 
PubMed 

Google Scholar 
Cox, R. W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29, 162–173 (1996).Article 
CAS 
PubMed 

Google Scholar 
Zhang, Y., Brady, M. & Smith, S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans. Med. Imaging 20, 45–57 (2001).Avants, B. B., Tustison, N. & Song, G. Advanced normalization tools (ANTS). Insight J. 2, 1–35 (2009).
Google Scholar 
Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).Article 
PubMed 

Google Scholar 
Jenkinson, M. & Smith, S. A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5, 143–156 (2001).Article 
CAS 
PubMed 

Google Scholar 
Berger, V. W. & Zhou, Y. Kolmogorov–Smirnov test: overview https://doi.org/10.1002/9781118445112.stat06558 (2014).Nachar, N. The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution. Tutor. Quant. Methods Psychol. 4, 13–20 (2008).Article 

Google Scholar 
Li, X. & Clucas, J. XinhuiLi/PipelineHarmonization: Pipeline Harmonization Version 0.0.0 Beta. Zenodo https://doi.org/10.5281/zenodo.5733801 (2021).