Benchmarking algorithms for single-cell multi-omics prediction and integration

Stoeckius, M. et al. Simultaneous epitope and transcriptome measurement in single cells. Nat. Methods 14, 865–868 (2017).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Peterson, V. M. et al. Multiplexed quantification of proteins and transcripts in single cells. Nat. Biotechnol. 35, 936–939 (2017).Article 
CAS 
PubMed 

Google Scholar 
Ma, S. et al. Chromatin potential identified by shared single-cell profiling of RNA and chromatin. Cell 183, 1103–1116.e20 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Chen, S., Lake, B. B. & Zhang, K. High-throughput sequencing of the transcriptome and chromatin accessibility in the same cell. Nat. Biotechnol. 37, 1452–1457 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Hao, Y. et al. Integrated analysis of multimodal single-cell data. Cell 184, 3573–3587.e29 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Gayoso, A. et al. Joint probabilistic modeling of single-cell multi-omic data with totalVI. Nat. Methods 18, 272–282 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Zhang, L., Zhang, J. & Nie, Q. DIRECT-NET: an efficient method to discover cis-regulatory elements and construct regulatory networks from single-cell multiomics data. Sci. Adv. 8, eabl7393 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Kartha, V. K. et al. Functional inference of gene regulation using single-cell multi-omics. Cell Genom. 2, 100166 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Li, C., Virgilio, M. C., Collins, K. L. & Welch, J. D. Multi-omic single-cell velocity models epigenome–transcriptome interactions and improves cell fate prediction. Nat. Biotechnol. 41, 387–398 (2023).Article 
CAS 
PubMed 

Google Scholar 
Gorin, G., Svensson, V. & Pachter, L. Protein velocity and acceleration from single-cell multiomics experiments. Genome Biol. 21, 39 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
La Manno, G. et al. RNA velocity of single cells. Nature 560, 494–498 (2018).Article 
PubMed 
PubMed Central 

Google Scholar 
Xu, W. et al. ISSAAC-seq enables sensitive and flexible multimodal profiling of chromatin accessibility and gene expression in single cells. Nat. Methods 19, 1243–1249 (2022).Article 
CAS 
PubMed 

Google Scholar 
Zhou, Z., Ye, C., Wang, J. & Zhang, N. R. Surface protein imputation from single cell transcriptomes by deep neural networks. Nat. Commun. 11, 651 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Bennett, H. M., Stephenson, W., Rose, C. M. & Darmanis, S. Single-cell proteomics enabled by next-generation sequencing or mass spectrometry. Nat. Methods 20, 363–374 (2023).Article 
CAS 
PubMed 

Google Scholar 
Gatto, L. et al. Initial recommendations for performing, benchmarking and reporting single-cell proteomics experiments. Nat. Methods 20, 375–386 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lance, C. et al. Multimodal single cell data integration challenge: results and lessons learned. In Proc. NeurIPS 2021 Competitions and Demonstrations Track (eds. Kiela, D. et al.) 162–176 (PMLR, 2022).Bartosovic, M., Kabbe, M. & Castelo-Branco, G. Single-cell CUT&Tag profiles histone modifications and transcription factors in complex tissues. Nat. Biotechnol. 39, 825–835 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Mimitou, E. P. et al. Scalable, multimodal profiling of chromatin accessibility, gene expression and protein levels in single cells. Nat. Biotechnol. 39, 1246–1258 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Stuart, T. et al. Comprehensive integration of single-cell data. Cell 177, 1888–1902.e21 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Welch, J. D. et al. Single-cell multi-omic integration compares and contrasts features of brain cell identity. Cell 177, 1873–887.e17 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lotfollahi, M. et al. Mapping single-cell data to reference atlases by transfer learning. Nat. Biotechnol. 40, 121–130 (2022).Article 
CAS 
PubMed 

Google Scholar 
Ashuach, T. et al. MultiVI: deep generative model for the integration of multimodal data. Nat. Methods 20, 1222–1231 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lakkis, J. et al. A multi-use deep learning method for CITE-seq and single-cell RNA-seq data integration with cell surface protein prediction and imputation. Nat. Mach. Intell. 4, 940–952 (2022).Article 
PubMed 
PubMed Central 

Google Scholar 
Wu, K. E., Yost, K. E., Chang, H. Y. & Zou, J. BABEL enables cross-modality translation between multiomic profiles at single-cell resolution. Proc. Natl Acad. Sci. USA 118, e2023070118 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Du, J.-H., Cai, Z. & Roeder, K. Robust probabilistic modeling for single-cell multimodal mosaic integration and imputation via scVAEIT. Proc. Natl Acad. Sci. USA 119, e2214414119 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lan, M., Zhang, S. & Gao, L. Efficient generation of paired single-cell multiomics profiles by deep learning. Adv. Sci 10, 2301169 (2023).Article 
CAS 

Google Scholar 
Wen, H. et al. Proc. 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery, 2022).Yang, K. D. et al. Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nat. Commun. 12, 31 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Baysoy, A., Bai, Z., Satija, R. & Fan, R. The technological landscape and applications of single-cell multi-omics. Nat. Rev. Mol. Cell Biol. 24, 695–713 (2023).Article 
CAS 
PubMed 

Google Scholar 
Cheng, M., Li, Z. & Costa, I. G. MOJITOO: a fast and universal method for integration of multimodal single-cell data. Bioinformatics 38, i282–i289 (2022).Article 
PubMed 
PubMed Central 

Google Scholar 
Lotfollahi, M., Litinetskaya, A. & Theis, F. J. Multigrate: single-cell multi-omic data integration. Preprint at bioRxiv https://doi.org/10.1101/2022.03.16.484643 (2022).Wang, R. H., Wang, J. & Li, S. C. Probabilistic tensor decomposition extracts better latent embeddings from single-cell multiomic data. Nucleic Acids Res. 51, e81 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Kim, H. J., Lin, Y., Geddes, T. A., Yang, J. Y. H. & Yang, P. CiteFuse enables multi-modal analysis of CITE-seq data. Bioinformatics 36, 4137–4143 (2020).Article 
CAS 
PubMed 

Google Scholar 
Ma, A. et al. Single-cell biological network inference using a heterogeneous graph transformer. Nat. Commun. 14, 964 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Jin, S., Zhang, L. & Nie, Q. scAI: an unsupervised approach for the integrative analysis of parallel single-cell transcriptomic and epigenomic profiles. Genome Biol. 21, 25 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Argelaguet, R. et al. MOFA+: a statistical framework for comprehensive integration of multi-modal single-cell data. Genome Biol. 21, 111 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Li, G. et al. A deep generative model for multi-view profiling of single-cell RNA-seq and ATAC-seq data. Genome Biol. 23, 20 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lynch, A. W. et al. MIRA: joint regulatory modeling of multimodal expression and chromatin accessibility in single cells. Nat. Methods 19, 1097–1108 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Singh, R., Hie, B. L., Narayan, A. & Berger, B. Schema: metric learning enables interpretable synthesis of heterogeneous single-cell modalities. Genome Biol. 22, 131 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Kriebel, A. R. & Welch, J. D. UINMF performs mosaic integration of single-cell multi-omic datasets using nonnegative matrix factorization. Nat. Commun. 13, 780 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Zhang, Z. et al. scMoMaT jointly performs single cell mosaic integration and multi-modal bio-marker detection. Nat. Commun. 14, 384 (2023).Article 
PubMed 
PubMed Central 

Google Scholar 
Ghazanfar, S., Guibentif, C. & Marioni, J. C. Stabilized mosaic single-cell data integration using unshared features. Nat. Biotechnol. 42, 284–292 (2024).Article 
CAS 
PubMed 

Google Scholar 
De Biasi, S. et al. Circulating mucosal-associated invariant T cells identify patients responding to anti-PD-1 therapy. Nat. Commun. 12, 1669 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Heumos, L. et al. Best practices for single-cell analysis across modalities. Nat. Rev. Genet. 24, 550–572 (2023).Article 
CAS 
PubMed 

Google Scholar 
Miao, Z., Humphreys, B. D., McMahon, A. P. & Kim, J. Multi-omics integration in the age of million single-cell data. Nat. Rev. Nephrol. 17, 710–724 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Argelaguet, R., Cuomo, A. S. E., Stegle, O. & Marioni, J. C. Computational principles and challenges in single-cell data integration. Nat. Biotechnol. 39, 1202–1215 (2021).Article 
CAS 
PubMed 

Google Scholar 
Wang, J. et al. Data denoising with transfer learning in single-cell transcriptomics. Nat. Methods 16, 875–878 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Huang, M. et al. SAVER: gene expression recovery for single-cell RNA sequencing. Nat. Methods 15, 539–542 (2018).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Hu, Y. et al. WEDGE: imputation of gene expression values from single-cell RNA-seq datasets using biased matrix decomposition. Brief. Bioinform. 22, bbab085 (2021).Article 
PubMed 

Google Scholar 
Truong, K.-L. et al. Killer-like receptors and GPR56 progressive expression defines cytokine production of human CD4+ memory T cells. Nat. Commun. 10, 2263 (2019).Article 
PubMed 
PubMed Central 

Google Scholar 
Fergusson, J. R. et al. CD161intCD8+ T cells: a novel population of highly functional, memory CD8+ T cells enriched within the gut. Mucosal Immunol. 9, 401–413 (2016).Article 
CAS 
PubMed 

Google Scholar 
Kung, P. C., Goldstein, G., Reinherz, E. L. & Schlossman, S. F. Monoclonal antibodies defining distinctive human T cell surface antigens. Science 206, 347–349 (1979).Article 
CAS 
PubMed 

Google Scholar 
Liang, Y. & Tedder, T. F. Identification of a CD20-, FcϵRIβ-, and HTm4-Related gene family: sixteen new MS4A family members expressed in human and mouse. Genomics 72, 119–127 (2001).Article 
CAS 
PubMed 

Google Scholar 
Ziegler-Heitbrock, H. W. L. & Ulevitch, R. J. CD14: cell surface receptor and differentiation marker. Immunol. Today 14, 121–125 (1993).Article 
CAS 
PubMed 

Google Scholar 
Stuart, T., Srivastava, A., Madad, S., Lareau, C. A. & Satija, R. Single-cell chromatin state analysis with Signac. Nat. Methods 18, 1333–1341 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Spitz, F. & Furlong, E. E. M. Transcription factors: from enhancer binding to developmental control. Nat. Rev. Genet. 13, 613–626 (2012).Article 
CAS 
PubMed 

Google Scholar 
Gertz, J. et al. Distinct properties of cell-type-specific and shared transcription factor binding sites. Mol. Cell 52, 25–36 (2013).Article 
CAS 
PubMed 

Google Scholar 
Kang, R. et al. EnhancerDB: a resource of transcriptional regulation in the context of enhancers. Database 2019, bay141 (2019).Article 
PubMed 
PubMed Central 

Google Scholar 
Buergel, T. et al. Metabolomic profiles predict individual multidisease outcomes. Nat. Med. 28, 2309–2320 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lewis, S. M. et al. Spatial omics and multiplexed imaging to explore cancer biology. Nat. Methods 18, 997–1012 (2021).Article 
CAS 
PubMed 

Google Scholar 
Li, B. et al. Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nat. Methods 19, 662–670 (2022).Article 
CAS 
PubMed 

Google Scholar 
Linderman, G. C. et al. Zero-preserving imputation of single-cell RNA-seq data. Nat. Commun. 13, 192 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Yuan, H. & Kelley, D. R. scBasset: sequence-based modeling of single-cell ATAC-seq using convolutional neural networks. Nat. Methods 19, 1088–1096 (2022).Article 
CAS 
PubMed 

Google Scholar 
Chen, A. et al. Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays. Cell 185, 1777–1792.e21 (2022).Article 
CAS 
PubMed 

Google Scholar 
Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015).Article 
PubMed 
PubMed Central 

Google Scholar 
Su, G. et al. Spatial multi-omics sequencing for fixed tissue via DBiT-seq. STAR Protoc. 2, 100532 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Liu, Y. et al. High-plex protein and whole transcriptome co-mapping at cellular resolution with spatial CITE-seq. Nat. Biotechnol. 41, 1405–1409 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Theodoris, C. V. et al. Transfer learning enables predictions in network biology. Nature 618, 616–624 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Cui, H. et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nat. Methods 21, 1470–1480 (2024).Article 
CAS 
PubMed 

Google Scholar 
Hao, M. et al. Large-scale foundation model on single-cell transcriptomics. Nat. Methods 21, 1481–1491 (2024).Article 
CAS 
PubMed 

Google Scholar 
Swanson, E. et al. Simultaneous trimodal single-cell measurement of transcripts, epitopes, and chromatin accessibility using TEA-seq. eLife 10, e63632 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Hand, D. J. & Till, R. J. A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach. Learn. 45, 171–186 (2001).Article 

Google Scholar 
Hubert, L. & Arabie, P. Comparing partitions. J. Classif. 2, 193–218 (1985).Article 

Google Scholar 
Strehl, A. & Ghosh, J. Cluster ensembles–a knowledge reuse framework for combining multiple partitions. J. Mach. Learn. Res. 3, 583–617 (2002).
Google Scholar 
Rousseeuw, P. J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987).Article 

Google Scholar 
Luecken, M. D. et al. Benchmarking atlas-level data integration in single-cell genomics. Nat. Methods 19, 41–50 (2022).Article 
CAS 
PubMed 

Google Scholar 
Korsunsky, I. et al. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat. Methods 16, 1289–1296 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A. & Theis, F. J. A test metric for assessing single-cell RNA-seq batch correction. Nat. Methods 16, 43–49 (2019).Article 
PubMed 

Google Scholar 
Luecken, M. D. et al. A sandbox for prediction and integration of DNA, RNA, and proteins in single cells. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks (eds. Vanschoren, J. & Yeung, S.) 13 (NeurIPS, 2021).Kumar, P. et al. Single-cell transcriptomics and surface epitope detection in human brain epileptic lesions identifies pro-inflammatory signaling. Nat. Neurosci. 25, 956–966 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Pombo Antunes, A. R. et al. Single-cell profiling of myeloid cells in glioblastoma across species and disease stage reveals macrophage competition and specialization. Nat. Neurosci. 24, 595–610 (2021).Article 
CAS 
PubMed 

Google Scholar 
Konturek-Ciesla, A. et al. Temporal multimodal single-cell profiling of native hematopoiesis illuminates altered differentiation trajectories with age. Cell Rep. 42, 112304 (2023).Article 
CAS 
PubMed 

Google Scholar 
Lukowski, S. W. et al. Absence of Batf3 reveals a new dimension of cell state heterogeneity within conventional dendritic cells. iScience 24, 102402 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Golomb, S. M. et al. Multi-modal single-cell analysis reveals brain immune landscape plasticity during aging and gut microbiota dysbiosis. Cell Rep. 33, 108438 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Chung, H. et al. Joint single-cell measurements of nuclear proteins and RNA in vivo. Nat. Methods 18, 1204–1212 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Dou, J. et al. Bi-order multimodal integration of single-cell data. Genome Biol. 23, 112 (2022).Article 
PubMed 
PubMed Central 

Google Scholar 
Kimmel, J. C. et al. Murine single-cell RNA-seq reveals cell-identity-and tissue-specific trajectories of aging. Genome Res. 29, 2088–2103 (2019).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lyu, P. et al. Gene regulatory networks controlling temporal patterning, neurogenesis, and cell-fate specification in mammalian retina. Cell Rep. 37, 109994 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Sun, W. et al. Spatial transcriptomics reveal neuron–astrocyte synergy in long-term memory. Nature 627, 374–381 (2024).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Hu, Y. et al. Benchmarking algorithms for single-cell multi-omics prediction and integration. Zenodo https://doi.org/10.5281/zenodo.10540843 (2024).