Learning together: Towards foundation models for machine learning interatomic potentials with meta-learning

Isayev, O. et al. Universal fragment descriptors for predicting properties of inorganic crystals. Nat. Commun. 8, 15679 (2017).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).Article 
CAS 
PubMed 

Google Scholar 
Mikulskis, P., Alexander, M. R. & Winkler, D. A. Toward interpretable machine learning models for materials discovery. Adv. Intell. Syst. 1, 1900045 (2019).Article 

Google Scholar 
Pilania, G. Machine learning in materials science: from explainable predictions to autonomous design. Comput. Mater. Sci. 193, 110360 (2021).Article 
CAS 

Google Scholar 
Ouyang, R., Curtarolo, S., Ahmetcik, E., Scheffler, M. & Ghiringhelli, L. M. Sisso: a compressed-sensing method for identifying the best low-dimensional descriptor in an immensity of offered candidates. Phys. Rev. Mater. 2, 083802 (2018).Article 
CAS 

Google Scholar 
Jha, D. et al. Elemnet: deep learning the chemistry of materials from only elemental composition. Sci. Rep. 8, 17593 (2018).Article 
PubMed 
PubMed Central 

Google Scholar 
Nandy, A. et al. Computational discovery of transition-metal complexes: from high-throughput screening to machine learning. Chem. Rev. 121, 9927–10000 (2021).Article 
CAS 
PubMed 

Google Scholar 
Keith, J. A. et al. Combining machine learning and computational chemistry for predictive insights into chemical systems. Chem. Rev. 121, 9816–9872 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Zhuo, Y., Mansouri Tehrani, A. & Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. J. Phys. Chem. 9, 1668–1673 (2018).CAS 

Google Scholar 
Smith, J. S., Isayev, O. & Roitberg, A. E. ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost. Chem. Sci. 8, 3192–3203 (2017).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Faber, F. A., Lindmaa, A., von Lilienfeld, O. A. & Armiento, R. Machine learning energies of 2 million elpasolite (ABC2D6) crystals. Phys. Rev. Lett. 117, 135502 (2016).Article 
PubMed 

Google Scholar 
Magedov, S., Koh, C., Malone, W., Lubbers, N. & Nebgen, B. Bond order predictions using deep neural networks. J. Appl. Phys. 129, 064701 (2021).Article 
CAS 

Google Scholar 
Nandi, S., Vegge, T. & Bhowmik, A. Multixc-qm9: large dataset of molecular and reaction energies from multi-level quantum chemical methods. Sci. Data 10, 783 (2023).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Smith, J. S. et al. The ani-1ccx and ani-1x data sets, coupled-cluster and density functional theory properties for molecules. Sci. Data 7, 134 (2020).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Axelrod, S. & Gómez-Bombarelli, R. GEOM, energy-annotated molecular conformations for property prediction and molecular generation. Sci. Data 9, 185 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Hoja, J. et al. Qm7-x, a comprehensive dataset of quantum-mechanical properties spanning the chemical space of small organic molecules. Sci. Data 8, 43 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Isert, C., Atz, K., Jiménez-Luna, J. & Schneider, G. Qmugs, quantum mechanical properties of drug-like molecules. Sci. Data 9, 273 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Schreiner, M., Bhowmik, A., Vegge, T., Busk, J. & Winther, O. Transition1x – a dataset for building generalizable reactive machine learning potentials. Sci. Data 9, 779 (2022).Article 
PubMed 
PubMed Central 

Google Scholar 
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K.-R. & Tkatchenko, A. Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 13890 (2017).Article 
PubMed 
PubMed Central 

Google Scholar 
Bartók, A. P., Payne, M. C., Kondor, R. & Csányi, G. Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett. 104, 136403 (2010).Article 
PubMed 

Google Scholar 
Behler, J. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. 134, 074106–14 (2011).Article 
PubMed 

Google Scholar 
Unke, O. T. et al. Machine learning force fields. Chem. Rev. 121, 10142–10186 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Mueller, T., Hernandez, A. & Wang, C. Machine learning for interatomic potential models. J. Chem. Phys. 152, 050902 (2020).Article 
CAS 
PubMed 

Google Scholar 
Bartók, A. P., Kermode, J., Bernstein, N. & Csányi, G. Machine learning a general-purpose interatomic potential for silicon. Phys. Rev. X 8, 041048 (2018).
Google Scholar 
Noé, F., Tkatchenko, A., Müller, K.-R. & Clementi, C. Machine learning for molecular simulation. Ann. Rev. Phys. Chem. 71, 361–390 (2020).Article 

Google Scholar 
Frenkel, D. & Smit, B. Understanding Molecular Simulation 2nd edn (Academic Press, Inc., USA, 2001).Weiner, S. J. et al. A new force field for molecular mechanical simulation of nucleic acids and proteins. J. Am. Chem. Soc. 106, 765–784 (1984).Article 
CAS 

Google Scholar 
Jorgensen, W. L. & Tirado-Rives, J. The OPLS [optimized potentials for liquid simulations] potential functions for proteins, energy minimizations for crystals of cyclic peptides and crambin. J. Am. Chem. Soc. 110, 1657–1666 (1988).Article 
CAS 
PubMed 

Google Scholar 
Bartlett, R. J. & Musiał, M. Coupled-cluster theory in quantum chemistry. Rev. Mod. Phys. 79, 291–352 (2007).Article 
CAS 

Google Scholar 
Tran, R. et al. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts. ACS Catal. 13, 3066–3084 (2023).Article 
CAS 

Google Scholar 
Smith, J. S. et al. Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning. Nat. Commun. 10, 2903 (2019).Article 
PubMed 
PubMed Central 

Google Scholar 
Chen, M. S. et al. Data-efficient machine learning potentials from transfer learning of periodic correlated electronic structure methods: Liquid water at AFQMC, CCSD, and CCSD(T) accuracy. J. Chem. Theory Comput. 19, 4510–4519 (2023).Article 
CAS 
PubMed 

Google Scholar 
Taylor, M. E. & Stone, P. Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009).
Google Scholar 
Pan, S. J. & Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2010).Article 

Google Scholar 
Zaverkin, V., Holzmüller, D., Bonfirraro, L. & Kästner, J. Transfer learning for chemically accurate interatomic neural network potentials. Phys. Chem. Chem. Phys. 25, 5383–5396 (2023).Article 
CAS 
PubMed 

Google Scholar 
Kovács, D. P. et al. Linear atomic cluster expansion force fields for organic molecules: Beyond RMSE. J. Chem. Theory Comput. 17, 7696–7711 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Finn, C., Abbeel, P. & Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, 1126–1135 (2017).Hospedales, T., Antoniou, A., Micaelli, P. & Storkey, A. Meta-learning in neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44, 5149–5169 (2022).PubMed 

Google Scholar 
Nichol, A., Achiam, J. & Schulman, J. On first-order meta-learning algorithms. Preprint at https://arxiv.org/abs/1803.02999 (2018).Huisman, M., van Rijn, J. N. & Plaat, A. A survey of deep meta-learning. Artif. Intell. Rev. 54, 4483–4541 (2021).Article 

Google Scholar 
Bommasani, R.et al. On the opportunities and risks of foundation models. Preprint at https://arxiv.org/abs/2108.07258 (2022).OpenAI. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).Cole, D., Mones, L. & Csányi, G. A machine learning based intramolecular potential for a flexible organic molecule. Faraday Discuss. 224, 247–264 (2020).Article 
CAS 
PubMed 

Google Scholar 
St. John, P. C., Guan, Y., Kim, Y., Kim, S. & Paton, R. S. Prediction of organic homolytic bond dissociation enthalpies at near chemical accuracy with sub-second computational cost. Nat. Commun. 11, 2328 (2020).Article 

Google Scholar 
Shao, Y., Dietrich, F. M., Nettelblad, C. & Zhang, C. Training algorithm matters for the performance of neural network potential: A case study of adam and the kalman filter optimizers. J. Chem. Phys. 155, 204108 (2021).Article 
CAS 
PubMed 

Google Scholar 
Fu, X.et al. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations., Transactions on Machine Learning Research, 2835–8856, (2023)Han, X. et al. Pre-trained models: Past, present and future. AI Open 2, 225–250 (2021).Article 

Google Scholar 
Hu*, W.et al. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (2020). https://openreview.net/forum?id=HJlWWJSFDH.Hendrycks, D., Lee, K. & Mazeika, M. Using pre-training can improve model robustness and uncertainty. In Proceedings of the 36th International Conference on Machine Learning, vol. 97, 2712–2721 (PMLR, 2019).Sun, Y. et al. Fingerprinting diverse nanoporous materials for optimal hydrogen storage conditions using meta-learning. Sci. Adv. 7, eabg3983 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Nie, J., Wang, N., Li, J., Wang, K. & Wang, H. Meta-learning prediction of physical and chemical properties of magnetized water and fertilizer based on lstm. Plant Methods 17, 119 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Wang, J., Zheng, S., Chen, J. & Yang, Y. Meta learning for low-resource molecular optimization. J. Chem. Inf. 61, 1627–1636 (2021).Article 
CAS 

Google Scholar 
Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 140022 (2014).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Batzner, S. et al. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun. 13, 2453 (2022).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Lubbers, N., Smith, J. S. & Barros, K. Hierarchical modeling of molecular energies using a deep neural network. J. Chem. Phys. 148, (2018).Neese, F., Wennmohs, F., Becker, U. & Riplinger, C. The ORCA quantum chemistry program package. J. Chem. Phys. 152, 224108 (2020).Frisch, M. J.et al. Gaussian 16 Revision C.01. (Gaussian Inc. Wallingford CT, 2016).Nakata, M., Shimazaki, T., Hashimoto, M. & Maeda, T. Pubchemqc pm6: data sets of 221 million molecules with optimized molecular geometries and electronic properties. J. Chem. Inf. 60, 5891–5899 (2020).Article 
CAS 

Google Scholar 
Yao, K., Herr, J. E., Toth, D., Mckintyre, R. & Parkhill, J. The tensormol-0.1 model chemistry: a neural network augmented with long-range physics. Chem. Sci. 9, 2261–2269 (2018).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O. & Roitberg, A. E. Less is more: sampling chemical space with active learning. J. Chem. Phys. 148, 241733 (2018).Article 
PubMed 

Google Scholar