We present an in-depth exploration of the Metabolite Inference with Spectrum Transformers (MIST) tool for annotating small-molecule mass spectrometry (MS) data, focusing on its reproducibility and generalizability. MIST innovates by integrating a ‘chemical formula transformer’ to process tandem MS spectra, aiming to bridge the substantial knowledge gap in untargeted MS studies, in which only a fraction of spectra are confidently annotated. Here we critically assessed the reproducibility of MIST by following the tool’s original training and testing protocols, encountering minor challenges but largely succeeding in replicating the results. We also evaluated the generalizability of MIST by applying it to an external dataset from the Critical Assessment of Small Molecule Identification 2022 challenge, showing insights into the model’s performance on previously unseen data. An ablation study further investigated the impact of various model features on database retrieval performance, suggesting that some algorithmic complexities may not significantly enhance the performance. Through rigorous evaluation, this study underscores the challenges and considerations in developing robust computational tools for MS data analysis. We advocate community-wide efforts in benchmarking, transparency and data sharing to foster advancements in metabolomics and computational biology.
