In the current study, we have developed the TME-Analyzer, a novel tool that captures intra- and inter-tissue heterogeneity with individualized and flow cytometry-like analysis. With TME-Analyzer, we have demonstrated easy retrieval of contextures of immune effector cells in TNBC tissues from multiplexed immunofluorescent images, including cell densities and inter-cellular distances in border and center regions of either tumor or stroma compartments. We benchmarked the TME-Analyzer against two established software tools, namely inForm and QuPath, and showed high concordance regarding tissue segmentation and cell phenotyping, with enhanced accuracy and utility. Subsequently, starting from hundreds of contexture-based parameters extracted with TME-Analyzer, we have built a 10-parameter classifier for survival of TNBC patients using multiplexed immunofluorescence images, which we validated using mass spectrometry images. This 10-parameter classifier pointed to the diverse applicability of the TME-Analyzer, and also revealed the impact of inter-immune cell distances towards survival of TNBC patients as well as its outperformance of recognized prognostic parameters, such as inflammation and stromal T cell densities.Tumors and their microscopic images are highly heterogeneous due to contextual variations in tissues and their regions, clinical variantions among patients, as well as technical variations in staining and imaging across laboratories. In order to extract data from various images in a uniform and high-throughput manner, we developed a real-time interactive analysis with a WYSIWYG interface. TME-Analyzer proved to be an easy-to-use, fast and reproducible analysis tool (Fig. 1B, Table 1), which enables exportation, sharing and standardization of analysis. With its modular analysis approach, it particularly provides versatility regarding extraction of image data from different platforms. In contrast to commercial benchmark software inForm, TME-Analyzer was more robust, i.e., requiring less algorithms for the same analysis, and efficient, i.e., able to analyze more images in the same time frame with less effort. In support of the above, TME-Analyzer uses a single algorithm rather than different algorithms for images with higher background signal, where the latter (as is the case for inForm) may yield incorrect image quantifications (Supplementary Table 4).The utility of TME-Analyzer was also favorable when compared to QuPath. This was due to two unique characteristics of TME-Analyzer. First, in TME-Analyzer, the analysis is saved for each image whereas for QuPath the analysis is saved per project. The former enables easy application of the analysis of a single image to a multitude of images, and easy modification of analysis parameters, as needed, per image. Consequently, TME-Analyzer but not QuPath analysis performed with MxIF images can easily be extended to MIBI-TOF images with minimal adjustments. Second, TME-Analyzer has a built-in image filtering module, while QuPath comes with a set of pre-defined filters only for pixel classification (e.g. tumor and stroma compartments), but not for cell-classification (see also Supplementary Table 4). Filtering for cell classification, however, is essential to compensate for microscopy artifacts prior to cell/nucleus intensity-based phenotyping. In TME-Analyzer this filtering module is used to perform size-based background correction, which enables the usage of a single algorithm with a single set of threshold values for all images. While random forest-based cell classification in QuPath yielded comparable results (Fig. 4), the machine learning analysis may remain difficult to interpret and modify.Furthermore, when analyzing MIBI-TOF images, we observed that the quantification of cell abundance, density and interspacing according to the TME-Analyzer was in agreement with DeepCell segmentation and other algorithms originally used to analyze the MIBI-TOF data set11. DeepCell has been specifically trained for the MIBI-TOF dataset that was used for validation purposes, and as a consequence performs better than StarDist in case of this dataset. StarDist, which is integrated in TME-Analyzer, has been specifically trained for immunofluorescence images20. While the shape of the segmented nuclei is clearly different between both methods, the number of cells and their relative distribution among cell phenotypes remain highly concordant. The latter observation is in agreement with recent work where random patch-based interrogation of tumor microenvironments was able to extract clinical relevance without any information of cell locations24. Indeed, we were able to demonstrate that TME-Analyzer performs well in analyzing images coming from either immunofluorescence or MIBI-TOF platforms. Moreover, TME-Analyzer is a modular tool enabling the option, when preferred, to incorporate any segmentation method or segmentations themselves from other methods. In addition to the TNBC analysis presented here, TME-Analyzer can interrogate multispectral images obtained for various different tumor pathologies, e.g., head and neck squamous-cell carcinoma25, metastatic urothelial carcinoma3,26, and glioma (non-published data).The composition of the micro-environment is of clinical value for various tumor types, including TNBC. For instance, the clinical value of the presence and location of tumor-infiltrating lymphocytes (TILs) has been well established for TNBC2,27,28,29. With the TME-Analyzer, we indeed observed higher densities of B cells, CD4 T cells, CD8 T cells and tumor associated macrophages (TAMs) in the center of inflamed but not non-inflamed tumors, being in line with previous reports using Mx-IF and IHC imaging2,30. Additionally, and as expected, phenotyped cells showed shorter distances to CD8 T cells in inflamed vs non-inflamed tumors.Besides the benchmarking of TME-Analyzer, we assessed whether TME-Analyzer was able to discriminate between patients with short- and long-term survival. To test this, we first identified maximally uncorrelated parameters and ranked them based on their prognostic potential. We then obtained a classifier using the top ten parameters. This approach applied to the validation cohort was indeed able to distinguish between patients with short- and long-term survivor. Interestingly, when testing different classification stringencies, we observed that the inclusion of the top six parameters or more could be validated. Additionally, 10 parameter classifiers obtained from a subset of the discovery cohort patients (16, 32 or 47 patients instead of full 63 patient cohort) could still be validated (data not shown). These two findings further demonstrate the robustness of our approach and justifying the selection of the top 10 parameters. Here, the parameter deconvolution to identify maximally uncorrelated parameters was a crucial step that has not been generally applied to multiplex images. While this approach has analogies in radiomics analysis31, dimensionality reduction in multiplex imaging data has been towards cell phenotyping and patient identification11,32,33, and not parameter decorrelation. Our classification approach also outperformed patient segregation based on T cell spatial phenotype from the pathological reports and T cell density quantified by TME-Analyzer, which are known prognosticators for TNBC2,11,34. When applied to an independent cohort, only the classifier retained its prognostic value. Furthermore, the classifier was also able to classify patients with survival less than 5 years with a sensitivity of 100% and a specificity of 85%. This was despite the completely different tissue processing and microscopy methods used in the two cohorts. While machine learning classifiers like random forest might further improve the performance, the averaging approach as followed in our study already yielded high accuracy. Our approach, in contrast to other classifiers, requires less computational time and allowed for easy interpretation of parameter values. The image analysis performed with TME-Analyzer with only slight adjustments between cohorts not only showed agreement with dedicated analysis, but also allowed identification of prognostic parameters. This demonstrates the robustness of TME-Analyzer and validates it as a versatile tool for accurately extracting information from tumor microenvironments.While our aim was to establish the TME-Analyzer as a tool for extracting immune contextures from images, one of the outcomes of our approach was the ranking of these parameters. The presented approach appeared more sensitive than the conventional Cox analysis, as according to Cox analysis only 6 out of 10 parameters were associated with survival, and only for the discovery cohort (Table 2). One argument potentially explaining the outperformance of our approach compared to other approaches, such as the Cox model, is the ability of our approach to overcome sampling bias. The Cox model depends on the composition of patients in a given cohort, whereas in the presented Monte Carlo approach small subsets of patients were re-sampled for each round of repetition. While this allows for limited sample size per round, repetition of this selection 1000 times enables enhanced statistical power. In the current example, given that there are about 3 × 1012 different 12-patient subsets (constituting shorter and longer survivors per round) of a 63-patient cohort, the 1000 subsets we analyzed during the 1000 repetitions effectively represent independent patient cohorts. Of the top 10 ranked contextual parameters observed for longer survivors, 1 parameter was large tumor area, 3 related to high densities of T cell subsets in different compartments, 3 to shorter distances among CD4 T cells, among TAMs and between NK and CD8 T cells, and 3 to longer distances between tumor cells and either TAMs, CD4 T cells, and other tumor cells (Fig. 5B). Notably, the majority of the parameters relate to intercellular distances, which underlines that besides the mere presence and location of CD8 T cells, intercellular distances among immune effector cells and/or tumor cells impact survival of TNBC patients.Interestingly, larger areas of tumor correlated with longer distances among tumor cells, which in turn may be related to reduced tumor cell aggregation, less small islands of tumor cells or tumor cells that are smaller in size, resulting in longer inter-cellular distances among tumor cells in long-lived when compared to short-lived patients. Furthermore, many prognostic parameters were strongly and positively interrelated, namely densities of CD4 T cells in tumor and stroma, density of CD8 T cells in stroma and distance from tumor cells to CD4 T cells. These parameters were negatively interrelated with distance among CD4 T cells and among TAMs. These findings extend earlier reports that point to the prognostic value of numbers and spatial orientation of these immune effector cells in TNBC2,11,34,35,36,37. Strikingly, most of the parameters from the classifier (6 out of 10) relate to cellular distances. In fact, for longer survivors, shorter distances were observed among TAMs and CD4 T cells, as well as between NK and CD8 T cells. The shorter distances may point to the presence of immune cell clusters that contain CD4 and CD8 T cells, TAMs and NK cells. It is of interest that particular CD4 T cell helper subsets might be critical components of such immune cell clusters, and may determine tumor evolution and responsiveness to immune checkpoint inhibitors. This would be in line with previous reports where we showed the presence of T follicular helper (Tfh) or T helper type-1 (Th1) cells in clusters found in either metastatic urothelial cancer or oral cavity cancer and their prognostic and predictive value, respectively3,25. Within these clusters, the presence of immune effector cells, such as NK cell and CD8 T cells, may ultimately facilitate anti-tumor responses38,39,40,41. Shorter distances observed among TAMs in longer survivors may suggest the presence of M1-like TAMs that may further support the anti-tumor activity of T cells42,43. Together, our findings in long-lived patients are in line with the presence of immune cell clusters reminiscent of tertiary lymphoid structures that have been reported to aid anti-tumor T cell responses. Specifically, the presence of tertiary lymphoid structures where immune cells cluster has been related to better clinical outcome in various solid tumors44, including TNBC11, ovarian cancer45, lung cancer46, metastatic urothelial cancer3 and oral cavity cancer1,25.The lower abundance and reduced clustering of immune effector cells in short-lived TNBC patients may point to active suppression of an anti-tumor T cell response. The non-inflamed TNBC can be subdivided into excluded and ignored phenotypes, each with unique features, regarding the involvement of extracellular matrix components and myeloid cells, respectively2. Tumor cell pathways, as for instance VEGF/TGFβ and WNT pathways2,47,48,49, may direct such features, and together with smaller tumor islands, dominate the suppressive TME in TNBC in shorter survivors. The beneficial self-clustering of macrophages, as discussed above for long-lived patients, might be dominated by tumor suppressive M1-like macrophages, the detrimental clustering of tumor cells and TAMs as observed for short-lived patients may be dominated by tumor promoting M2-like macrophages2,50,51,52,53. In fact, recent reports demonstrated that an M2-like gene signature2 and a higher TAM to T cell ratio30 are associated with worse prognosis, while not finding a direct relationship between abundance of TAMs and survival2,30, and argue that further studies are needed to define the exact composition of TAMs in relation to their contextual prognostic value in TNBC. Similarly, detrimental clustering of tumor cells and CD4 T cells as observed for short-lived patients could be due to CD4 T cells being predominantly composed of regulatory T cells (Treg) that are able to keep the activity of CD8 T cells in check54,55. The higher abundance of M2-like TAMs and/or Tregs would be in line with a micro-milieu in poor survivors that facilitates polarization of myeloid and T cells towards a suppressive phenotype.One limitation of our study is the likely difference between the discovery and validation cohorts when it comes to stage and treatment of TNBC. While the tissues from the discovery cohort came from treatment-naïve TNBC patients, tissues from the validation cohort also came from patients with TNBC recurrence following treatment. Additionally, following sampling, patients from both cohorts were subjected to different follow-up treatments. Therefore, the testing of the contextual classifier in larger TNBC patient cohorts, taking into account newer treatment modalities, such as anti-PD1 and sacituzumab govetican, would aid the translation of our findings towards clinical usage at the level of patient selection. Such clinical application would also benefit from further reduction in the number of markers studied to make it compatible with for instance a more standard immunofluorescence setup of 4 markers. Another limitation is the lack of characterization of TAM and CD4 T cell subsets. Further studies may include markers of such subsets, which could enhance our understanding and the development of possible interventions. A final limitation relates to the TME-Analyzer software, which is an.exe file that can be run on Windows computers, where images are read in their entirety into the RAM. The users with different operating systems can make use of the provided Python code instead, whereas its usage with large images and/or limited RAM is advised to be implemented with tiled analysis. Here, in addition to the available RAM limiting the size of the image that can be analyzed, an additional constraint lies with the StarDist module, which can handle images of sizes up to approximately 4 M pixels. It should be noted that the image analysis field is highly dynamic with a continuous development of software tools both commercial (e.g., HALO, Indica Labs, USA) and open-source (e.g., the community support of QuPath). Moreover, in addition to random forest, artificial neural network and K nearest neighbor based machine learning cell phenotyping algorithms often incorporated into new software tools, high accuracy has also recently been reported for convoluted neural network based phenotyping from multiplex images56.In conclusion, the TME-Analyzer captures the heterogeneity of the immune microenvironment of patient’s tumors with a new WYSIWYG interface and real-time interactive analysis. This tool enables extraction and analysis of data from multiple platforms in a uniform, high-throughput and interactive manner, and showed high concordance with two benchmark software tools. TME-Analyzer demonstrated enhanced accuracy as well as utility regarding immune phenotyping. Notably, TME-Analyzer demonstrated its value in building a contextual classifier for survival of TNBC patients. Besides showing the diverse applicability of this software, our findings revealed the impact of inter-immune cell distances towards survival of TNBC patients as well as its outperformance of recognized prognostic parameters, such as inflammation and stromal T cell densities. TME-Analyzer and the clinical analysis we present here can be easily and quickly applied to large cohorts of patients irrespective of imaging modality used, making it a powerful tool for studying TME spatial architecture.
