CycloNET pipelineFigure 1A shows CycloNET’s overall workflow. The input data are raw fluorescent images obtained through cyclic immunofluorescent microscopy. The computational pipeline collates acquired multichannel images, performs image registration, detects individual cells, and presents summary statistics. In the current work, we used images generated by the fast analytical screening technique (FAST) as previously reported11,17; FAST uses custom-designed antibody probes to cyclicly image 20–40 molecular markers within an hour11,17 (Fig. 1B). A given specimen underwent rounds of imaging and quenching cycles; at each cycle, several FOVs were captured, resulting in up to 5 gigapixels of visual information per sample. We implemented a graphical user interface to streamline image analysis (Fig. 1C). Under the hood, it was equipped with (i) a preprocessor that automatically grouped images by cycle and FOV, (ii) a registration module that detected translations between imaging cycles, and (iii) a pre-trained neural network for task-specific cell segmentation.Fig. 1Overview of CycloNET analysis pipeline. (A) Schematic for computationally assisted biopsy analysis. After immunofluorescent images are collected via cyclic staining, CycloNET automatically aligns images from different cycles, produces a single-cell segmentation mask, and generates 2-dimensional representations for multiplexed biomarker data. (B) Cyclic staining protocol. A sample is stained, and several fields of view, m, are captured by a microscope. The fluorescent signal is then quenched, and the background “quenched signal” is imaged. This process is repeated N times to produce the full dataset. (C) Analysis user interface. A simple user interface was developed to group sample images and execute the CycloNET modules described above.Image registrationBetween fluorescent staining and destaining cycles, sample slides were manually removed and remounted onto the microscope, introducing spatial offsets. Though negligible for tissue-level assessments, these shifts required careful correction for accurate single-cell metrics. Standard imaging software for spatial alignment faced challenges, including (i) algorithms that were suboptimal for large image sets and (ii) errors in registration due to sample debris, low cellularity, or low signal-to-noise ratio (SNR). We addressed these drawbacks in our custom-designed processing pipeline, optimized for cyclic imaging.Pre-processingFigure 2A illustrates the CycloNET image registration algorithm. The algorithm processes image datasets derived from several staining cycles (denoted as N cycles). Multiple fields of view (FOVs) are collected for each staining cycle to capture as many cells as possible. In this scenario, each FOV tile set can be represented as FOVi,j, where i indicates an individual field of view (i = 1, 2, …, M) and j indicates the particular cycle number (j = 1, 2, …, N). We found that images from any two cycles had similar spatial offsets across all FOVs, allowing us to align full cycles by using a single FOV per cycle. Thus, the FOV numbers to be processed for alignment is reduced from (M × N × C) to just (N × C), where C is the number of fluorescent channels used in microscopy. To further minimize dataset size and improve accuracy in low-signal images, the C-channel fluorescent signals were normalized and collapsed by maximum projection, resulting in a set of N images. By using the maximum projection across individual channels, we enhanced the visibility of cells, as some cells might be visible in one channel and not another, providing more “anchoring points” for accurate alignment. Finally, a reference is selected among these N collapsed images, to which all other cycles are aligned. These are labeled in Fig. 2A as “Reference” and “Cycle images.”Fig. 2Correction of inter-cycle image offsets. (A) Image registration algorithm. A single specimen undergoes N imaging cycles, each cycle producing M FOVs. One cycle is chosen as the “reference” to which all other cycles are aligned. For a single FOV, cycles are collapsed by maximum projection, and a threshold mask is computed to roughly determine the presence of cell-sized objects. Each image is then downsampled and randomly cropped. Finally, the masked cross-correlation is computed over t trials, and the mean shift is returned. (B) Visual examples for cycle registration. The top row shows the original data after maximum projection. The second row demonstrated the threshold mask used to search for cell-sized objects. The heat maps in the third-row show where the correlation between the current cycle and the reference image is highest. The final row shows the aligned images. Red circles highlight a cell cluster of interest. In the final row, this cluster is in the same position for all cycles.The translations between the N images were calculated by masked cross-correlation. This strategy reduced the impact of cellular debris by focusing only on cell-shaped foreground objects specified by the cell mask (see “Methods and materials” for details). At the time of cross-correlation calculation, images were also downsampled two-fold and randomly cropped to further reduce dataset size. This process was repeated for some t trials, with different random crops at each trial. Finally, the mean shift (after removing outliers) is calculated and assigned to all FOVs in a given staining cycle.Figure 2B shows snapshots of the image registration process. The maximum projection across individual channels increased the number of cells available as anchoring points for the cross-correlation algorithm. One such anchoring point is indicated by red circles, demonstrating how a cluster of cells shifts to overlap with the reference cycle after alignment. We tested our algorithm on a set of five human specimens (FOVs per sample: 20, 24, 35, 30, 36). On average, a single field of view with seven imaging cycles was aligned in 42 s, while a full specimen containing ~ 20 FOVs typically aligns in under 6 min using an Intel® Core™ i7-6850 K processor. Alignment failed in 5% of 142 FOVs, typically in FOVs with very low contrast in the reference signal or low cellularity.Cell segmentation and validationFollowing alignment, CycloNET generated segmentation masks for individual cells by applying a pre-trained neural network. To detect immune cells in the sample, the algorithm was trained on image stacks consisting of CD45 and nucleus (DAPI) channels, as described in Fig. 3A. Using a modified U-Net structure, we designed a neural network to produce a 2-dimensional array where each pixel was labeled as either: (i) background, (ii) cell interior, or (iii) cell boundary25 (Fig. 3A). The encoded features used to construct segmentation masks were regularized by secondary auto-encoding and cell-counting tasks26, resulting in a loss function,$$L = {L_{seg}}+{\lambda _c}\cdot{L_{count}} + {\lambda _a}\cdot{L_{auto}} + \beta \cdot{L_{reg}} = ({L_{ce}} + \alpha \cdot {L_{jacc}}) + {\lambda _c}\cdot{L_{count}} + {\lambda _a}\cdot{L_{auto}} + \beta \cdot{L_{reg}}$$where the segmentation loss, Lseg, was split into the cross entropy loss (Lce) for pixel-by-pixel classification and the per-class Jaccard loss metric (Ljacc). Lcount and Lauto represent the mean squared error loss metrics for the cell counting and autoencoding tasks, respectively. Finally, Lreg represents the L2 regularization loss applied to all weights in the network, and α, λc, λa, β assign weights to each loss type. The network was trained using the CD45 and nuclear stains for 17 human specimen FOVs. At the end of the training, the average F1 score among all pixel classes was 0.94. A single FOV was scanned for CD45 + immune cells in 17.9 s (n = 134).Figure 3B shows the comparison between manual and CycloNET segmentations. Defining the cell boundary class was found to be crucial for separating cells in contact with each other. To evaluate the functional performance of our segmentation network, we interrogated the correlation between biomarker values after automated and manual segmentation on the test set. Among all 134 image FOVs in our test group, the correlation for the number of cells counted was 0.92, with a 0.11 false positive and 0.13 false negative cell detection rate. In single cells, the correlation in cell area was 0.936 (Fig. 3C), and the correlation for mean fluorescent intensity was greater than 0.996 for all markers (Fig. 3D).Fig. 3Single-cell segmentation by a neural network. (A) Neural network architecture. The CD45 and DAPI (nuclear) stains were used as the input to a modified U-Net architecture to produce a 3-class segmentation mask. The bottleneck features were also used for a secondary auto-encoding task. (B) Segmentation boundaries. Green and red boundaries, respectively, represent cell boundary pixels by manual and CycloNET segmentations. Yellow represents the overlap between the two methods. (C) Functional comparisons. (Left) A comparison is made between the areas of manually segmented cells and neural network-predicted areas. (Right) Similarly, the mean fluorescent signal per cell was compared using manual and algorithmic segmentation.Single-cell profiling and clustering in patient samplesWe finally applied the entire CycloNET process to an example patient dataset. The input data were taken from seven cycles of staining (Fig. 4A, top row), with each cycle probing three protein markers found in immune cells. The algorithm corrected spatial shifts between cycles (Fig. 4A, middle) and segmented immune cells based on CD45 and nucleus stains (Fig. 4A, bottom). Single-cell masks were then applied to the aligned image stacks to extract marker expression in individual cells (Fig. 4B).Fig. 4Fluorescence image stack to single-cell metrics. (A) We visualize the full image stack of fluorescent signals as single-cell data is extracted. (Top row) For each cycle, we visualize 3 biomarker signals in an RGB image. A single immune cell is highlighted by a red circle, which is at different positions in each imaging cycle. (Middle row) The image stacks are aligned for inter-cycle translations. (Bottom row) Among the many cells seen in the window, immune cells are segmented by the neural network. Each cell is highlighted by a white border and a numeric ID. (B) The fluorescent signal within each cell’s interior is mean aggregated, and a z score is calculated for all cells. The heat map shows each cell’s relative signal strength for each of the 18 markers.To further aid in interpreting the large single-cell dataset, we incorporated dimensionality reduction and an unsupervised clustering protocol. Dimensionality reduction was performed by the uniform manifold approximation and projection (UMAP) algorithm, and potential subpopulation clusters were identified by the Hierarchical density-based spatial clustering of applications with noise (HDBSCAN) algorithm (see Methods for details)27,28. Figure 5 shows an exemplary profiling result of an HNSCC FNA specimen. CycloNET identified 1528 immune cells in this sample and generated single-cell data for 17 protein markers. After dimensionality reduction, a total of 9 distinct clusters were assigned by the HDBSCAN algorithm (Fig. 5A). Each cluster was found to have a unique biomarker profile based on average intensities (Fig. 5B).Fig. 5Patient data visualization. (A) Dimensionality reduction. The multi-dimensional single-cell dataset generated from a patient sample was projected onto a two-dimensional manifold by the UMAP algorithm. Cells were clustered by the unsupervised HBDSCAN algorithm. (B) Mean marker intensities were calculated for each cluster, and relative marker intensities were visualized in a heat map. (C) Common immune cell types were overlaid to individual cells in existing clusters. (E) CycloNET subclassified single-cell types according to their distinct phenotypes. For a given immune cell type, its relative presence among unsupervised clusters is summarized. (D) Example images show macrophages and neutrophils belonging to different unsupervised clusters.We investigated the detected clusters by identifying common immune cell phenotypes (Table 2). Neutrophils and macrophages were among the most abundant, while natural killer and dendritic cells were the least present (Fig. 5C). Some of the manually defined phenotype assignments (e.g., CD4 + T cells) closely matched existing clusters, while others spread across different clusters (e.g., macrophages). A close look revealed different phenotypes in a nominal cell type (Fig. 5D). For example, neutrophils were found in clusters 1 and 3, separated mainly by p16 expression. Similarly, a sizable percentage of macrophage cells were found to differ in CD11c expression (Fig. 5E). Overall, the results demonstrate CycloNET’s capability of streamlining multi-dimensional single-cell phenotyping.Table 2 Marker combinations for immune cell phenotyping.
