In this section, we describe DeepDeWedge and evaluate its effectiveness on real and simulated data.The DeepDeWedge algorithmDeepDeWedge takes a single tilt series t as input and produces a denoised, missing-wedge-filled tomogram. The method can also be applied to a dataset containing multiple (typically up to 10) tilt series from different samples of the same type, for example, sections of different cells, which share the same cell type. DeepDeWedge consists of the following three steps, illustrated in Fig. 1:
1.
Step: Data preparation. First, split the tilt series t into two sub-tilt-series t0 and t1 with the even/odd split or the frame-based split. The even/odd split partitions the tilt series into even and odd projections based on their order of acquisition. The frame-based split can be applied if the tilt series is collected using dose fractionation and entails averaging only the even and odd frames recorded at each tilt angle. We recommend the frame-based splitting approach whenever possible. After splitting, we have two sub-tilt series.
Next, reconstruct both sub-tilt-series independently with FBP and apply CTF correction. This yields a pair of two coarse reconstructions \(({\rm{FBP}}\,({{{\bf{t}}}}^{0}),\,{\rm{FBP}}\,({{{\bf{t}}}}^{1}))\) of the sample’s 3D density. Finally, use a 3D-sliding window procedure to extract N overlapping cubic sub-tomogram pairs \({\{({{{{\bf{v}}}}^{0}_{i}},{{{{\bf{v}}}}^{1}_{i}})\}}_{i=1}^{N}\) from both FBP reconstructions. The size and number N of these sub-tomogram cubes is a hyperparameter. Experiments on synthetic data presented in the supplementary information suggest that larger sub-tomograms tend to yield better results up to a point.
2.
Step: Model fitting. Fit a randomly initialized network fθ, we use a U-Net14, with weights θ by repeating the following steps until convergence:
(a)
Generate model inputs and targets: For each of the sub-tomogram pairs \({\{({{{{\bf{v}}}}^{0}_{i}},{{{{\bf{v}}}}^{1}_{i}})\}}_{i=1}^{N}\) generated in Step 1, sample a rotation \({{{\bf{R}}}}_{{\varphi }_{i}}\) parameterized by Euler angles φi from the uniform distribution on the group of 3D rotations, and construct a model input \({\tilde{{{\bf{v}}}}}^{{{\rm{0}}}}_{i,{\varphi }_{i}}\) and target \({{{{\bf{v}}}}^{{{\rm{1}}}}_{i,{\varphi }_{i}}}\) by applying the rotation \({{{\bf{R}}}}_{{\varphi }_{i}}\) to both sub-tomograms and adding an artificial missing wedge to the rotated sub-tomogram \({{{\bf{R}}}}_{{\varphi }_{i}}{{{{\bf{v}}}}^{0}_{i}}\), as shown in the centre panel of Fig. 1. The missing wedge is added by taking the Fourier transform of the rotated sub-tomograms and multiplying them with a binary 3D mask M that zeros out all Fourier components that lie inside the missing wedge. Repeating this procedure for all sub-tomogram pairs \({\{({{{{\bf{v}}}}^0_{i}},{{{{\bf{v}}}}^1_{i}})\}}_{i=1}^{N}\) yields a set of N triplets consisting of model input, target sub-tomogram, and angle \({\{({\tilde{{{\bf{v}}}}}^{0}_{i,{\varphi }_{i}},{{{{\bf{v}}}}^{1}_{i,{\varphi }_{i}}},{\varphi }_{i})\}}_{i=1}^{N}\).
(b)
Update the model: Update the model weights θ by performing gradient steps to minimize the per-sample loss$$\ell \left({{\boldsymbol{\theta }}},i\right)={\left\Vert \left({{\bf{M}}}{{{\bf{M}}}}_{{\varphi }_{i}}+2{{{\bf{M}}}}^{C}{{{\bf{M}}}}_{{\varphi }_{i}}\right){{\bf{F}}}\left({{{\rm{f}}}}_{\theta }\left({\tilde{{{\bf{v}}}}}^{{{\rm{0}}}}_{i,{\varphi }_{i}}\right)-{{{{\bf{v}}}}^{{{\rm{1}}}}_{i,{\varphi }_{i}}}\right)\right\Vert }_{2}^{2}.$$
(2)
Here, \({{{\bf{M}}}}_{{\varphi }_{i}}\) is the rotated version of the wedge mask M and MC ≔ I − M is the complement of the mask. For the gradient updates, we use the Adam optimizer15 and perform a single pass through the N model input and target sub-tomograms generated before.
(c)
Update the missing wedges of the model inputs: For each, i = 1, …, N, update the missing wedge in the i-th sub-tomogram \({{{{\bf{v}}}}^0_{i}}\) produced in Step 1 by passing it through the current model fθ, and inserting the predicted content of the missing wedge, as follows$${{{{\bf{v}}}}^0_{i}}\leftarrow {{{\bf{F}}}}^{-1}\left({{\bf{M}}}{{\bf{F}}}{{{{\bf{v}}}}^0_{i}}+{{{\bf{M}}}}^{C}{{\bf{F}}}{{{\rm{f}}}}_{\theta }\left({{{{\bf{v}}}}^0_{i}}\right)\right).$$
(3)
In the next input-target generation step, the model inputs \({\{{\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}}\}}_{i=1}^{N}\) are constructed using the updated sub-tomograms. We do not update the missing wedges of the sub-tomograms \({\{{{{{\bf{v}}}}^1_{i}}\}}_{i=1}^{N}\) used to generate the model targets, since their missing wedges are masked out in the per-sample loss, c.f. Eq. (2), and therefore play no role in model fitting.
3.
Step: Tomogram refinement. Pass the original, non-updated sub-tomograms \({\{{{{{\bf{v}}}}^0_{i}}\}}_{i=1}^{N}\) through the fitted model \({f}_{{{{\boldsymbol{\theta }}}} ^{*}}\) from Step 2, and reassemble the model outputs \({\{{f}_{{{{\boldsymbol{\theta }}}} ^{*}}\left({{{{\bf{v}}}}^0_{i}}\right)\}}_{i=1}^{N}\) into a full-sized tomogram. Repeat the same for the sub-tomograms \({\{{{{{\bf{v}}}}^{1}_{i}}\}}_{i=1}^{N}\). Finally, average both reconstructions to obtain the final denoised and missing-wedge corrected tomogram.
Fig. 1: Illustration of DeepDeWedge.For simplicity, we show the 2D tilt series images as 1D Fourier slices and all 3D tomograms as 2D objects in the Fourier domain. Recall that tilt series images, tomograms, and sub-tomograms are objects in the image domain. The figure shows the splitting approach where the tilt series is split into even and odd projections.Motivation for the three steps of the algorithmIn Step 1, we split the tilt series into two disjoint parts to obtain measurements with independent noise. As the noise on the individual projections or frames is assumed to be independent, the reconstructions FBP (t0) and FBP (t1) are noisy observations of the same underlying sample with independent noise terms. Those are used in Step 2 for the self-supervised Noise2Noise-inspired loss.Tilt series splitting is also used in popular implementations of Noise2Noise-like denoising methods for cryo-ET11,12,13. The frame-based splitting procedure was proposed by Buchholz et al.11, who found that it can improve the performance of Noise2Noise-like denoising over the even/odd split.Step 2 of DeepDeWedge is to fit a neural network to perform denoising and missing wedge reconstruction, for which we have designed a specific loss function ℓ (Eq. (2)). We provide a brief justification for the loss function here; a detailed theoretical motivation is presented in the following section.As the masks \({{\bf{M}}}{{{\bf{M}}}}_{{\varphi }_{i}}\), and \({{{\bf{M}}}}^{C}{{{\bf{M}}}}_{{\varphi }_{i}}\) are orthogonal, the loss value ℓ(i, θ) can be expressed as the sum of two the two terms \({\Vert {{\bf{M}}}{{{\bf{M}}}}_{{\varphi }_{i}}{{\bf{F}}}({{{\rm{f}}}}_{\theta }({\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}})-{{{{\bf{v}}}}^1_{i,{\varphi }_{i}}})\Vert }_{2}^{2}\), and \({\Vert 2{{{\bf{M}}}}^{C}{{{\bf{M}}}}_{{\varphi }_{i}}{{\bf{F}}}({{{\rm{f}}}}_{\theta }({\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}})-{{{{\bf{v}}}}^1_{i,{\varphi }_{i}}})\Vert }_{2}^{2}\). The first summand, i.e. \({\Vert {{\bf{M}}}{{{\bf{M}}}}_{{\varphi }_{i}}{{\bf{F}}}({{{\rm{f}}}}_{\theta }({\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}})-{{{{\bf{v}}}}^1_{i,{\varphi }_{i}}})\Vert }_{2}^{2}\), is the squared L2 distance between the network output and the target sub-tomogram \({{{{\bf{v}}}}^1_{i,{\varphi }_{i}}}\) on all Fourier components that were not masked out by the two missing wedge masks M and \({{{\bf{M}}}}_{{\varphi }_{i}}\). As we assume the noise in the target to be independent of the noise in the input \({\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}}\), minimizing this part incentivizes the network to learn to denoise these Fourier components. This is inspired by the Noise2Noise principle.The second summand, i.e. \({\Vert 2{{{\bf{M}}}}^{C}{{{\bf{M}}}}_{{\varphi }_{i}}{{\bf{F}}}({{{\rm{f}}}}_{\theta }({\tilde{{{\bf{v}}}}}_{i,{\varphi }_{i}}^0)-{{{{\bf{v}}}}^1_{i,{\varphi }_{i}}})\Vert }_{2}^{2}\), incentivizes the network fθ to restore the data that we artificially removed with the mask M, and can be considered as a Noisier2Noise-like loss16 (see Supplementary Information 1 for background). For this part, it is important that we rotate both volumes, which moves their original missing wedges to a new, random location.In the last part of Step 2, we correct the missing wedges of the sub-tomograms \({\{{{{{\bf{v}}}}^{0}_{i}}\}}_{i=1}^{N}\) using the current model fθ. Therefore, as the model fitting proceeds, the model inputs \({\{{\tilde{{{\bf{v}}}}}^0_{i,{\varphi }_{i}}\}}_{i=1}^{N}\) will more and more resemble sub-tomograms with only one missing wedge, i.e. the one we artificially remove from the partially corrected sub-tomograms \({\{{{{{\bf{v}}}}^0_{i}}\}}_{i=1}^{N}\) with the mask \({{{\bf{M}}}}.\) This is a heuristic which is supposed to help the model perform well on the original sub-tomograms \({\{{{{{\bf{v}}}}^{0}_{i}}\}}_{i=1}^{N}\) and \({\{{{{{\bf{v}}}}^1_{i}}\}}_{i=1}^{N}\), which have only one missing wedge and which we use as model inputs in Step 3. An analogous approach for Noisier2Noise-based image denoising was proposed by Zhang et al.17.In Step 3, we use the fitted model from Step 2 to produce the final denoised and missing wedge-filled tomogram. To use all the information contained in the tilt series t for the final reconstruction, we separately refine the sub-tomograms from the FBP reconstructions \({\rm{FBP}}\left({{{\bf{t}}}}^{0}\right)\) and \({\rm{FBP}}\left({{{\bf{t}}}}^{1}\right)\) of the sub-tilt-series t0 and t1, and average them. For this, we apply a special normalization to the model inputs, which is described in Supplementary Information 2.Theoretical motivation for the loss functionHere, we present a theoretical result that motivates the choice of our per-sample loss ℓ defined in Eq. (2). The discussion in this section does not consider the heuristic of updating the missing wedges of the model inputs, which is part of DeepDeWedge’s model fitting step. Moreover, we consider an idealized setup that deviates from practice to motivate our loss.We assume access to data that consists of many noisy, missing-wedge-affected 3D observations of a fixed ground truth 3D structure \({{{\bf{v}}}}^{*}\in {{\mathbb{R}}}^{N\times N\times N}\). Specifically, data is generated as two measurements (in the form of volumes) of the unknown ground-truth volume of interest, \({{\bf{v}}}^{*}\), as$${{{\bf{v}}}}^{0}={{{\bf{F}}}}^{-1}{{\bf{M}}}{{\bf{F}}}\left({{{\bf{v}}}}^{*}+{{{\bf{n}}}}^{0}\right),\qquad {{{\bf{v}}}}^{1}={{{\bf{F}}}}^{-1}{{\bf{M}}}{{\bf{F}}}\left({{{\bf{v}}}}^{*}+{{{\bf{n}}}}^{1}\right),$$
(4)
where \({{{\bf{n}}}}^{0},{{{\bf{n}}}}^{1}\in {{\mathbb{R}}}^{N\times N\times N}\) are random noise terms and \({\bf{M}} \in \{0,1\}^{N \times N \times N}\) is the missing wedge mask. From the first measurement, we generate a noisier observation \({\tilde{{{\bf{v}}}}}^{0}={{{\bf{F}}}}^{-1}\tilde{{{\bf{M}}}}{{\bf{F}}}{{{\bf{v}}}}^{0}\) by applying a second missing wedge mask \(\tilde{{{\bf{M}}}}\). The noisier observation has two missing wedges: the wedge introduced by the first and the wedge introduced by the second mask. We assume that the two masks follow a joint and symmetric distribution, e.g. that for each mask, a random wedge is chosen uniformly at random. Figure 2 illustrates four data points in 2D.Fig. 2: Illustration of a single ground-truth structure \({{\bf{v}}}^{*}\), and four model input-target pairs corresponding to four different random positions of the original (blue) and additional (red) missing wedge for the setup of our theoretical motivation.For simplicity, we visualize a 2D structure with no additive noise and random in-plane rotations. The inset boxes in each patch show the absolute values of the Fourier transforms of the images and regions that are zeroed out by the missing wedge masks.We then train a neural network fθ to minimize the loss$${{\rm{L}}}({{\boldsymbol{\theta }}})={{\mathbb{E}}}_{{{\bf{M}}},\tilde{{{\bf{M}}}},{{{\bf{n}}}}^{0},{{{\bf{n}}}}^{1}}\left[{\left\Vert \left(\tilde{{{\bf{M}}}}{{\bf{M}}}+2{\tilde{{{\bf{M}}}}}^{C}{{\bf{M}}}\right){{\bf{F}}}\left({{{\rm{f}}}}_{\theta }\left({\tilde{{{\bf{v}}}}}^{0}\right)-{{{\bf{v}}}}^{1}\right)\right\Vert }_{2}^{2}\right],$$
(5)
where the expectation is over the random masks and the noise terms. Note that this resembles training on infinitely many data points, with a very similar loss to the original loss (2); the main difference is that in the original loss, the volume is rotated randomly but the mask M is fixed, while in the setup considered in this section, the volume is fixed but the masks M and \(\tilde{{{\bf{M}}}}\) are random.After training, we can use the network to estimate the ground-truth volume by applying the network to another noisy observation \({\tilde{{{\bf{v}}}}}^{0}\). The following proposition, whose proof can be found in Supplementary Information 4, establishes that this is equivalent to training the network on a supervised loss to reconstruct the input \({\tilde{{{\bf{v}}}}}^{0}\), provided the two masks are non-overlapping.Proposition 1Assume that the noise n1 is zero-mean and independent of the noise n0, and of the masks \(({{\bf{M}}},\;\tilde{{{\bf{M}}}})\), and assume that the noise n0 is also independent of the masks. Moreover, assume that the joint probability distribution P of the missing wedge masks M and \(\tilde{{{\bf{M}}}}\) is symmetric, i.e. \({{\rm{P}}}({{\bf{M}}},\;\tilde{{{\bf{M}}}})={{\rm{P}}}(\tilde{{{\bf{M}}}},\;{{\bf{M}}})\), and that the missing wedges do not overlap. Then the loss L is proportional to the supervised loss$${{\rm{R}}}({{\boldsymbol{\theta }}})={{\mathbb{E}}}_{{{\bf{M}}},\tilde{{{\bf{M}}}},{{{\bf{n}}}}^{0}}\left[{\left\Vert {{{\rm{f}}}}_{0}\left({\tilde{{{\bf{v}}}}}^{0}\right)-{{{\bf{v}}}} ^{ * }\right\Vert }_{2}^{2}\right],$$
(6)
i.e. L (θ) = R(θ) + c, where c is a numerical constant independent of the network parameters θ.In practice, we do not apply our approach to the problem of reconstructing a single fixed structure \({{\bf{v}}}^{*}\) from multiple pairs of noisy observations with random missing wedges. Instead, we consider the problem of reconstructing several unique biological samples using a small dataset of tilt series. To this end, we fit a model with an empirical estimate of risk similar to the one considered in Proposition 1. We fit the model on sub-tomogram pairs extracted from the FBP reconstructions of the even and odd sub-tilt series, which exhibit independent noise. Moreover, as already mentioned above, in the setup of our algorithm, the two missing wedge masks M and \(\tilde{{{\bf{M}}}}\) themselves are not random. However, as we randomly rotate the model input sub-tomograms during model fitting, the missing wedges appear at a random location with respect to an arbitrary fixed orientation of the sub-tomogram.Related workDeepDeWedge builds on Noise2Noise-based denoising methods and is related to the denoising and missing wedge-filling method IsoNet4. We first discuss the relation between DeepDeWedge and Noise2Noise-based methods, which do not reconstruct the missing wedge.Noise2Noise-based denoising algorithms for cryo-ET as implemented in CryoCARE11 or Warp13 take one or more tilt series as input and return denoised tomograms. A randomly initialized network fθ is fitted for denoising on sub-tomograms of FBP reconstructions FBP(t0) and FBP(t1) of sub-tilt-series t0 and t1 obtained from a full tilt series t. The model is fitted by minimizing a loss function (typically the mean-squared error), between the output of the model fθ applied to one noisy sub-tomogram and the corresponding other noisy sub-tomogram. The fitted model is then used to denoise the two reconstructions FBP(t0) and FBP(t1), which are then averaged to obtain the final denoised tomogram. Contrary to those denoising methods, DeepDeWedge fits a network not only to denoise, but also to fill the missing wedge.Our method is most closely related to IsoNet, which can also do denoising and missing wedge reconstruction. IsoNet takes a small set of, say, one to ten tomograms and produces denoised, missing-wedge corrected versions of those tomograms. Similar to DeepDeWedge, a randomly initialized network is fitted for denoising and missing wedge reconstruction on sub-tomograms of these tomograms, however, the fitting process is different: Inspired by Noisier2Noise, the model is fitted on the task of mapping sub-tomograms that are further corrupted with an additional missing wedge and additional noise onto their non-corrupted versions. After each iteration, the intermediate model is used to predict the content of the original missing wedges of all sub-tomograms. The predicted missing wedge content is inserted into all sub-tomograms, which serve as input to the next iteration of the algorithm.Different to IsoNet, our denoising approach is Noise2Noise-like, as in CryoCARE. This leads to better denoising performance, as we will see later, as well as requiring fewer assumptions and no hyperparameter tuning. Specifically, Noisier2Noise-like denoising requires knowledge of the noise model and strength (see Supplementary Information 1). As this knowledge is typically unavailable, Liu et al.4 propose approximate noise models from which the user has to choose.After model fitting, the user must manually decide which iteration and noise level gave the best reconstruction. Thus, IsoNet’s Noisier2Noise-inspired denoising approach requires several hyperparameters for which good values exist but are unknown. Therefore, IsoNet requires tuning to achieve good results. Our denoising approach introduces no additional hyperparameters and does not require knowledge of the noise model and strength.The main commonality between DeepDeWedge and IsoNet is the Noisier2Noise-like mechanism for missing wedge reconstruction, which consists of artificially removing another wedge from the sub-tomograms and fitting the model to reconstruct the wedge.Moreover, like IsoNet, DeepDeWedge fills in the missing wedges of the model inputs. In IsoNet, one also has to fill in the missing wedges of the model targets. This is necessary because, contrary to our loss ℓ form Eq. (2), IsoNet’s loss function does not ignore the targets’ missing wedges via masking in the Fourier domain.Another line of work related to DeepDeWedge considers domain-specific tomographic reconstruction methods that incorporate prior knowledge of biological samples into the reconstruction process to compensate for missing wedge artefacts, for example, ICON18, and MBIR19. For an overview of such reconstruction methods, we refer to the introductory sections of works by Ding et al.20 and Bohning et al.21. Liu et al.4 found that IsoNet outperforms both ICON and MBIR.DeepDeWedge is also conceptually related to un-trained neural networks, which reconstruct an image or volume based on fitting a neural network to data22,23. Un-trained networks also only rely on fitting a neural network to given measurements. However, they rely on the bias of convolutional neural networks towards natural images24,25, whereas in our setup, we fit a network on measurement data to be able to reconstruct from the same measurements.For cryo-EM-related problems other than tomographic reconstruction, deep learning approaches for missing data reconstruction and denoising have also recently been proposed. Zhang et al.26 proposed a method to restore the state of individual particles inside tomograms, and Liu et al.27 proposed a variant of IsoNet to resolve the preferred orientation problem in single-particle cryo-EM27.Experiments on purified Saccharomyces cerevisiae 80S ribosomesIn this and the following experiments, we compare DeepDeWedge to (a re-implementation of) CryoCARE, IsoNet, and a two-step approach of fitting IsoNet to tomograms denoised with CryoCARE. The two-step approach, which we call CryoCARE + IsoNet, is considered a state-of-the-art pipeline for denoising and missing wedge reconstruction.The first dataset we consider is the commonly used EMPIAR-10045 dataset, which contains seven tilt series collected from samples of purified S. cerevisiae 80S Ribosomes.Figure 3 shows a tomogram refined with IsoNet, CryoCARE + IsoNet and DeepDeWedge using the even/odd tilt series splitting approach. Note that while IsoNet’s built-in Noisier2Noise-like denoiser removes some of the noise contained in the FBP reconstruction, its performance is considerably worse than that of Noise2Noise-based CryoCARE. This can be seen by comparing to the result of applying IsoNet with a disabled denoiser to a tomogram denoised with CryoCARE. DeepDeWedge produces a denoised and missing-wedge-corrected tomogram similar to the CryoCARE + IsoNet combination. The main difference between these reconstructions is that the DeepDeWedge-refined tomogram has a smoother background and contains fewer high-frequency components.Fig. 3: Slices through 3D reconstructions of a tomogram containing purified S. cerevisiae 80S ribosomes (EMPIAR-10045, tomogram 5).The red lines in each slice indicate the positions of the remaining two slices. We also show the central x–z-slice through the logarithm of the magnitude of the Fourier transform of each reconstruction.Regarding missing wedge correction, we find the performance of DeepDeWedge and IsoNet to be similar. In slices parallel to the x–z-plane, where the effects of the missing wedge on the FBP reconstruction are most prominent, both IsoNet and DeepDeWedge reduce artefacts and correct artificial elongations of the ribosomes. The central x–z-slices through the reconstructions’ Fourier transforms confirm that all methods but FBP fill in most of the missing wedge.Experiments on flagella of Chlamydomonas
reinhardtii
Next, we evaluate DeepDeWedge on another real-world dataset of tomograms of the flagella of C. reinhardtii, which is the tutorial dataset for CryoCARE. Since we observed above that IsoNet performs better when applying it to CryoCARE-denoised tomograms, we compare only to CryoCARE + IsoNet. In addition, we investigate the impact of splitting the tilt-series into even and odd projections versus using the frame-based split for DeepDeWedge and CryoCARE + IsoNet.The reconstructions obtained with all methods are shown in Fig. 4. We find that when using the even/odd-based split, CryoCARE + IsoNet produces a crisper reconstruction than DeepDeWedge (see zoomed-in region). This may be because the model inputs and targets of IsoNet stem from denoised FBP reconstruction of the full tilt series in which the information is more densely sampled than in the sub-tomograms of the even and odd FBP to the reconstructions used for fitting DeepDeWedge.Fig. 4: Slices through 3D reconstructions of the flagella of C. reinhardtii when using different reconstruction methods.The red lines in each slice indicate the positions of the remaining two slices. We also show the central x–z-slice through the logarithm of the magnitude of the Fourier transform of each reconstruction.When using the frame-based splitting method, in which DeepDeWedge also operates on the more densely sampled FBP reconstruction, DeepDeWedge removes more noise than CryoCARE + IsoNet and produces higher contrast, which is most noticeable in the x–z-slice. Especially in background areas, the DeepDeWedge reconstruction has fewer high-frequency components and is smoother. Therefore, the CryoCARE + IsoNet reconstruction is slightly more faithful to the FBP reconstruction but is also noisier.The central x–z-slices through the reconstructions’ Fourier transforms’ indicate that both CryoCARE + IsoNet and DeepDeWedge fill in most of the missing wedge. Both methods fix the missing-wedge-caused distortions of the microtubules exhibited by the FBP reconstruction, as seen in the x–z-slices. DeepDeWedge reconstructs more of the flagellas’ outer parts.Experiments on the ciliary transit zone of C. reinhardtii
Finally, we apply DeepDeWedge to an in situ dataset. We chose EMPIAR-1107828, which contains a tilt series of the ciliary transit zone of C. reinhardtii. The crowded cellular environment and low contrast and SNR of the tilt series make EMPIAR-11078 significantly more challenging for denoising and missing wedge reconstruction than the two datasets from our previous experiments.Slices through reconstructions of two tomograms obtained with FBP, CryoCARE + IsoNet and DeepDeWedge are shown in Fig. 5. In the x–z- and z–y-planes, the DeepDeWedge reconstructions are more crisp and less noisy than those produced with CryoCARE + IsoNet. Especially in the x–z-plane, where the effects of the missing wedge are strongest, DeepDeWedge produces higher contrast than CryoCARE + IsoNet and removes more of the artefacts. Again, the CryoCARE + IsoNet reconstructions contain more high-frequency components and are closer to the FBP reconstruction, whereas the DeepDeWedge reconstructions are smoother and more denoised, especially in empty or background regions.Fig. 5: Slices through 3D reconstructions of tomograms showing the ciliary transit zone of C. reinhardtii.The red lines in each slice indicate the positions of the remaining two slices. We also show the central x–z-slice through the logarithm of the magnitude of the Fourier transform of each reconstruction. The row labels follow the naming convention of EMPIAR-11078.Remarkably, as can be seen in the zoomed-in regions in the second row of Fig. 5, both CryoCARE + IsoNet and DeepDeWedge reconstruct parts of the sample that are barely present in the FBP reconstruction since they are perpendicular to the electron beam direction, which means that a large portion of their Fourier components are masked out by the missing wedge.Note that both CryoCARE + IsoNet and DeepDeWedge appear less effective at reconstructing the missing wedge compared to the two experiments presented above. This is indicated by the central x–z-slices through the Fourier transform of the reconstructions and is likely due to the challenging, crowded nature and low SNR of the data.Experiments on synthetic dataHere, we compare DeepDeWedge to IsoNet, CryoCARE, and the CryoCARE + IsoNet combination on synthetic data to quantify our qualitative findings on real data.We used a dataset by Gubins et al.29 containing 10 noiseless synthetic ground truth volumes with a spatial resolution of 10 Å3/voxel and a size of 179 × 512 × 512 voxels. All volumes contain typical objects found in cryo-ET samples, such as proteins (up to 1500 uniformly rotated samples from a set of 13 structures), membranes, and gold fiducial markers.For our comparison, we fitted models on the first three tomograms of the SHREC 2021 dataset. We used the Python library tomosipo30 to compute clean projections of size 512 × 512 in the angular range ±60° with 2° increment. From these clean tilt series, we generated datasets with different noise levels by adding pixel-wise independent Gaussian noise to the projections. We simulated three datasets with tilt series SNR 1/2, 1/4, and 1/6, respectively.To measure the overall quality of a tomogram \(\hat{{{\bf{v}}}}\) obtained with any of the three methods, we calculated the normalized correlation coefficient \(\,{\rm{CC}}\,(\hat{{{\bf{v}}}},{{{\bf{v}}}}^{*})\) between the reconstruction \(\hat{{{\bf{v}}}}\) and the corresponding ground truth \({{\bf{v}}}^{*}\), which is defined as$${\rm{CC}}\,(\hat{{{\bf{v}}}},{{{\bf{v}}}}^{*})=\frac{\left\langle \hat{{{\bf{v}}}}-\,{\rm{mean}}\,(\hat{{{\bf{v}}}}),{{{\bf{v}}}}^{*}-\,{\rm{mean}}\,({{{\bf{v}}}}^{*})\right\rangle} {{\left\Vert \hat{{{\bf{v}}}}-{\rm{mean}} (\hat{{{\bf{v}}}})\right\Vert }_{2}{\left\Vert {\bf{v}}^{*}-{\rm{mean}} ({\bf{v}}^{*})\right\Vert }_{2}}.$$
(7)
By definition, it holds that \(0\le \,{\rm{CC}}\,(\hat{{{\bf{v}}}},{{{\bf{v}}}}^{*})\le 1\), and the higher the correlation between reconstruction and ground truth, the better. The correlation coefficient measures the reconstruction quality (both the denoising and the missing wedge reconstruction capabilities) of the methods. To isolate the denoising performance of a method from its ability to reconstruct the missing wedge, we also report the correlation coefficient between the refined reconstructions and the ground truth after applying a 60° missing wedge filter to both of them. We refer to this metric as “CC outside the missing wedge” and it is used to compare the denoising performance of CryoCARE, which does not perform missing wedge reconstruction.As a central application of cryo-ET is the analysis of biomolecules, we also report the resolution of all proteins in the refined tomograms. For this, we extracted all proteins from the ground truth and refined tomograms and calculated the average 0.143 Fourier shell correlation cutoff (0.143-FSC) between the refined proteins and the ground truth ones. The 0.143-FSC is commonly used in cryo-EM applications. Its unit is Angstroms, and it seeks to express up to which spatial frequency the information in the reconstruction is reliable. In contrast to the correlation coefficient, a lower 0.143-FSC value is better. To measure how well each method filled in the missing wedges of the structures, we also report the average 0.143-FSC calculated only on the true and predicted missing wedge data. We refer to this value as (average) “0.143-FSC inside the missing wedge”.Figure 6 shows the metrics for decreasing SNR of the tilt series. All metrics suggest that DeepDeWedge yields higher-quality reconstructions than IsoNet, CryoCARE, and CryoCARE + IsoNet.Fig. 6: Comparison of DeepDeWedge (with even/odd split) to IsoNet, CryoCARE and the combination of IsoNet and CryoCARE for increasing noise levels on the tilt series images.We inverted the y-axes such that higher is better for all metrics.CryoCARE achieves a lower correlation coefficient than IsoNet in the high-SNR regime, while the order is reversed for low SNR. A likely explanation is that for lower noise levels, the correlation coefficient is more sensitive to the missing wedge artefacts in the reconstructions. CryoCARE does not perform missing wedge reconstruction, so it has a lower correlation coefficient than IsoNet for higher SNR. For lower SNR, the correlation coefficient is dominated by the noise. Regarding denoising, we and others9 have observed that CryoCARE performs better than IsoNet, which is confirmed here by looking at the correlation coefficient outside the missing wedge. As expected, CryoCARE + IsoNet combines the strengths of both methods. Compared to this combination, the overall quality of DeepDeWedge reconstructions is on par or better, depending on the noise level.The FSC metrics in the second row of Fig. 6 indicate that the average resolution of the proteins in the refined tomograms is approximately the same for IsoNet and DeepDeWedge and that they perform similarly for missing wedge reconstruction.