Logo: to the web site of Uppsala University

We introduce an end-to-end unsupervised (or weakly supervised) image registration method that blends conventional medical image registration with contemporary deep learning techniques from computer vision. Our method downsamples both the fixed and the moving images into multiple feature map levels where a displacement field is estimated at each level and then further refined throughout the network. We train and test our model on three different datasets. In comparison with the initial registrations we find an improved performance using our model, yet we expect it would improve further if the model was fine-tuned for each task. The implementation is publicly available (https://github.com/ngunnar/learning-a-deformable-registration-pyramid).

Spatiotemporal imaging has applications in e.g. cardiac diagnostics, surgical guidance, and radiotherapy monitoring, In this paper, we explain the temporal motion by identifying the underlying dynamics, only based on the sequential images. Our dynamical model maps the inputs of observed high-dimensional sequential images to a low-dimensional latent space wherein a linear relationship between a hidden state process and the lower-dimensional representation of the inputs holds. For this, we use a conditional variational auto-encoder (CVAE) to nonlinearly map the higher dimensional image to a lower-dimensional space, wherein we model the dynamics with a linear Gaussian state-space model (LG-SSM). The model, a modified version of the Kalman variational auto-encoder, is end-to-end trainable, and the weights, both in the CVAE and LG-SSM, are simultaneously updated by maximizing the evidence lower bound of the marginal likelihood. In contrast to the original model, we explain the motion with a spatial transformation from one image to another. This results in sharper reconstructions and the possibility of transferring auxiliary information, such as segmentation, through the image sequence. Our experiments, on cardiac ultrasound time series, show that the dynamic model outperforms traditional image registration in execution time, to a similar performance. Further, our model offers the possibility to impute and extrapolate for missing samples.

Image monitoring and guidance during medical examinations can aid both diagnosis and treatment. However, the sampling frequency is often too low, which creates a need to estimate the missing images. We present a probabilistic motion model for sequential medical images, with the ability to both estimate motion between acquired images and forecast the motion ahead of time. The core is a low-dimensional temporal process based on a linear Gaussian state-space model with analytically tractable solutions for forecasting, simulation, and imputation of missing samples. The results, from two experiments on publicly available cardiac datasets, show reliable motion estimates and an improved forecasting performance using patient-specific adaptation by online learning.

Intra-interventional imaging is a tool for monitoring and guiding ongoing treatment sessions. Ideally one would like the full 3D image at high temporal resolution, this is however not possible due to the acquisition time. In this study, we consider the scenario when the observations are sparse and consist only of 2D image slices through the 3D volume. Given 2D-2D image registrations between a predefined 3D volume and the observations, we propose a method to estimate the full 3D motion. This 3D motion enables the reconstruction of the 3D anatomy. Our method relies on a conditioning-based denoising diffusion model and generates estimates given the 2D sparse observations. We reduce the dimensionality of the diffusion process by embedding the data in a lower dimensional space using principal component analysis. The model is evaluated in two experiments: first on synthetically generated data and then using medical lung images. Our observations show that the estimates are stable across the entire volume and within 1 mm of the lower bound defined by the reconstruction error.

Background: To compensate for organ motion during a radiotherapy session,margins are normally added to the tumor region. With MR-linacs, it is nowpossible to monitor the motion by acquiring 2D cine-MRI in real-time.

Purpose: In this paper, we propose a method to estimate the entire 3D motiongiven sparse information in the form of 2D images of the anatomy.

Methods: The methods consist of three models: two 2D motion modelswith forecasting possibility and one 2D-to-3D extrapolation model to estimatethe 3D motion at each point in time. In the experiments, we use real imagesfrom patients treated with an MRI-linac system, where seven patients wereused for training and two for evaluation. The experiment was two-fold; onebased on a phase-sorted 4D CT with known motions, and, one based on acine-MRI sequence, where the ground-truth 3D motion was unknown.

Results: Our model estimates the 3D motion given two 2D image observationsin the coronal and sagittal orientation with an average error of 0.43mm in the entire anatomy. In the PTV, the average error was 0.82 mm and 0.56mm for the two patients in the evaluation cohort. For the cine-MRI sequence,our approach achieved results comparable to previously published centroidtracking while also providing a complete deformable 3D motion estimate.

Conclusions: We present a method to estimate full 3D motion from soarsedata in the form of 2D images suitable for the MR-linac.