Logo: to the web site of Uppsala University

Obesity is associated with an increased risk of morbidity and mortality. Achieving a healthy body composition, which involves maintaining a balance between fat and muscle mass, is important for metabolic health and preventing chronic diseases. Computed tomography (CT) imaging offers detailed insights into the body’s internal structure, aiding in understanding body composition and its related factors. In this feasibility study, we utilized CT image data from 2,724 subjects from the large metabolic health cohort studies SCAPIS and IGT. We train and evaluate an uncertainty-aware deep regression based ResNet-50 network, which outputs its prediction as mean and variance, for quantification of cross-sectional areas of liver, visceral adipose tissue (VAT), and thigh muscle. This was done using collages of three single-slice CT images from the liver, abdomen, and thigh regions. The model demonstrated promising results with the evaluation metrics – including R-squared (R²) and mean absolute error (MAE) for predictions. Additionally, for interpretability, the model was evaluated with saliency analysis based on Grad-CAM (Gradient-weighted Class Activation Mapping) at stages 2, 3, and 4 of the network. Deformable image registration to a template subject further enabled cohort saliency analysis that provide group-wise visualization of image regions of importance for associations to biomarkers of interest. We found that the networks focus on relevant regions for each target, according to prior knowledge.

Early and accurate prediction of clinical outcomes holds great potential for patient prognostics and personalized treatment planning. Development of automated methods for estimation of medical image-based clinical parameters (e.g. total metabolic tumor volume, TMTV) could pave the way for predicting advanced clinical outcomes not explicitly available in the images, such as overall survival. We developed an automated framework that extracted tissue-wise multi-channel 2D projections from whole-body FDG-PET/CT volumes, by separating tissues based on CT Hounsfield units, and used a DenseNet-121 to estimate the TMTV from the projections. For transparency and interpretability, an image registration-based cohort saliency analysis was proposed. The network was applied on the autoPET cohort (501 scans representing lymphoma, lung cancer, melanoma) and evaluated using a single channel method (baseline) and a multi-channel method (proposed), for the purpose of comparison. The incorporation of multiple channels demonstrated an advantage in the TMTV prediction, outperforming the baseline model with a ΔMAE = –14.34 ml; ΔR2 = 0.1584; ΔICC = 0.1316 (p-value = 0.0098). The Pearson correlation coefficient (r) was computed between the ground truth (GT) tumor projections and the aggregated saliency maps. Statistical comparison, via bootstrapping, showed that the proposed model consistently outperformed the baseline, with significantly higher r across all cancer types and both sexes, except for melanoma in females. This implied that the aggregated saliency maps generated by the proposed model showed higher correspondence with the GT, compared to the baseline model. Our approach offers a promising and interpretable framework for the automated prediction of TMTV, with further potential to also predict advanced clinical outcomes.

We propose a method for multimodal deformable image registration which combines a powerful deep learning approach to generate CoMIRs, dense image-like representations of multimodal image pairs, with INSPIRE, a robust framework for monomodal deformable image registration. We introduce new equivariance constraints to improve the consistency of CoMIRs under deformation. We evaluate the method on three publicly available multimodal datasets, one remote sensing, one histological, and one cytological. The proposed method demonstrates general applicability and consistently outperforms state-of-the-art registration tools \elastixname and VoxelMorph. We share source code of the proposed method and complete experimental setup as open-source at: https://github.com/MIDA-group/CoMIR_INSPIRE.

We present INSPIRE, a top-performing general-purpose method for deformable image registration. INSPIRE brings distance measures which combine intensity and spatial information into an elastic B-splines-based transformation model and incorporates an inverse inconsistency penalization supporting symmetric registration performance. We introduce several theoretical and algorithmic solutions which provide high computational efficiency and thereby applicability of the proposed framework in a wide range of real scenarios. We show that INSPIRE delivers highly accurate, as well as stable and robust registration results. We evaluate the method on a 2D dataset created from retinal images, characterized by presence of networks of thin structures. Here INSPIRE exhibits excellent performance, substantially outperforming the widely used reference methods. We also evaluate INSPIRE on the Fundus Image Registration Dataset (FIRE), which consists of 134 pairs of separately acquired retinal images. INSPIRE exhibits excellent performance on the FIRE dataset, substantially outperforming several domain-specific methods. We also evaluate the method on four benchmark datasets of 3D magnetic resonance images of brains, for a total of 2088 pairwise registrations. A comparison with 17 other state-of-the-art methods reveals that INSPIRE provides the best overall performance. Code is available at github.com/MIDA-group/inspire

We propose a method for multimodal deformable image registration which combines a powerful deep learning approach to generate CoMIRs, dense image-like representations of multimodal image pairs, with INSPIRE, a robust framework for monomodal deformable image registration. We introduce new equivariance constraints to improve the consistency of CoMIRs under deformation. We evaluate the method on three publicly available multimodal datasets, one remote sensing, one histological, and one cytological. The proposed method demonstrates general applicability and consistently outperforms state-of-the-art registration tools \elastixname and VoxelMorph. We share source code of the proposed method and complete experimental setup as open-source at: https://github.com/MIDA-group/CoMIR_INSPIRE.

Multimodal image alignment involves finding spatial correspondences between volumes varying in appearance and structure. Automated alignment methods are often based on local optimization that can be highly sensitive to initialization. We propose a novel efficient algorithm for computing similarity of normalized gradient fields (NGF) in the frequency domain, which we globally optimize to achieve rigid multimodal 3D image alignment. We validate the method experimentally on a dataset comprised of 20 brain volumes acquired in four modalities (T1w, Flair, CT, [18F] FDG PET), synthetically displaced with known transformations. The proposed method exhibits excellent performance on all six possible modality combinations and outperforms the four considered reference methods by a large margin. An important advantage of the method is its speed; global rigid alignment of 3.4 Mvoxel volumes requires approximately 40 seconds of computation, and the proposed algorithm outperforms a direct algorithm for the same task by more than three orders of magnitude. Open-source code is provided.

Multimodal image registration is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. Two similarity measures widely used in multimodal image registration are mutual information (MI) and similarity of normalized gradient fields (NGF). We propose efficient algorithms for computing MI and similarity of NGF for all discrete axis-aligned shifts in the frequency domain. These fast algorithms enable highly reliable global registration of multimodal images, also for very large displacements,  which we confirm by their performance evaluation on a number of different pairs of modalities.

We consider four datasets, and observe that global maximization of MI is the best choice for two datasets/applications in 2D, while global maximization of similarity of NGF performs best on the remaining two datasets, of which one consists of 2D images, and the other consists of 3D data. This confirms the relevance of both methods; their properties recommend them for application in different scenarios.    

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration of multimodal biomedical and medical 2D and 3D images. We compare the performance of four Generative Adversarial Network (GAN)-based I2I translation methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on four publicly available multimodal (2D and 3D) datasets and compare with the performance of registration achieved by several well-known approaches acting directly on multimodal image data. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. The evaluated representation learning method, which aims to find abstract image-like representations of the information shared between the modalities, manages better, and so does the Mutual Information maximisation approach, acting directly on the original multimodal images. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval), including method implementations, evaluation code, and all datasets, for further reproducing and benchmarking.

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically.

The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image translation methods to facilitate multimodal registration; (vii) the Stochastic Distance Transform, using the theory of discrete random sets to offer improved noise-insensitivity to distance computations; (viii) extensive evaluation of the proposed image registration methods on a number of different datasets mainly from (bio)medical imaging, where they exhibit excellent performance, and reliability, suggesting wide utility.