Logo: to the web site of Uppsala University

Accurate and efficient cell detection is crucial in many biomedical image analysis tasks. We evaluate the performance of several Deep Learning (DL) methods for cell detection in Papanicolaoustained cytological Whole Slide Images (WSIs), focusing on accuracy of predictions and computational efficiency. We examine recent off-the-shelf algorithms as well as custom-designed detectors, applying them to two datasets: the CNSeg Dataset and the Oral Cancer (OC) Dataset. Our comparison includes well-established segmentation methods such as StarDist, Cellpose, and the Segment Anything Model 2 (SAM2), alongside centroid-based Fully Convolutional Regression Network (FCRN) approaches. We introduce a suitable evaluation metric to assess the accuracy of predictions based on the distance from ground truth positions. We also explore the impact of dataset size and data augmentation techniques on model performance. Results show that centroid-based methods, particularly the Improved Fully Convolutional Regression Network (IFCRN) method, outperform segmentation-based methods in terms of both detection accuracy and computational efficiency. This study highlights the potential of centroid-based detectors as a preferred option for cell detection in resource-limited environments, offering faster processing times and lower GPU memory usage without compromising accuracy.

Background and objectives:

Oral cancer is a global health challenge. The disease can be successfully treated if detected early, but the survival rate drops significantly for late stage cases. There is a growing interest in a shift from the current standard of invasive and time-consuming tissue sampling and histological examination, towards non-invasive brush biopsies and cytological examination, facilitating continued risk group monitoring. For cost effective and accurate cytological analysis there is a great need for reliable computer-assisted data-driven approaches. However, infeasibility of accurate cell-level annotation hinders model performance, and limits evaluation and interpretation of the results. This study aims to improve AI-based oral cancer detection by introducing additional information through multimodal imaging and deep multimodal information fusion.

Methods:

We combine brightfield and fluorescence whole slide microscopy imaging to analyze Papanicolaou-stained liquid-based cytology slides of brush biopsies collected from both healthy and cancer patients. Given the challenge of detailed cytological annotations, we utilize a weakly supervised deep learning approach only relying on patient-level labels. We evaluate various multimodal information fusion strategies, including early, late, and three recent intermediate fusion methods.

Results:

Our experiments demonstrate that: (i) there is substantial diagnostic information to gain from fluorescence imaging of Papanicolaou-stained cytological samples, (ii) multimodal information fusion improves classification performance and cancer detection accuracy, compared to single-modality approaches. Intermediate fusion emerges as the leading method among the studied approaches. Specifically, the Co-Attention Fusion Network (CAFNet) model achieves impressive results, with an F1 score of 83.34% and an accuracy of 91.79% at cell level, surpassing human performance on the task. Additional tests highlight the importance of accurate image registration to maximize the benefits of the multimodal analysis.

Conclusion:

This study advances the field of cytopathology by integrating deep learning methods, multimodal imaging and information fusion to enhance non-invasive early detection of oral cancer. Our approach not only improves diagnostic accuracy, but also allows an efficient, yet uncomplicated, clinical workflow. The developed pipeline has potential applications in other cytological analysis settings. We provide a validated open-source analysis framework and share a unique multimodal oral cancer dataset to support further research and innovation.

Microphysiological systems (MPSs) provide the capability to continuously monitor cell cultures for an improved understanding of various biological phenomena. A common problem, however, is the formation of air bubbles that can affect the experiments [1]. Receiving timely notifications about bubble formation in MPSs can be beneficial for an experimenter, as early detection makes it possible to eliminate the bubble or direct it into a waste outlet. 

Brightfield (BF) microscopy imaging is a common method for monitoring MPSs, and it also allows for the observation of bubbles. However, if bubbles are to be monitored visually using BF images, the observer must remain vigilant and continuously monitor the situation. Modern machine learning (ML) techniques offer both automation and accuracy. Bubble detection using ML could be useful for both offline post-processing and also become part of an online system for bubble notification in MPSs. Despite the potential, there are no studies focused on applying ML models for bubble detection in MPSs. This absence can be attributed to the lack of sufficiently large datasets from MPSs, as these systems are often used in research environments where it is challenging to collect large amounts of data from the same device or under consistent settings.

We explore whether combining BF microscopy data from different MPSs could enable the development of a versatile ML model for bubble detection in MPSs. Considering that training of ML models usually requires a substantial amount of data, we explore different strategies for learning from small datasets expected in our observed scenarios. Transfer learning is a suitable approach in the ML community [2]. We evaluate the performance of models pre-trained on large, non-MPS-related ML image datasets through transfer learning, comparing them with models trained on data combined from smaller MPS datasets.

[1] Yang, W., Li, T., Liao, S et al. (2024). TrAC Trends in Analytical Chemistry, 117569 doi:10.1016/j.trac.2024.117569

[2]  Weiss, K., Khoshgoftaar, T. M. and Wang D. (2016). Journal of Big data, 3, 1-40. doi:10.1186/s40537-016-0043-6

Friendship and rapport play an important role in the formation of constructive social interactions, and have been widely studied in education due to their impact on learning outcomes. Given the growing interest in automating the analysis of such phenomena through Machine Learning, access to annotated interaction datasets is highly valuable. However, no dataset on child-child interactions explicitly capturing rapport currently exists. Moreover, despite advances in the automatic analysis of human behavior, no previous work has addressed the prediction of rapport in child-child interactions in educational settings. We present UpStory — the Uppsala Storytelling dataset: a novel dataset of naturalistic dyadic interactions between primary school aged children, with an experimental manipulation of rapport. Pairs of children aged 8–10 participate in a task-oriented activity: designing a story together, while being allowed free movement within the play area. We promote balanced collection of different levels of rapport by using a within-subjects design: self-reported friendships are used to pair each child twice, either minimizing or maximizing pair separation in the friendship network. The dataset contains data for 35 pairs, totaling 3 h 40 m of audiovisual recordings. It includes two video sources, and separate voice recordings per child. An anonymized version of the dataset is made publicly available, containing per-frame head pose, body pose, and face features. Finally, we confirm the informative power of the UpStory dataset by establishing baselines for the prediction of rapport. A simple approach achieves 68% test accuracy using data from one child, and 70% test accuracy aggregating data from a pair.

Close, fulfilling interactions with other classmates are an important part of a child’s learning experience, and have been shown to improve educational outcomes. This is especially apparent in guided play activities in which children need to coordinate to succeed. However, until the recent publication of the UpStory dataset, no child-child interaction dataset with explicit control for rapport was available, leading to a lack of methods for automatic rapport prediction in pair play. In this study, we perform the first-of-its-kind evaluation of end-to-end Computer Vision techniques for child-child rapport prediction, and compare our Deep Learning-based results to the feature-based Machine Learning approaches reported in the UpStory paper. The results show that, under a thorough training and evaluation procedure, end-to-end learning under-performs when compared to feature-based methods.

Correlated Multimodal Imaging (CMI) gathers information about the same specimen with two or more modalities that–combined–create a composite and complementary view of the sample (including insights into structure, function, dynamics and molecular composition). CMI allows one to reach beyond what is possible with a single modality and describe biomedical processes within their overall spatio-temporal context and gain a mechanistic understanding of cells, tissues, and organisms in health and disease by untangling their molecular mechanisms within their native environment. The field of CMI has grown substantially over the last decade and previously unanswerable biological questions have been solved by applying novel CMI workflows. To disseminate these workflows and comprehensively share the scattered knowledge present within the CMI community, an initiative was started to bring together imaging, image analysis, and biomedical scientists and work towards an open community that promotes and disseminates the field of CMI. This community project was funded for the last 4 years by an EU COST Action called COMULIS (COrrelated MUltimodal imaging in the LIfe Sciences). In this review we share some of the showcases and lessons learnt from the action. We also briefly look ahead at how we anticipate building on this initial initiative.

In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations (embedding the different modalities in a common space) with robust feature extraction and bag-of-words models for efficient and reliable retrieval. We illustrate its advantages through a replacement study, exploring a number of feature extractors and learned representations, as well as through comparison to recent (cross-modality) CBIR methods. For the task of (sub-)image retrieval on a (publicly available) dataset of brightfield and second harmonic generation microscopy images, the results show that our approach is superior to all tested alternatives. We discuss the shortcomings of the compared methods and observe the importance of equivariance and invariance properties of the learned representations and feature extractors in the CBIR pipeline. Code is available at: https://github.com/MIDA-group/CrossModal_ImgRetrieval.

The current medical standard for setting an oral cancer (OC) diagnosis is histological examination of a tissue sample taken from the oral cavity. This process is time-consuming and more invasive than an alternative approach of acquiring a brush sample followed by cytological analysis. Using a microscope, skilled cytotechnologists are able to detect changes due to malignancy; however, introducing this approach into clinical routine is associated with challenges such as a lack of resources and experts. To design a trustworthy OC detection system that can assist cytotechnologists, we are interested in deep learning based methods that can reliably detect cancer, given only per-patient labels (thereby minimizing annotation bias), and also provide information regarding which cells are most relevant for the diagnosis (thereby enabling supervision and understanding). In this study, we perform a comparison of two approaches suitable for OC detection and interpretation: (i) conventional single instance learning (SIL) approach and (ii) a modern multiple instance learning (MIL) method. To facilitate systematic evaluation of the considered approaches, we, in addition to a real OC dataset with patient-level ground truth annotations, also introduce a synthetic dataset—PAP-QMNIST. This dataset shares several properties of OC data, such as image size and large and varied number of instances per bag, and may therefore act as a proxy model of a real OC dataset, while, in contrast to OC data, it offers reliable per-instance ground truth, as defined by design. PAP-QMNIST has the additional advantage of being visually interpretable for non-experts, which simplifies analysis of the behavior of methods. For both OC and PAP-QMNIST data, we evaluate performance of the methods utilizing three different neural network architectures. Our study indicates, somewhat surprisingly, that on both synthetic and real data, the performance of the SIL approach is better or equal to the performance of the MIL approach. Visual examination by cytotechnologist indicates that the methods manage to identify cells which deviate from normality, including malignant cells as well as those suspicious for dysplasia. We share the code as open source.

Whole slide pathological image classification using slide-level labels often relies on multiple instance learning. Multiple instance learning based approaches are particularly challenging with whole slide cytology images, where the vast number of instances can make it difficult to identify key instances, especially when they are scarce. In this work we evaluate whether using representations learnt from patches from only normal slides is effective for instance-level decision making. We aim for interpretable slide-level decision making for whole slide cytology images. We focus on the effectiveness of a self-supervised contrastive learning framework within a one-class classifier setting, assessing its ability to learn the appearances of normal cells from a limited number of normal slides and subsequently identify abnormal cells (key instances) on test slides. We evaluate our approach on a publicly available cytology dataset, achieving a Recall@400 score of 0.1938, considerably improving over the 0.1109 score obtained using a weakly supervised approach.

Mediastinal structure measurements are important for the radiologist's review of computed tomography pulmonary angiography (CTPA) examinations. In the reporting process, radiologists make measurements of diameters, volumes, and organ densities for image quality assessment and risk stratification. However, manual measurement of these features is time consuming. Here, we sought to develop a time-saving automated algorithm that can accurately detect, segment and measure mediastinal structures in routine clinical CTPA examinations. In this study, 700 CTPA examinations collected and annotated. Of these, a training set of 180 examinations were used to develop a fully automated deterministic algorithm. On the test set of 520 examinations, two radiologists validated the detection and segmentation performance quantitatively, and ground truth was annotated to validate the measurement performance. External validation was performed in 47 CTPAs from two independent datasets. The system had 86-100% detection and segmentation accuracy in the different tasks. The automatic measurements correlated well to those of the radiologist (Pearson's r 0.68-0.99). Taken together, the fully automated algorithm accurately detected, segmented, and measured mediastinal structures in routine CTPA examinations having an adequate representation of common artifacts and medical conditions.