Partners

Abstract:
Deep learning methods are gaining momentum in radiology. In this work, we investigate the usefulness of vision-language models (VLMs) and large language models for binary few-shot classification of medical images. We utilize the GPT-4 model to generate text descriptors that encapsulate the shape and texture characteristics of objects in medical images. Subsequently, these GPT-4 generated descriptors, alongside
VLMs pre-trained on natural images, are employed to classify chest X-rays and breast ultrasound images. Our results indicate that few-shot classification of medical images using VLMs and GPT-4 generated descriptors is a viable approach. However, accurate classification requires the exclusion of certain descriptors from the calculations of the classification scores. Moreover, we assess the ability of VLMs to evaluate shape
features in breast mass ultrasound images. This is performed by comparing VLM-based results generated for shape-related text descriptors with the actual values of the shape features calculated using segmentation masks. We further investigate the degree of variability among the sets of text descriptors produced by GPT-4. Our work provides several important insights about the application of VLMs for medical image analysis.

Keywords:
medical image classification, ision-language models, arge language models, ew-shot learning

Abstract:
Pairwise image registration is a necessary prerequisite for brain image comparison and data integration in neuroscience and radiology. In this work, we explore the efficacy of implicit neural representations (INRs) in improving the performance of brain image registration in magnetic resonance imaging. In this setting, INRs serve as a continuous and coordinate based approximation of the deformation field obtained through a multi-layer perceptron. Previous research has demonstrated that sinusoidal representation networks (SIRENs) surpass ReLU models in performance. In this study, we first broaden the range of activation functions to further investigate the registration performance of implicit networks equipped with activation functions that exhibit diverse oscillatory properties. Specifically, in addition to the SIRENs and ReLU, we evaluate activation functions based on snake, sine+, chirp and Morlet wavelet functions. Second, we conduct experiments to relate the hyper-parameters of the models to registration performance. Third, we propose and assess various techniques, including cycle consistency loss, ensembles and cascades of implicit networks, as well as a combined image fusion and registration objective, to enhance the performance of implicit registration networks beyond the standard approach. The investigated implicit methods are compared to the VoxelMorph convolutional neural network and to the symmetric image normalization (SyN) registration algorithm from the Advanced Normalization Tools (ANTs). Our findings not only highlight the remarkable capabilities of implicit networks in addressing pairwise image registration challenges, but also showcase their potential as a powerful and versatile off-the-shelf tool in the fields of neuroscience and radiology.

Abstract:
Explaining deep learning models in a way that humans
can easily understand is essential for responsible artificial
intelligence applications. Attribution methods constitute an
important area of explainable deep learning. The attribu-
tion problem involves finding parts of the network’s input
that are the most responsible for the model’s output. In this
work, we demonstrate that implicit neural representations
(INRs) constitute a good framework for generating visual
explanations. Firstly, we utilize coordinate-based implicit
networks to reformulate and extend the extremal perturba-
tions technique and generate attribution masks. Experimen-
tal results confirm the usefulness of our method. For in-
stance, by proper conditioning of the implicit network, we
obtain attribution masks that are well-behaved with respect
to the imposed area constraints. Secondly, we present an it-
erative INR-based method that can be used to generate mul-
tiple non-overlapping attribution masks for the same image.
We depict that a deep learning model may associate the im-
age label with both the appearance of the object of interest
as well as with areas and textures usually accompanying the
object. Our study demonstrates that implicit networks are
well-suited for the generation of attribution masks and can
provide interesting insights about the performance of deep
learning models

Abstract:
Segmentation of biomedical images is often ambiguous and complicated by noise, varying
contrasts, and imaging artifacts. We address the challenge of segmenting images of brain
tissue in which gene expression has been localized using in situ hybridization. Since gene
expression patterns differ widely between genes, it can be difficult to correctly discriminate
pixels positive for gene expression. In testing different segmentation networks, we observed
that each network had its own trade-offs between sensitivity and precision. To exploit
the benefits of all trained networks, we developed a meta-network that learns to combine
multiple segmentation maps from diverse segmentation architectures to generate a final
segmentation that best matches the ground-truth label. In our experiments, the meta-
network outperforms ensembles that simply average segmentation maps.

Keywords:
meta-learning, segmentation, gene expression

Abstract:
We present the first automated pipeline to create an atlas of in situ hybridization gene expression in the adult marmoset brain in the same stereotaxic space. The pipeline consists of segmentation of gene expression from microscopy images and registration of images to a standard space. Automation of this pipeline is necessary to analyze the large volume of data in the genome-wide whole-brain dataset, and to process images that have varying intensity profiles and expression patterns with minimal human bias. To reduce the number of labelled images required for training, we develop a semi-supervised segmentation model. We further develop an iterative algorithm to register images to a standard space, enabling comparative analysis between genes and concurrent visualization with other datasets, thereby facilitating a more holistic understanding of primate brain structure and function.

Keywords:
contrastive learning, gene atlas, segmen-tation, semi-supervised learning, registration

Abstract:
We propose a novel image registration method based on implicit neural representations that addresses the challenging problem of registering a pair of brain images with similar anatomical structures, but where one image contains additional features or artifacts that are not present in the other image. To demonstrate its effectiveness, we use 2D microscopy in situ hybridization gene expression images of the marmoset brain. Accurately quantifying gene expression requires image registration to a brain template, which is difficult due to the diversity of patterns causing variations in visible anatomical brain structures. Our approach uses implicit networks in combination with an image exclusion loss to jointly perform the registration and decompose the image into a support and residual image. The support image aligns well with the template, while the residual image captures individual image characteristics that diverge from the template. In experiments, our method provided excellent results and outperformed other registration techniques.

Keywords:
brain, deep learning, gene expression, implicit neural representations, registration