Image Processing, Machine Learning and Visualization for Tissue Analysis

Knowledge discovery for understanding mechanisms of disease requires the integration of multiple sources of data collected at various magnifications and by different imaging techniques. Using spatial information, we can build maps of tissue and cells in which it is possible to extract, e.g., measurements of cell morphology, protein expression, and gene expression. These measurements reveal knowledge about cells such as their identity, origin, density, structural organization, activity, and interactions with other cells and cell communities. Knowledge that can be correlated with survival and drug effectiveness. This thesis presents multidisciplinary projects that include a variety of methods for image and data analysis applied to images coming from fluorescence- and brightfield microscopy. In brightfield images, the number of proteins that can be observed in the same tissue section is limited. To overcome this, we identified protein expression coming from consecutive tissue sections and fused images using registration to quantify protein co-expression. Here, the main challenge was to build a framework handling very large images with a combination of rigid and non-rigid image registration. Using multiplex fluorescence microscopy techniques, many different molecular markers can be used in parallel, and here we approached the challenge to decipher cell classes based on marker combinations. We used ensembles of machine learning models to perform cell classification, both increasing performance over a single model and to get a measure of confidence of the predictions. We also used resulting cell classes and locations as input to a graph neural network to learn cell neighborhoods that may be correlated with disease. Finally, the work leading to this thesis included the creation of an interactive visualization tool, TissUUmaps. Whole slide tissue images are often enormous and can be associated with large numbers of data points, creating challenges which call for advanced methods in processing and visualization. We built TissUUmaps so that it could visualize millions of data points from in situ sequencing experiments and enable contextual study of gene expression directly in the tissue at cellular and sub-cellular resolution. We also used TissUUmaps for interactive image registration, overlay of regions of interest, and visualization of tissue and corresponding cancer grades produced by deep learning methods. The aforementioned methods and tools together provid