Universal encoding of pan-cancer histology by deep texture representations

Cancer histological images contain rich biological and clinical information, but quantitative representation can be problematic and has prevented the direct comparison and accumulation of large-scale datasets. Here, we show successful universal encoding of cancer histology by deep texture representations (DTRs) produced by a bilinear convolutional neural network. DTR-based, unsupervised histological profiling, which captures the morphological diversity, is applied to cancer biopsies and reveals relationships between histologic characteristics and the response to immune checkpoint inhibitors (ICIs). Content-based image retrieval based on DTRs enables the quick retrieval of histologically similar images using The Cancer Genome Atlas (TCGA) dataset. Furthermore, via comprehensive comparisons with driver and clinically actionable gene mutations, we successfully predict 309 combinations of genomic features and cancer types from hematoxylin-and-eosin-stained images. With its mounting capabilities on accessible devices, such as smartphones, universal encoding for cancer histology has a strong impact on global equalization for cancer diagnosis and therapies. [Display omitted] •DTR is a universal encoder for cancer histology using a deep neural network•DTR enables quick and accurate retrieval of histologically similar images•DTR enables genomic feature prediction from histology images in various cancer types•DTR-based image retrieval system is publicly accessible from PC and smartphone Komura et al. develop a method to encode cancer histology using a deep neural network. The utility of the method is validated in three clinically useful applications: unsupervised analysis of cancer morphology to detect histological subtypes, retrieving histologically similar images among large datasets, and predicting genomic features from histology images