Prognosis prediction based on liver histopathological image via graph deep learning and transformer

Liver cancer is one of the leading causes of cancer-related deaths globally. Accurately predicting the prognosis of liver cancer patients is crucial for improving their treatment and developing new anticancer drugs. However, analyzing whole slide images is time-consuming and labor-intensive for pathologists. Although deep learning methods can improve analysis efficiency, cancer prognosis prediction remains challenging due to the need for both histological features and context-aware representations to accurately infer patient survival probabilities. Several context-aware models based on graph neural networks have been proposed for weakly supervised deep learning. However, most of these methods extract WSI features using a classification network pretrained on ImageNet, which does not include cancer cell-level images. Additionally, most GNN-based methods employ a fixed number of graph convolutional layers, limiting their ability to learn multi-scale information. To address these limitations, we propose Multi-Trans-GACN, a context-aware parallel multi-scale GNN based on Transformers. Multi-Trans-GACN hierarchically aggregates instance-level histology features on different scales in the liver cancer microenvironment. A Transformer-based scale attention mechanism is utilized to combine the features extracted from different scales. We also propose a method that utilizes InceptionV3, pretrained on cellular-level liver cancer images, to construct graph structures for liver cancer images. We evaluated Multi-Trans-GACN on two liver cancer datasets. Compared to existing methods, our approach achieved significant improvements in the C-index by 5.3 and 2.50, demonstrating its superior performance in liver cancer prognosis prediction tasks. The code id available in https://github.com/z19991013/MTG. [Display omitted] •The prognosis of cancer patients by deep learning method is worth discussing because it can provide improved treatment.•We used a network pretrained with over 60,000 patches from an additional 90 patient WSIs to extract features.•We combine graph neural networks with Transformer which effectively improves accuracy.•We conducted experiments on our method on two datasets and reach the best performance among the existing method.