NST: A nuclei segmentation method based on transformer for gastrointestinal cancer pathological images

Gastrointestinal cancer is a prevalent disease, and analyzing pathological images is crucial for its diagnosis and treatment. Considering the characteristics of pathological images, we propose a novel cell nucleus segmentation method based on Vision Transform, namely NST. Our proposed method consists of a Deformable Attention Transformer (DAT) encoder capturing four different levels of feature; a Coordinate Attention Module (CAM) handling shallow-level features in different dimensions; a Dense Aggregation Module (DAM) integrating deep-level features; and a Similarity Aggregation Module (SAM) combining features to generate pixel-level segmentation predictions. Meanwhile, to fill the data gap in the field of cell nucleus segmentation, we acquire, annotate and present a new dataset of gastrointestinal cancer pathology images named GCNS. Moreover, we conducted a series of experiments, and the experimental results indicate that our proposed method achieves state-of-the-art performance, as high as a 0.725 Dice Score on the GCNS dataset. [Display omitted] •Our method utilize the limited data without augmenting and perform excellently.•Our method based on Vision Transformer displays good robustness and generalizability.•Our method works with the assistance of multi-depth aggregation modules.•We conducted experiments on our new proposed gastrointestinal dataset.•The contrast experiments on the dataset show the comparative advantage of our method.