Automatic segmentation of rectal tumors from MRI using multiscale densely connected convolutional neural network based on attention mechanism

Rectal cancer is one of the most common malignancies in the gastrointestinal tract. Currently, magnetic resonance imaging has become a vital tool in diagnosing and treating patients with rectal cancer. Notably, early diagnosis of rectal cancer can help improve patient survival rate; however, the clinical expertise of physicians is a limiting factor. Therefore, we propose an attention-based multiscale densely connected convolutional neural network based on an attention mechanism to improve the accuracy of diagnosis by automatically segmenting rectal tumors from two-dimensional (2D) magnetic resonance images (MRI) using computer-aided diagnostic techniques. First, to address the inability of U-Net (a classical segmentation network for medical images) and extract rich semantic features and the inconsistent shape and size of tumors between different patients, we replace the conventional convolutional blocks in the U-Net network with multiscale densely connected convolutional blocks. Second, to make the network focus better on global contextual information, we add central blocks with atrous convolution in the final coding layer or the last coding layer. Finally, we add a hybrid attention mechanism to each decoder module to help the model focus on the features of the rectal tumor region. We validated the effectiveness of the proposed method using 3773 2D MRI datasets from 572 patients. The sensitivity, specificity, Dice correlation coefficient, and Hausdorff distance of MRI rectal tumor segmentation were 85.47%, 86.35%, 94.71%, and 7.88mm, respectively. The results showed that the proposed method outperforms conventional approaches. Moreover, the proposed method has better segmentation results in the rectal tumor segmentation task and can provide physicians with the second-most important clinical diagnostic opinion.