CT male pelvic organ segmentation using fully convolutional networks with boundary sensitive representation

•Robust and accurate soft labels are assigned to voxels near the boundary based on both spatial cue and semantic cue.•A multi-label cross-entropy loss that uses soft labels and hard labels as supervision is to perform segmentation.•A localization model is designed to focus on candidate regions, which can contribute significantly to better performance.•The experimental results on a challenging CT dataset show that our method outperforms the state-of-the-art methods. [Display omitted] Accurate segmentation of the prostate and organs at risk (e.g., bladder and rectum) in CT images is a crucial step for radiation therapy in the treatment of prostate cancer. However, it is a very challenging task due to unclear boundaries, large intra- and inter-patient shape variability, and uncertain existence of bowel gases and fiducial markers. In this paper, we propose a novel automatic segmentation framework using fully convolutional networks with boundary sensitive representation to address this challenging problem. Our novel segmentation framework contains three modules. First, an organ localization model is designed to focus on the candidate segmentation region of each organ for better performance. Then, a boundary sensitive representation model based on multi-task learning is proposed to represent the semantic boundary information in a more robust and accurate manner. Finally, a multi-label cross-entropy loss function combining boundary sensitive representation is introduced to train a fully convolutional network for the organ segmentation. The proposed method is evaluated on a large and diverse planning CT dataset with 313 images from 313 prostate cancer patients. Experimental results show that the performance of our proposed method outperforms the baseline fully convolutional networks, as well as other state-of-the-art methods in CT male pelvic organ segmentation.