A region-based convolutional network for nuclei detection and segmentation in microscopy images

•The GA-RPN module is employed to generate candidate proposals that are more suitable for nuclei detection.•An IoU branch is proposed to regress the IoU between the detection boxes and their corresponding ground truth.•A fusioned box score is proposed as the ranking keyword in SoftNMS to select proper prediction bounding boxes.•The proposed method achieves improved nuclei detection and segmentation performances. Automated detection and segmentation of nuclei in microscopy images are of significant importance to biomedical research and clinical practice, including nuclear morphology analysis, cancer diagnosis and grading. However, these tasks are still challenging due to large numbers of adhered and clustered nuclei. Modern CNN-based nuclei detection and segmentation methods rely on bounding box regression and non-maximum suppression to locate the nuclei, which would lead to inferior localized bounding boxes of the adhered and clustered nuclei. In this paper, we propose a region-based convolutional network to tackle this challenge. In particular, a GA-RPN module integrating the guided anchoring(GA) into the region proposal network(RPN) is employed to generate candidate proposals that are more suitable for nuclei detection. A new branch is proposed to regress the intersection over union(IoU) between the detection boxes and their corresponding ground truth for locating the bounding box. To reduce the undetection of adhered and clustered nuclei, we pass a fusioned box score(FBS) into soft non-maximum suppression(SoftNMS) to preserve the true positive candidate boxes. The experiments are performed on two public challenging datasets, which are designed to challenge an algorithm’s ability to generalize across different varieties. The results empirically demonstrate that our method has better detection and segmentation capability than existing state-of-the-art methods. In conclusion, our method can improve the performances and provide a potential for sophisticated cell-level analysis within digital tissue images.