DB-FCN: An end-to-end dual-branch fully convolutional nucleus detection model

•Propose a dual-branch model that realizes the mutual benefits of both detections.•DB-FCN combines the advantages of both one-stage and two-stage object detections.•Coarse detection suppresses the background of the image to improve the accuracy.•We designs a three-stage training method to fully exploit the model’s performance.•DB-FCN achieves highest scores on various datasets. Detecting cells or nuclei in histopathological image analysis is a crucial prerequisite for subsequent cancer diagnosis. By precisely locating cell nuclei, pathologists can quantitatively analyze the morphology of each cell nucleus. This enables accurate cancer grading, allowing for the implementation of tailored treatment plans based on distinct cancer stages. Manual nucleus detection methods are labor-intensive, making the development of automatic cell nucleus detection algorithms highly necessary. However, due to the small size of the nucleus and increased adhesions between cells, tissue staining may be uneven, leading to a higher occurrence of false positives in the results of the current automatic nucleus detection algorithm. This paper introduces an end-to-end dual-branch-based fully convolutional neural network (DB-FCN) that effectively addresses the aforementioned challenges, thereby enhancing the accuracy of automatic nucleus detection. This algorithm introduces for the first time the use of two detection branches, namely the coarse detection branch and the fine detection branch, to accomplish the detection task. The role of the coarse detection branch is to identify all cell regions in the pathological image as comprehensively as possible and then transmit the detection results as prior information to the fine detection branch. The fine detection branch is necessary solely to conduct more precise detection based on the coarse detection results. Given that the coarse detection branch has already eliminated interference from many background regions, the fine detection branch can focus on detecting the nucleus region, thereby significantly enhancing the efficiency and accuracy of model detection. The detection model proposed in this paper was evaluated on three classic datasets and compared with many existing detection algorithms. Compared with other algorithms, the detection algorithm proposed in this paper has made significant progress in detecting cell nuclei in histopathological images.