Optic disc segmentation by U-net and probability bubble in abnormal fundus images

•The deep learning architecture fused with the model-driven probability bubble approach is proposed to segment OD in abnormal fundus images to improve the performance when lack sufficient training samples in medical community.•A brand-new unsupervised probability bubble technique is figured out according to the position relationship between retinal vessels an OD, by which the main blood vessels are fitted by line segments through hough transform, and the density of intersection points of the lines indicates the probability of OD.•The joint probability of data-driven U-net and model-driven probability bubble approach is calculated to locate the OD. The localization based on the joint probability is more robust than each independent method, which ensures effectiveness of the proposed method.•The work in this paper provides a reference for the application of deep learning in medical community that the model-driven approach can be fused into the architecture and promote the performance when there are insufficient training samples available and deep learning can’t provide an accurate result. Segmenting optic disc (OD) in abnormal fundus images is a challenge task because of many distractions such as illumination variations, blurry boundary, occlusion of retinal vessels and big bright lesions. Data-driven deep learning is effective and robust to illumination variations, blurry boundary and occlusion in the normal fundus images but sensitive to big bright lesions in abnormal images. In this paper, an automatic OD segmentation method fusing U-net with model-driven probability bubble approach is proposed in abnormal fundus images. The probability bubble is conceived according to the position relationship between retinal vessels and OD, and the localization result is fused into the output layer of U-net through calculating the joint probability. The proposed method takes the advantage of the deep learning architecture and improves the architecture’s performance by including the model-driven position constraint when lack of sufficient training data. Experiments show that the proposed method successfully removes the distraction of bright lesions in abnormal fundus images and obtains a satisfying OD segmentation on three public databases: Kaggle, MESSIDOR and NIVE, and it outperforms existing methods with a very high accuracy.