Unsupervised segmentation method for cuboidal cell nuclei in histological prostate images based on minimum cross entropy

•We propose a segmentation method for cuboidal cells nuclei in prostate images.•We validate the method by quantitative evaluation.•We compare the method with other segmentation methods.•The method allows segmentation with a mean accuracy of 97%.•The method is effective in images of normal, hyperplasia and cancer tissue. This paper presents a novel segmentation method for cuboidal cell nuclei in images of prostate tissue stained with hematoxylin and eosin. The proposed method allows segmenting normal, hyperplastic and cancerous prostate images in three steps: pre-processing, segmentation of cuboidal cell nuclei and post-processing. The pre-processing step consists of applying contrast stretching to the red (R) channel to highlight the contrast of cuboidal cell nuclei. The aim of the second step is to apply global thresholding based on minimum cross entropy to generate a binary image with candidate regions for cuboidal cell nuclei. In the post-processing step, false positives are removed using the connected component method. The proposed segmentation method was applied to an image bank with 105 samples and measures of sensitivity, specificity and accuracy were compared with those provided by other segmentation approaches available in the specialized literature. The results are promising and demonstrate that the proposed method allows the segmentation of cuboidal cell nuclei with a mean accuracy of 97%.