Deep learning for cell image segmentation and ranking

•A methodology capable of segmenting both free-lying and clumps of abnormal cells with high overlapping rate from digitized images of conventional Pap smears.•The segmentation algorithm applies a convolutional neural network trained with more than a million patch-images to identify abnormal cell regions.•We used the average area of segmented regions to rank images according to the probability that they contain abnormal cells.•Results showed that the proposed methodology achieved more accurate results and performed faster when compared with other existing methods.•All the codes and the database will be published upon paper acceptance. Ninety years after its invention, the Pap test continues to be the most used method for the early identification of cervical precancerous lesions. In this test, the cytopathologists look for microscopic abnormalities in and around the cells, which is a time-consuming and prone to human error task. This paper introduces computational tools for cytological analysis that incorporate cell segmentation deep learning techniques. These techniques are capable of processing both free-lying and clumps of abnormal cells with a high overlapping rate from digitized images of conventional Pap smears. Our methodology employs a preprocessing step that discards images with a low probability of containing abnormal cells without prior segmentation and, therefore, performs faster when compared with the existing methods. Also, it ranks outputs based on the likelihood of the images to contain abnormal cells. We evaluate our methodology on an image database of conventional Pap smears from real scenarios, with 108 fields-of-view containing at least one abnormal cell and 86 containing only normal cells, corresponding to millions of cells. Our results show that the proposed approach achieves accurate results (MAP = 0.936), runs faster than existing methods, and it is robust to the presence of white blood cells, and other contaminants.