Automated Cervical Cytology Image Cell Segmentation Using Enhanced MultiResUNet With DCT and Spectral Domain Attention Mechanisms

Cervical cancer, the most pervasive malignancy among women world-wide has a prolonged latent period necessitating early diagnosis with appropriate treatment for significantly increasing the survival rate and life expectancy. Screening cytology images, like Pap smear images are used for its diagnosis. The traditional methods of manually screening the images are time-intensive and highly prone to human error leading to the development of automated screening methods built on deep learning techniques to improve the efficacy and reliability of the diagnosis. Precise cytoplasm and nucleus segmentation in cervical cytology images is a crucial phase of this automated image analysis, but due to the unavailability of higher quality open-source cervical datasets, the deep learning-based automated cytology segmentation is still in the developing stage and is a vital area of research. Hence, this work proposes a novel deep multi-residual encoder-decoder network enhanced with spectral domain block and channel attention analyzing Discrete cosine transform (DCT)-based spectral features for efficient and accurate semantic segmentation of cytoplasm and nucleus regions in the complex cytology images of the cervix. The proposed model has been standardized on a recent publicly accessible high-quality Cx22 cervical segmentation dataset, the only large dataset known to be free from false negatives. While our work gives competitive results for cytoplasm segmentation with a remarkable dice coefficient of 94.5%, it outperforms for nucleus segmentation with a huge margin of 4% at a dice-coefficient of 81.1% with decent model complexity. The paper also presents the generalization capability of the proposed deep model by evaluating it on two other standard cervical image datasets namely, ISBI 2014 and CCEDD dataset. The evaluation resulted in competent cytoplasm segmentation results and enhanced nucleus segmentation results for both datasets, thereby, proving the model's generalization efficiency.