CerviFusionNet: A multi-modal, hybrid CNN-transformer-GRU model for enhanced cervical lesion multi-classification

Cervical lesions pose a significant threat to women’s health worldwide. Colposcopy is essential for screening and treating cervical lesions, but its effectiveness depends on the doctor’s experience. Artificial intelligence-based solutions via colposcopy images have shown great potential in cervical lesions screening. However, some challenges still need to be addressed, such as low algorithm performance and lack of high-quality multi-modal datasets. Here, we established a multi-modal colposcopy dataset of 2,273 HPV+ patients, comprising original colposcopy images, acetic acid reactions at 60s and 120s, iodine staining, diagnostic reports, and pathological results. Utilizing this dataset, we developed CerviFusionNet, a hybrid architecture that merges convolutional neural networks and vision transformers to learn robust representations. We designed a temporal module to capture dynamic changes in acetic acid sequences, which can boost the model performance without sacrificing inference speed. Compared with several existing methods, CerviFusionNet demonstrated excellent accuracy and efficiency. [Display omitted] •Establishing a multi-modal dataset for cervical lesions classification•We developed CerviFusionNet, a novel AI model to process multi-modal dataset•CerviFusionNet achieved high-precision prediction for cervical lesions diagnosis•We evaluated the model’s robustness and generalization on the external dataset Medical imaging; Cervical smear; Classification of bioinformatical subject; Artificial intelligence