Computational analysis of histological images from hematoxylin and eosin-stained oral epithelial dysplasia tissue sections

Oral epithelial dysplasia is a precancerous lesion that presents alterations in the shape and size of cell nuclei and can be graded as mild, moderate and severe. The conventional process for diagnosis of this lesion is complex, time-consuming and subject to errors. The use of digital systems in histological analysis can aid specialists to obtain data that allows a robust and fast investigation of the lesion. This work presents a method for dysplasia quantification in histopathological images of the oral cavity using machine learning models. The methodology includes the steps of nuclei segmentation, post-processing, feature extraction and classification. On the segmentation step, the Mask R-CNN neural network was trained using nuclei masks, where objects were detected. The post-processing step employed morphological operations to remove false positive and negative areas. Then, 23 morphological and non-morphological features such as area, orientation, solidity and entropy were computed and a polynomial classifier was employed to distinguish the images among the lesion’s grades. This approach was applied in a dataset with 296 regions of mice tongue images, where 9155 cell nuclei were identified and analysed . Metrics such as accuracy and area under the ROC curve were employed to evaluate the methodology by comparing it with the gold standard marked by specialists and other methods present in the literature. This work presents a novel study for the classification of automated grading of oral dysplasia lesions based on the association of CNN segmentation and polynomial algorithm. The segmentation step resulted in accuracies ranging from 88.92% to 90.35% and the classification step obtained area under the ROC curve ranging from 0.88 to 0.97. When compared to other algorithms present in the literature, our methods showed more relevant results, obtaining higher accuracy and AUC values. These values showed that the proposed methodology contributed to the state-of-the-art and can be used as a tool to aid pathologists with precise values for investigating dysplastic tissue lesions. •A new approach for OED cell nuclei detection for the grades of the lesion.•Association of morphological and texture features with a polynomial classifier.•A dataset of images from mice tongues was employed in this study.•AUC values up to 0.96 were obtained for classifying the lesion grades.•A novel method for automatic grading of oral dysplasia.