Differentiation and risk stratification of basal cell carcinoma with deep learning on histopathologic images and measuring nuclei and tumor microenvironment features

Background Nuclear pleomorphism and tumor microenvironment (TME) play a critical role in cancer development and progression. Identifying most predictive nuclei and TME features of basal cell carcinoma (BCC) may provide insights into which characteristics pathologists can use to distinguish and stratify this entity. Objectives To develop an automated workflow based on nuclei and TME features from basaloid cell tumor regions to differentiate BCC from trichoepithelioma (TE) and stratify BCC into high‐risk (HR) and low‐risk (LR) subtypes, and to identify the nuclear and TME characteristics profile of different basaloid cell tumors. Methods The deep learning systems were trained on 161 H&E ‐stained sections which contained 51 sections of HR‐BCC, 50 sections of LR‐BCC and 60 sections of TE from one institution (D1), and externally and independently validated on D2 (46 sections) and D3 (76 sections), from 2015 to 2022. 60%, 20% and 20% of D1 data were randomly splitted for training, validation and testing, respectively. The framework comprised four stages: tumor regions identification by multi‐head self‐attention (MSA) U‐Net, nuclei segmentation by HoVer‐Net, quantitative feature by handcrafted extraction, and differentiation and risk stratification classifier construction. Pixel accuracy, precision, recall, dice score, intersection over union (IoU) and area under the curve (AUC) were used to evaluate the performance of tumor segmentation model and classifiers. Results MSA‐U‐Net model detected tumor regions with 0.910 precision, 0.869 recall, 0.889 dice score and 0.800 IoU. The differentiation classifier achieved 0.977 ± 0.0159, 0.955 ± 0.0181, 0.885 ± 0.0237 AUC in D1, D2 and D3, respectively. The most discriminative features between BCC and TE contained Homogeneity, Elongation, T‐T_meanEdgeLength, T‐T_Nsubgraph, S‐T_HarmonicCentrality, S‐S_Degrees. The risk stratification model can well predict HR‐BCC and LR‐BCC with 0.920 ± 0.0579, 0.839 ± 0.0176, 0.825 ± 0.0153 AUC in D1, D2 and D3, respectively. The most discriminative features between HR‐BCC and LR‐BCC comprised IntensityMin, Solidity, T‐T_minEdgeLength, T‐T_Coreness, T‐T_Degrees, T‐T_Betweenness, S‐T_Degrees. Conclusions This framework hold potential for future use as a second opinion helping inform diagnosis of BCC, and identify nuclei and TME features related with malignancy and tumor risk stratification.