ResQu-Net: Effective prostate’s peripheral zone segmentation leveraging the representational power of attention-based mechanisms

•We propose a novel Deep Learning model, namely the ResQu-Net, specifically tailored for Peripheral Zone Segmentation on T2W MR images.•Six DL architectures were implemented and compared using three openly available datasets with 392 patient cases.•Qualitative Explainability Analysis reveals that conventional segmentation metrics, such as Dice Score, and Hausdorff Distance, might not reflect properly the performance of a DL segmentation model. Prostate cancer is a leading cause of male cancer worldwide. With more than 70 % of prostate cancers arising in the peripheral zone of the prostate, accurate segmentation of this region is of paramount importance for the effective diagnosis and treatment of the disease. Although peripheral zone is well recognized as one of the most challenging regions to delineate within the prostate, no algorithms specifically tailored for this segmentation task are currently available. The present study introduces a new deep learning (DL) algorithm, named as ResQu-Net, which is designed to accurately segment the peripheral zone (PZ) of the prostate on T2-weighted magnetic resonance imaging (MRI). Using three publicly available datasets, the ResQu-Net outperformed the six DL segmentation models used for comparison, namely the Attention U-Net, the Dense2U-Net, the Proper-Net, the TransU-net, the U-Net, and the USE-Net, demonstrating superior performance for different anatomical regions, such as the apex, the midgland and the base. The assessment of the suggested approach was conducted not only quantitatively (Sensitivity, Balanced Accuracy, Dice Score, 95 % Hausdorff Distance, and Average Surface Distance) but also qualitatively. For the qualitative evaluation the feature maps obtained from the last layers of each model were compared with the Density Map of the Ground Truth annotations using root mean squared error. Overall, the ResQu-Net model exhibits improvedperformance compared to other models, of more than 5 % and 1.87 mm in terms of Dice Score and 95 % Hausdorff Distance, respectively. These advancements may contribute significantly in addressing the challenges associated with PZ segmentation, and ultimately enabling improved clinical decision-making and patient outcomes.