Ex vivo radiation sensitivity assessment for individual head and neck cancer patients using deep learning-based automated nuclei and DNA damage foci detection

•The model assesses patient-specific radiosensitivity in ex vivo HNSCC tissue.•The model is based on 53BP1 DNA damage foci and nuclei microscopy images.•The model is based on a deep learning and conventional image analysis techniques.•This model can replace manual foci analysis for ex vivo HNSCC tissue.•The model reduces the image-analysis time and avoids inter-observer variability. Tumor biopsy tissue response to ex vivo irradiation is potentially an interesting biomarker for in vivo tumor response, therefore, for treatment personalization. Tumor response ex vivo can be characterized by DNA damage response, expressed by the large-scale presence of DNA damage foci in tumor nuclei. Currently, characterizing tumor nuclei and DNA damage foci is a manual process that takes hours per patient and is subjective to inter-observer variability, which is not feasible in for clinical decision making. Therefore, our goal was to develop a method to automatically segment nuclei and DNA damage foci in tumor tissue samples treated with radiation ex vivo to characterize the DNA damage response, as potential biomarker for in vivo radio-sensitivity. Oral cavity tumor tissue of 21 patients was irradiated ex vivo (5 or 0 Gy), fixated 2 h post-radiation, and used to develop our method for automated nuclei and 53BP1 foci segmentation. The segmentation model used both deep learning and conventional image-analysis techniques. The training (22 %), validation (22 %), and test set (56 %) consisted of thousands of manually segmented nuclei and foci. The segmentations and number of foci per nucleus in the test set were compared to their ground truths. The automatic nuclei and foci segmentations were highly accurate (Dice = 0.901 and Dice = 0.749, respectively). An excellent correlation (R2 = 0.802) was observed for the foci per nucleus that outperformed reported inter-observation variation. The analysis took ∼ 8 s per image. This model can replace manual foci analysis for ex vivo irradiation of head-and-neck squamous cell carcinoma tissue, reduces the image-analysis time from hours to minutes, avoids the problem of inter-observer variability, enables assessment of multiple images or conditions, and provides additional information about the foci size. Thereby, it allows for reliable and rapid ex vivo radio-sensitivity assessment, as potential biomarker for response in vivo and treatment personalization.