Predicting BRCA mutation and stratifying targeted therapy response using multimodal learning: a multicenter study

The status of genes plays a crucial role in the treatment decision-making process for multiple cancer types. However, due to high costs and limited resources, a demand for genetic testing among patients is currently unmet. Notably, not all patients with mutations achieve favorable outcomes with poly (ADP-ribose) polymerase inhibitors (PARPi), indicating the necessity for risk stratification. In this study, we aimed to develop and validate a multimodal model for predicting gene status and prognosis with PARPi treatment. We included 1695 slides from 1417 patients with ovarian, breast, prostate, and pancreatic cancers across three independent cohorts. Using a self-attention mechanism, we constructed a multi-instance attention model (MIAM) to detect gene status from hematoxylin and eosin (H&E) pathological images. We further combined tissue features from the MIAM model, cell features, and clinical factors (the MIAM-C model) to predict mutations and progression-free survival (PFS) with PARPi therapy. Model performance was evaluated using area under the curve (AUC) and Kaplan-Meier analysis. Morphological features contributing to MIAM-C were analyzed for interpretability. Across the four cancer types, MIAM-C outperformed the deep learning-based MIAM in identifying the genotype. Interpretability analysis revealed that high-attention regions included high-grade tumors and lymphocytic infiltration, which correlated with mutations. Notably, high lymphocyte ratios appeared characteristic of mutations. Furthermore, MIAM-C predicted PARPi therapy response (log-rank