Validated machine learning tools to distinguish immune checkpoint inhibitor, radiotherapy, COVID-19 and other infective pneumonitis

•Accurate differentiation between checkpoint inhibitor, radiation, and infective pneumonitis is crucial but challenging.•Correct diagnosis is made difficult by overlapping clinical presentations and radiological patterns.•Machine learning radiomic models can distinguish checkpoint inhibitor and radiation pneumonitis from COVID-19, non-COVID-19 infective pneumonitis, and each other.•Such tools have potential as a second reader to support oncologists, radiologists, and chest physicians in cases of diagnostic uncertainty. Pneumonitis is a well-described, potentially disabling, or fatal adverse effect associated with both immune checkpoint inhibitors (ICI) and thoracic radiotherapy. Accurate differentiation between checkpoint inhibitor pneumonitis (CIP) radiation pneumonitis (RP), and infective pneumonitis (IP) is crucial for swift, appropriate, and tailored management to achieve optimal patient outcomes. However, correct diagnosis is often challenging, owing to overlapping clinical presentations and radiological patterns. In this multi-centre study of 455 patients, we used machine learning with radiomic features extracted from chest CT imaging to develop and validate five models to distinguish CIP and RP from COVID-19, non-COVID-19 infective pneumonitis, and each other. Model performance was compared to that of two radiologists. Models to distinguish RP from COVID-19, CIP from COVID-19 and CIP from non-COVID-19 IP out-performed radiologists (test set AUCs of 0.92 vs 0.8 and 0.8; 0.68 vs 0.43 and 0.4; 0.71 vs 0.55 and 0.63 respectively). Models to distinguish RP from non-COVID-19 IP and CIP from RP were not superior to radiologists but demonstrated modest performance, with test set AUCs of 0.81 and 0.8 respectively. The CIP vs RP model performed less well on patients with prior exposure to both ICI and radiotherapy (AUC 0.54), though the radiologists also had difficulty distinguishing this test cohort (AUC values 0.6 and 0.6). Our results demonstrate the potential utility of such tools as a second or concurrent reader to support oncologists, radiologists, and chest physicians in cases of diagnostic uncertainty. Further research is required for patients with exposure to both ICI and thoracic radiotherapy.