Confidence-aware multi-modality learning for eye disease screening

Multi-modal ophthalmic image classification plays a key role in diagnosing eye diseases, as it integrates information from different sources to complement their respective performances. However, recent improvements have mainly focused on accuracy, often neglecting the importance of confidence and robustness in predictions for diverse modalities. In this study, we propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information using a multi-distribution fusion perspective. Specifically, our method first utilizes normal inverse gamma prior distributions over pre-trained models to learn both aleatoric and epistemic uncertainty for uni-modality. Then, the normal inverse gamma distribution is analyzed as the Student’s t distribution. Furthermore, within a confidence-aware fusion framework, we propose a mixture of Student’s t distributions to effectively integrate different modalities, imparting the model with heavy-tailed properties and enhancing its robustness and reliability. More importantly, the confidence-aware multi-modality ranking regularization term induces the model to more reasonably rank the noisy single-modal and fused-modal confidence, leading to improved reliability and accuracy. Experimental results on both public and internal datasets demonstrate that our model excels in robustness, particularly in challenging scenarios involving Gaussian noise and modality missing conditions. Moreover, our model exhibits strong generalization capabilities to out-of-distribution data, underscoring its potential as a promising solution for multimodal eye disease screening. •We propose a novel confidence-aware multi-modality eye disease screening method.•We present MoSt to dynamically adapt to heavy-tailed data distributions and capture modality-specific uncertainty.•We introduce an innovative confidence-based ranking regularization term for multi-modal eye disease screening.•We conduct sufficient experiments on the public and in-house datasets with different eye diseases, which clearly verify the accuracy, robustness, and reliability to normal and OOD test samples.