A novel multi-task learning network for skin lesion classification based on multi-modal clues and label-level fusion

In this paper, we propose a multi-task learning (MTL) network based on the label-level fusion of metadata and hand-crafted features by unsupervised clustering to generate new clustering labels as an optimization goal. We propose a MTL module (MTLM) that incorporates an attention mechanism to enable the model to learn more integrated, variable information. We propose a dynamic strategy to adjust the loss weights of different tasks, and trade off the contributions of multiple branches. Instead of feature-level fusion, we propose label-level fusion and combine the results of our proposed MTLM with the results of the image classification network to achieve better lesion prediction on multiple dermatological datasets. We verify the effectiveness of the proposed model by quantitative and qualitative measures. The MTL network using multi-modal clues and label-level fusion can yield the significant performance improvement for skin lesion classification. •The first MTL for skin lesion classification using multi-modal clues and label-level fusion.•The MTL network using branch ensemble and task interaction for multidomain knowledge.•The label-level fusion of multi-modal data using unsupervised clustering.•The metadata and hand-crafted features combined to avoid feature unalignment.•The hyperbolic weight adjustment strategy to balance the MTL and accelerate training.