Multimodal fuzzy granular representation and classification

In a complex classification task, samples are represented by various types of multimodal features, including structured data, text, images, video, audio, etc. These data are usually high dimensionally, large-sized, structurally complex, and semantically inconsistent. The representation, translation, alignment, fusion and co-learning of multimodal data are core technical challenges to traditional classification tasks. Kernel functions are applied in dealing with multimodal data for extracting some nonlinear information. However, they cannot consider the aspects of complex structures and uncertain semantics in a multimodal classification task. Fuzzy granular computing emerges as a powerful vehicle to handle the structured and uncertain multimodal data. In this paper, we propose a framework of multimodal classification based on kernel functions and fuzzy granular computing. First, a fuzzy granulation based on kernel functions is introduced to extract nonlinear features for the multimodal classification. Then, a model of multimodal fuzzy classification including fuzzy granular representation, fusion and learning for multimodal data is constructed. Finally, we design an efficient fuzzy granular classification algorithm for big multimodal data based on the proposed model. Experimental results demonstrate the effectiveness of our proposed model and its corresponding algorithm. Graphical abstract