Semantic-based cross-modal knowledge association method and device, equipment and storage medium

The invention relates to the technical field of multi-modal data application, and discloses a cross-modal knowledge association method and device based on semantics, equipment and a storage medium. The method comprises the following steps: acquiring cross-modal knowledge and language description corresponding to the cross-modal knowledge as a semantic cross-modal data set; a cross-modal knowledge association model is constructed, the cross-modal knowledge association model comprises a semantic mask sub-model and a fusion transformation factor generation sub-model, and the semantic mask sub-model is used for performing mask processing on a semantic cross-modal data set to generate partial data most related to language description; the fusion transformation factor generation sub-model is used for processing the semantic cross-modal data set to generate transformation factors; and inputting the semantic cross-modal data set into the trained cross-modal knowledge association model, and outputting a cross-modal kn