Robot control method and device based on visual language pre-training model and medium

The invention relates to a robot control method and device based on a visual language pre-training model and a medium, and the method comprises the steps: obtaining real-time visual perception information and a natural language instruction, taking the visual perception information and the natural language instruction as the input of a control strategy deep learning network model, obtaining a corresponding robot action instruction; wherein the training process of the control strategy deep learning network model comprises the following steps: building a simulation environment for robot control, generating a first training data set in the simulation environment, and constructing the control strategy deep learning network model containing visual language pre-training to pre-train the control strategy deep learning network model; and a real scene data set is collected and processed, a second training data set is generated, small sample migration training and fine model parameter adjustment are performed on the pre