SYSTEM AND METHOD FOR NAVIGATING A VEHICLE USING LANGUAGE INSTRUCTIONS

The invention relates to system (e.g. a neural network agent system) for outdoor navigating a vehicle using language instructions is provided. The system is configured to navigate the vehicle along an instructed navigation path which is represented by landmarks along the navigation path and subpaths between adjacent landmarks. The system comprises: · a language model configured to output semantic navigation information based on inputted language instructions describing the instructed navigation path, said semantic navigation information comprising at least one subpath description and at least one landmark description, · a memory module configured to store an already traveled path from the latest passed landmark, such that a state of the memory is continuously and/or repeatedly updated during traveling, · a directional matching model configured to estimate similarity between a subpath description and the current memory state, in order to output a directional matching score, · a landmark matching model configured to match the landmark description to a current image captured from the environment of the vehicle, in order to output a landmark matching score, · an action space probability prediction network configured to estimate possible traveling directions based on the current image, the memory state and the subpath description, in order to output an action space probability, and · an action model configured to predict a next navigation step at the current location based on the directional matching score, the landmark matching score and the estimation of the space probability. The invention further relates to a method for training a respective system.