PROCESSING SEQUENCES OF MULTI-MODAL ENTITY FEATURES USING CONVOLUTIONAL NEURAL NETWORKS

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing sequences of multi-modal entity data using convolutional neural networks. One of the methods includes receiving an input sequence of multi-modal feature vectors characterizing an entity over a time window, wherein each multi-modal feature vector in the input sequence corresponds to a different time interval during the time window; processing the input sequence of multi-modal feature vectors using a convolutional neural network to generate a latent sequence that comprises a plurality of latent feature vectors; processing the latent sequence of latent feature vectors using an aggregation neural network to generate an aggregated feature vector; and processing the aggregated feature vector using an output neural network to generate a prediction that characterizes the entity after the time window.