GENERATING PREDICTIONS FOR NON-STATIONARY DATA USING DISTRIBUTIONS OVER OUTPUT HEAD WEIGHTS

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing online inference and learning using a neural network and Bayesian filtering. In one aspect, a method includes: receiving a data stream including a respective input at each of multiple time steps; and processing the data stream to generate a respective predicted output at each time step, including, at each time step: receiving the input at the time step; obtaining a set of distribution parameters for the time step that parametrizes a transition distribution for the time step; generating a set of weights for the time step using the set of distribution parameters for the time step; parametrizing an output network head of the neural network with the set of weights for the time step; and processing the input at the time step using the neural network to generate the predicted output at the time step.