METHODS AND SYSTEMS FOR ONLINE SELECTION OF NUMBER FORMATS FOR NETWORK PARAMETERS OF A NEURAL NETWORK

Methods and neural network accelerator for online selection of number formats for network parameters of a neural network. The neural network accelerator comprises at least one network processing hardware unit configured to receive network parameters for layers of the neural network and perform one or more neural network operations on the received network parameters in accordance with the neural network; a statistics collection hardware unit configured to collect one or more statistics on a first set of network parameters for a layer while the neural network accelerator is performing a pass of the neural network; and a format conversion hardware unit configured to convert a second set of network parameters to a number format selected based on the collected one or more statistics, the second set of network parameters comprising (i) the first set of network parameters and/or another set of network parameters for the layer, or (ii) a set of network parameters for a subsequent pass of the neural network corresponding to the first set of network parameters.